In just over a decade, metagenomics has developed into a powerful and productive method in microbiology and microbial ecology. The ability to retrieve and organize bits and pieces of genomic DNA from any natural context has opened a window into the vast universe of uncultivated microbes. Tremendous progress has been made in computational approaches to interpret this sequence data but none can completely recover the complex information encoded in metagenomes. A number of challenges stand in the way. Simplifying assumptions are needed and lead to strong limitations and potential inaccuracies in practice. Critically, methodological improvements are difficult to gauge due to the lack of a general standard for comparison. Developers also face a substantial burden to individually evaluate existing approaches, which consumes time and computational resources, and may introduce unintended biases.

The Critical Assessment of Metagenome Interpretation (CAMI) is a community-led initiative that tackles these problems by aiming for an independent, comprehensive and bias-free evaluation of methods. In the first CAMI challenge running from March to July 2015, it provided three simulated benchmark metagenome datasets of different organismal complexities and sizes. These were generated from around ~700 newly sequenced genomes and ~600 circular elements (plasmids, viruses, other circular elements) not included in public databases during the challenge. These are now available here, together with gold standards for assembly, genome and taxonomic binning and taxonomic profiling, the underlying genome sequences, NCBI and ARB reference sequences snapshots from before the challenge and the reference NCBI taxonomy used. In addition, 3 test (toy) data sets are provided that were simulated from public genomes before the challenge. For the most realistic evaluation of reference based methods on the challenge data sets, usually taxonomic binners and profilers, the provided reference sequences or other sequence collections from before challenge should be used as references, as by now all underlying genomes have been deposited at NCBI or EBI.

Additional details

Related datasets:

doi:10.5524/100344 IsCitedBy doi:10.5524/102350
doi:10.5524/100344 Cites doi:10.5524/102408

Additional information:

https://data.cami-challenge.org/participate





Sample IDTaxonomic IDCommon NameGenbank NameScientific NameSample Attributes
CAMI_high1235509synthetic metagenomesynthetic metagenome Description:a 75 Gb time series dataset with five samples from a high complexity community with correlated log normal abundance distributions (596 genomes and 478 circular elements; not included in the reference sequence collections also provided in this archive)
CAMI_low1235509synthetic metagenomesynthetic metagenome Description:a 15 Gb single sample dataset from a low complexity community with log normal abundance distribution (40 genomes and 20 circular elements; not included in the reference sequence collections also provided in this archive)
CAMI_medium1235509synthetic metagenomesynthetic metagenome Description:a 40 Gb differential log normal abundance dataset with two samples of a medium complexity community (132 genomes and 100 circular elements; not included in the reference sequence collections also provided in this archive) and long and short insert sizes
CAMI_TOY_high1235509synthetic metagenomesynthetic metagenome Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. 5 Hiseq (small insert size) 15 Gbp samples (time series) from 450 genomes (included in the reference sequence collections also provided in this archive) 15 Giga base pairs (each sample) Insert size mean: 180 bp Insert size stddev: 18 bp Read length: 2x100 bp
CAMI_TOY_low1235509synthetic metagenomesynthetic metagenome Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. Genomes: 30 (included in the reference sequence collections also provided in this archive), Total Size: 15 Gbp, Read length: 2x100 bp, Insert size mean: 180 bp, Insert size stddev: 10%.
CAMI_TOY_medium1235509synthetic metagenomesynthetic metagenome Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. Two samples, differential abundance 2 Hiseq (small insert size) differential abundance 15 Gbp samples from 225 genomes (included in the reference sequence collections also provided in this archive). From the same two differential abundance community profiles, 2 Hiseq (5kb insert size) 0.75 Gbp samples
Displaying 1-6 of 6 Sample(s).




File NameSample IDData TypeFile FormatSizeRelease Date 
CAMI_TOY_mediumMixed archiveTAR31.26 GB2017-08-11
CAMI_TOY_lowMixed archiveTAR14.35 GB2017-08-11
CAMI_TOY_highMixed archiveTAR75 GB2017-08-11
Mixed archiveTAR90.26 MB2017-08-11
CAMI_highMixed archiveTAR49.67 GB2017-08-11
CAMI_lowMixed archiveTAR10.48 GB2017-08-11
CAMI_mediumMixed archiveTAR26.46 GB2017-08-11
Mixed archiveTAR7.63 GB2017-08-11
Mixed archiveTAR147.18 GB2017-08-11
Mixed archiveTAR11.86 GB2019-02-26
Displaying 1-10 of 12 File(s).
Date Action
August 11, 2017 Dataset publish
February 26, 2019 File program_results.tar.gz updated
February 26, 2019 File readme_100344.txt updated
February 26, 2019 readme_100344.txt: file attribute updated
February 26, 2019 File readme_100344.txt updated
February 26, 2019 program_results.tar.gz: additional file attribute added
February 26, 2019 File program_results.tar.gz updated
June 14, 2023 Relationship added : DOI 102408