Benchmark data sets, software results and reference data for the first CAMI challenge.
Dataset type: Metagenomic, Software
Data released on August 11, 2017
Sczyrba A; Hofman P; Belmann P; Koslicki D; Janssen S; Dröge J; Gregor I; Majda S; Fiedler J; Dahms E; Bremges A; Fritz A; Garrido-Oter R; Jørgensen TS; Shapiro N; Blood PD; Gurevich A; Bai Y; Turaev D; DeMaere MZ; Chikhi R; Nagarajan N; Quince C; Meyer F; Balvočiūtė M; Hansen LH; Sørensen SJ; H. Chia BK; Denis B; Froula JL; Wang Z; Egan R; Kang DD; Cook JJ; Deltel C; Beckstette M; Lemaitre C; Peterlongo P; Rizk G; Lavenier D; Wu Y; Singer SW; Jain C; Strous M; Klingenberg H; Meinicke P; Barton M; Lingner T; Lin H; Liao Y; Z. Silva GG; Cuevas DA; Edwards RA; Saha S; Piro VC; Renard BY; Pop M; Klenk H; Göker M; Kyrpides NC; Woyke T; Vorholt JA; Schulze-Lefert P; Rubin EM; Darling AE; Rattei T; McHardy AC (2017): Benchmark data sets, software results and reference data for the first CAMI challenge. GigaScience Database. http://dx.doi.org/10.5524/100344
In just over a decade, metagenomics has developed into a powerful and productive method in microbiology and microbial ecology. The ability to retrieve and organize bits and pieces of genomic DNA from any natural context has opened a window into the vast universe of uncultivated microbes. Tremendous progress has been made in computational approaches to interpret this sequence data but none can completely recover the complex information encoded in metagenomes. A number of challenges stand in the way. Simplifying assumptions are needed and lead to strong limitations and potential inaccuracies in practice. Critically, methodological improvements are difficult to gauge due to the lack of a general standard for comparison. Developers also face a substantial burden to individually evaluate existing approaches, which consumes time and computational resources, and may introduce unintended biases.
The Critical Assessment of Metagenome Interpretation (CAMI) is a community-led initiative that tackles these problems by aiming for an independent, comprehensive and bias-free evaluation of methods. In the first CAMI challenge running from March to July 2015, it provided three simulated benchmark metagenome datasets of different organismal complexities and sizes. These were generated from around ~700 newly sequenced genomes and ~600 circular elements (plasmids, viruses, other circular elements) not included in public databases during the challenge. These are now available here, together with gold standards for assembly, genome and taxonomic binning and taxonomic profiling, the underlying genome sequences, NCBI and ARB reference sequences snapshots from before the challenge and the reference NCBI taxonomy used. In addition, 3 test (toy) data sets are provided that were simulated from public genomes before the challenge. For the most realistic evaluation of reference based methods on the challenge data sets, usually taxonomic binners and profilers, the provided reference sequences or other sequence collections from before challenge should be used as references, as by now all underlying genomes have been deposited at NCBI or EBI.
Additional details
Related datasets:
doi:10.5524/100344 IsCitedBy doi:10.5524/102350
doi:10.5524/100344 Cites doi:10.5524/102408
Additional information:
Sample ID | Taxonomic ID | Common Name | Genbank Name | Scientific Name | Sample Attributes |
---|---|---|---|---|---|
CAMI_high | 1235509 | synthetic metagenome | synthetic metagenome | Description:a 75 Gb time series dataset with five samples from a high complexity community with correlated log normal abundance distributions (596 genomes and 478 circular elements; not included in the reference sequence collections also provided in this archive) | |
CAMI_low | 1235509 | synthetic metagenome | synthetic metagenome | Description:a 15 Gb single sample dataset from a low complexity community with log normal abundance distribution (40 genomes and 20 circular elements; not included in the reference sequence collections also provided in this archive) | |
CAMI_medium | 1235509 | synthetic metagenome | synthetic metagenome | Description:a 40 Gb differential log normal abundance dataset with two samples of a medium complexity community (132 genomes and 100 circular elements; not included in the reference sequence collections also provided in this archive) and long and short insert sizes | |
CAMI_TOY_high | 1235509 | synthetic metagenome | synthetic metagenome | Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. 5 Hiseq (small insert size) 15 Gbp samples (time series) from 450 genomes (included in the reference sequence collections also provided in this archive) 15 Giga base pairs (each sample) Insert size mean: 180 bp Insert size stddev: 18 bp Read length: 2x100 bp | |
CAMI_TOY_low | 1235509 | synthetic metagenome | synthetic metagenome | Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. Genomes: 30 (included in the reference sequence collections also provided in this archive), Total Size: 15 Gbp, Read length: 2x100 bp, Insert size mean: 180 bp, Insert size stddev: 10%. | |
CAMI_TOY_medium | 1235509 | synthetic metagenome | synthetic metagenome | Description:a toy data set simulated from public genomes. Can be used for testing tools (gold standards provided). THIS IS NOT A CHALLENGE DATA SET. Two samples, differential abundance 2 Hiseq (small insert size) differential abundance 15 Gbp samples from 225 genomes (included in the reference sequence collections also provided in this archive). From the same two differential abundance community profiles, 2 Hiseq (5kb insert size) 0.75 Gbp samples |