Supporting data for "Field of Genes: Using Apache Kafka as a Bioinformatic Data Repository"

Dataset type: Software
Data released on March 20, 2018

Lawlor B; Lynch R; Mac Aogain M; Walsh P (2018): Supporting data for "Field of Genes: Using Apache Kafka as a Bioinformatic Data Repository" GigaScience Database.


Bioinformatic research is increasingly dependent on large-scale data sets, accessed either from private or public repositories. An example of a public repository is NCBI's RefSeq. These repositories must decide in what form to make their data available. Unstructured data can be put to almost any use, but are limited in how access to them can be scaled. Highly structured data offer improved performance for specific algorithms but limit the wider usefulness of the data. We present an alternative: lightly structured data stored in Apache Kafka in a way that is amenable to parallel access and streamed processing, including subsequent transformations into more highly structured representations. We contend that this approach could provide a flexible and powerful nexus of bioinformatic data, bridging the gap between low-structure on one hand, and high-performance and scale on the other. To demonstrate this, we present a proof of concept version of NCBI's RefSeq database using this technology. We measure the performance and scalability characteristics of this alternative with respect to flat files.
The proof of concept scales almost linearly as more compute nodes are added, outperforming the standard approach using files.
Apache Kafka merits consideration as a fast and more scalable but general-purpose way to store and retrieve bioinformatic data, for public, centralized reference datasets like RefSeq, and private clinical and experimental data.

Additional details

Read the peer-reviewed publication(s):

Lawlor, B., Lynch, R., Mac Aogáin, M., & Walsh, P. (2018). Field of genes: using Apache Kafka as a bioinformatic data repository. GigaScience, 7(4). doi:10.1093/gigascience/giy036

Additional information:

File NameSample IDData TypeFile FormatSizeRelease Date 
mixed archivearchive92.3 KB2018-03-19
ReadmeTEXT1.86 KB2018-03-19
Displaying 1-2 of 2 File(s).
Funding body Awardee Award ID Comments
European Commission P Walsh 324365 Seventh framework programme
Horizon 2020 Framework Programme 644186 Marie Skaodowska-Curie
Irish Research Council (IE) Michael Mac Aogain EPSPD/2015/32
Science Foundation Ireland Paul Walsh 16/IFA/4342
Date Action
March 20, 2018 Dataset publish
July 4, 2018 Manuscript Link added : 10.1093/gigascience/giy036