Genozip: a universal extensible genomic data compressor.

Divon Lan,Yassine Souilmi,Bastien Llamas,Ray Tobler,Janet Kelso

doi:10.1093/bioinformatics/btab102

Divon Lan, Yassine Souilmi + Show 3 more

Open Access

https://doi.org/10.1093/bioinformatics/btab102

Copy DOI

Abstract

We present Genozip, a universal and fully featured compression software for genomic data. Genozip is designed to be a general-purpose software and a development framework for genomic compression by providing five core capabilities—universality (support for all common genomic file formats), high compression ratios, speed, feature-richness and extensibility. Genozip delivers high-performance compression for widelyused genomic data formats in genomics research, namely FASTQ, SAM/BAM/CRAM, VCF, GVF, FASTA, PHYLIP and 23andMe formats. Our test results show that Genozip is fast and achieves greatly improved compression ratios, even when the files are already compressed. Further, Genozip is architected with a separation of the Genozip Framework from file-format-specific Segmenters and data-type-specific Codecs. With this, we intend for Genozip to be a general-purpose compression platform where researchers can implement compression for additional file formats, as well as new codecs for data types or fields within files, in the future. We anticipate that this will ultimately increase the visibility and adoption of these algorithms by the user community, thereby accelerating further innovation in this space.Availability and implementationGenozip is written in C. The code is open-source and available on http://www.genozip.com. The package is free for non-commercial use. It is distributed through the Conda package manager, github, and as a Docker container on DockerHub. Genozip is tested on Linux, Mac and Windows.Supplementary informationSupplementary data are available at Bioinformatics online.

Highlights

Genomic data production is growing rapidly as sequencing prices continue to drop, making data storage and transfer a core issue for researchers, healthcare providers, service facilities and private companies
Many novel algorithms have emerged in recent years that effectively compress one or more of the data types embedded in genomic files [e.g.GTShark (Deorowicz and Danek, 2019) and SPRING (Chandak et al, 2019)]
We evaluated the performance of Genozip by compressing genomic files as they most commonly appear in real-world research and clinical situations—namely, already compressed in fastq.gz, BAM, CRAM and vcf.gz formats

Summary

Introduction

Genomic data production is growing rapidly as sequencing prices continue to drop, making data storage and transfer a core issue for researchers, healthcare providers, service facilities and private companies. Many novel algorithms have emerged in recent years that effectively compress one or more of the data types embedded in genomic files [e.g.GTShark (Deorowicz and Danek, 2019) and SPRING (Chandak et al, 2019)]. These algorithms are typically implemented within a rudimentary software package that inadvertently lacks the breadth of features required for a software to be useful in many real-world use cases; most importantly, most work with only one of the common file formats. Genozip offers five core capabilities: 1. Universality—Genozip supports all common genomic file formats—FASTQ, SAM/BAM/CRAM, VCF, GVF, FASTA, PHYLIP and 23andMe

Methods

Results

Conclusion