Compression of Structured High-Throughput Sequencing Data

Fabien Campagne,Jill P Mesirov,Nyasha Chambwe,Kevin C Dorff,James T Robinson

doi:10.1371/journal.pone.0079871

Fabien Campagne, Jill P Mesirov + Show 3 more

Open Access

https://doi.org/10.1371/journal.pone.0079871

Copy DOI

Abstract

Large biological datasets are being produced at a rapid pace and create substantial storage challenges, particularly in the domain of high-throughput sequencing (HTS). Most approaches currently used to store HTS data are either unable to quickly adapt to the requirements of new sequencing or analysis methods (because they do not support schema evolution), or fail to provide state of the art compression of the datasets. We have devised new approaches to store HTS data that support seamless data schema evolution and compress datasets substantially better than existing approaches. Building on these new approaches, we discuss and demonstrate how a multi-tier data organization can dramatically reduce the storage, computational and network burden of collecting, analyzing, and archiving large sequencing datasets. For instance, we show that spliced RNA-Seq alignments can be stored in less than 4% the size of a BAM file with perfect data fidelity. Compared to the previous compression state of the art, these methods reduce dataset size more than 40% when storing exome, gene expression or DNA methylation datasets. The approaches have been integrated in a comprehensive suite of software tools (http://goby.campagnelab.org) that support common analyses for a range of high-throughput sequencing assays.

Highlights

Many scientific disciplines, including high-energy physics, astronomy and more recently biology, generate increasing volumes of data from automated measurement instruments
The approach was integrated in a comprehensive software system, which includes the Goby framework, IGV [8], BWA [9] and GSNAP [10], and demonstrates proof of principle for compression, visualization and analysis of high-throughput sequencers (HTS) data with these new approaches
We developed codecs for general compression methods (PB data compressed with the GZip or BZip2 methods, Figure 1A), and a Hybrid codec that provides very strong compression of alignment data, while retaining the flexibility of Protocol Buffers technology (PB) schema evolution

Summary

Introduction

Many scientific disciplines, including high-energy physics, astronomy and more recently biology, generate increasing volumes of data from automated measurement instruments. Modern high-throughput sequencers (HTS) are producing a large fraction of new biological data and are being successfully applied to study genomes, transcriptomes, epigenomes or other data modalities with a variety of new assays that take advantage of the throughput of sequencing methods [1,2,3]. Sequencing throughput has more than doubled every year for the last ten years [4] resulting in storage requirements on the order of tens of terabytes of primary and secondary high throughput sequencing data in a typical laboratory. Major sequencing centers in the USA and worldwide typically require several tens of petabytes of storage to store reads and secondary data during the lifetime of their projects. Read and alignment data are deposited in sequence archives to enable other groups to reanalyze the data. How to store these data to minimize storage costs, maximize computational efficiency for data analysis, increase network transfer speeds to facilitate collaborative studies, or to facilitate reanalysis or perusal of data stored in archives

Methods

Results

Discussion

Conclusion