FCLQC: fast and concurrent lossless quality scores compressor

Minhyeok Cho,Albert No

doi:10.1186/s12859-021-04516-7

Abstract

BackgroundAdvances in sequencing technology have drastically reduced sequencing costs. As a result, the amount of sequencing data increases explosively. Since FASTQ files (standard sequencing data formats) are huge, there is a need for efficient compression of FASTQ files, especially quality scores. Several quality scores compression algorithms are recently proposed, mainly focused on lossy compression to boost the compression rate further. However, for clinical applications and archiving purposes, lossy compression cannot replace lossless compression. One of the main challenges for lossless compression is time complexity, where it takes thousands of seconds to compress a 1 GB file. Also, there are desired features for compression algorithms, such as random access. Therefore, there is a need for a fast lossless compressor with a reasonable compression rate and random access functionality.ResultsThis paper proposes a Fast and Concurrent Lossless Quality scores Compressor (FCLQC) that supports random access and achieves a lower running time based on concurrent programming. Experimental results reveal that FCLQC is significantly faster than the baseline compressors on compression and decompression at the expense of compression ratio. Compared to LCQS (baseline quality score compression algorithm), FCLQC shows at least 31x compression speed improvement in all settings, where a performance degradation in compression ratio is up to 13.58% (8.26% on average). Compared to general-purpose compressors (such as 7-zip), FCLQC shows 3x faster compression speed while having better compression ratios, at least 2.08% (4.69% on average). Moreover, the speed of random access decompression also outperforms the others. The concurrency of FCLQC is implemented using Rust; the performance gain increases near-linearly with the number of threads.ConclusionThe superiority of compression and decompression speed makes FCLQC a practical lossless quality score compressor candidate for speed-sensitive applications of DNA sequencing data. FCLQC is available at https://github.com/Minhyeok01/FCLQC and is freely available for non-commercial usage.

Highlights

IntroductionThe amount of sequencing data increases explosively
Advances in sequencing technology have drastically reduced sequencing costs
This section describes the experimental results of proposed lossless quality scores compressor Fast and Concurrent Lossless Quality scores Compressor (FCLQC) as well as experimental setups

Summary

Introduction

The amount of sequencing data increases explosively. Since FASTQ files (standard sequencing data formats) are huge, there is a need for efficient compression of FASTQ files, especially quality scores. Several quality scores compression algorithms are recently proposed, mainly focused on lossy compression to boost the compression rate further. There is a need for a fast lossless compressor with a reasonable compression rate and random access functionality. Since the Human Genome Project (HGP), sequencing technology has developed rapidly [1]. Proposed Generation Sequencing (NGS) technologies support massive parallel sequencing, which lowers sequencing costs. The sequencing data is mainly stored in FASTQ format, which is widely being used in bioinformatics. The size of the FASTQ file is gigantic, where the size of human genome data ranges from tens to hundreds of gigabytes. The size of the homo sapiens FASTQ file SRR13587127 obtained from the Illumina HiSeq machine is 111 GB

Objectives

Results

Conclusion