Distributed hybrid-indexing of compressed pan-genomes for scalable and fast sequence alignment.

Altti Ilari Maarala,Ossi Arasalo,Daniel Valenzuela,Veli Mäkinen,Keijo Heljanko

doi:10.1371/journal.pone.0255260

Altti Ilari Maarala, Ossi Arasalo + Show 3 more

Open Access

https://doi.org/10.1371/journal.pone.0255260

Copy DOI

Abstract

Computational pan-genomics utilizes information from multiple individual genomes in large-scale comparative analysis. Genetic variation between case-controls, ethnic groups, or species can be discovered thoroughly using pan-genomes of such subpopulations. Whole-genome sequencing (WGS) data volumes are growing rapidly, making genomic data compression and indexing methods very important. Despite current space-efficient repetitive sequence compression and indexing methods, the deployed compression methods are often sequential, computationally time-consuming, and do not provide efficient sequence alignment performance on vast collections of genomes such as pan-genomes. For performing rapid analytics with the ever-growing genomics data, data compression and indexing methods have to exploit distributed and parallel computing more efficiently. Instead of strict genome data compression methods, we will focus on the efficient construction of a compressed index for pan-genomes. Compressed hybrid-index enables fast sequence alignments to several genomes at once while shrinking the index size significantly compared to traditional indexes. We propose a scalable distributed compressed hybrid-indexing method for large genomic data sets enabling pan-genome-based sequence search and read alignment capabilities. We show the scalability of our tool, DHPGIndex, by executing experiments in a distributed Apache Spark-based computing cluster comprising 448 cores distributed over 26 nodes. The experiments have been performed both with human and bacterial genomes. DHPGIndex built a BLAST index for n = 250 human pan-genome with an 870:1 compression ratio (CR) in 342 minutes and a Bowtie2 index with 157:1 CR in 397 minutes. For n = 1,000 human pan-genome, the BLAST index was built in 1520 minutes with 532:1 CR and the Bowtie2 index in 1938 minutes with 76:1 CR. Bowtie2 aligned 14.6 GB of paired-end reads to the compressed (n = 1,000) index in 31.7 minutes on a single node. Compressing n = 13,375,031 (488 GB) GenBank database to BLAST index resulted in CR of 62:1 in 575 minutes. BLASTing 189,864 Crispr-Cas9 gRNA target sequences (23 MB in total) to the compressed index of human pan-genome (n = 1,000) finished in 45 minutes on a single node. 30 MB mixed bacterial sequences were (n = 599) were blasted to the compressed index of 488 GB GenBank database (n = 13,375,031) in 26 minutes on 25 nodes. 78 MB mixed sequences (n = 4,167) were blasted to the compressed index of 18 GB E. coli sequence database (n = 745,409) in 5.4 minutes on a single node.

Highlights

Fast progress in High-throughput sequencing (HTS) technology has increased the sequencing throughput and decreased the whole-genome sequencing (WGS) price over a thousand-fold during the last 15 years [1]
The results show that the compression ratio increases in proportion to the dictionary size
We design and implement a distributed version of the compressed hybrid indexing method based on Relative Lempel-Ziv (RLZ) compression for scaling pan-genomic read alignment and sequence search to perform on large pan-genomes in practice

Summary

Introduction

Fast progress in High-throughput sequencing (HTS) technology has increased the sequencing throughput and decreased the whole-genome sequencing (WGS) price over a thousand-fold during the last 15 years [1]. At the same time, sequencing data volumes are growing several orders of magnitude, and the number of assembled whole-genomes increases rapidly as well. Storing, indexing, and searching genomic data requires a large amount of high-performance storage space, working memory, computing power, and network capacity. Read alignment and sequence matching are routine methods in many genomic studies. To exploit the accumulating genomic data from genome research to clinical practice [2, 3] the efficient compression and indexing methods on large genomic data sets become an urgent need [4]

Methods

Results

Conclusion