PipeMEM: A Framework to Speed Up BWA-MEM in Spark with Low Overhead.

Lingqi Zhang,Shoubin Dong,Cheng Liu

doi:10.3390/genes10110886

Abstract

(1) Background: DNA sequence alignment process is an essential step in genome analysis. BWA-MEM has been a prevalent single-node tool in genome alignment because of its high speed and accuracy. The exponentially generated genome data requiring a multi-node solution to handle large volumes of data currently remains a challenge. Spark is a ubiquitous big data platform that has been exploited to assist genome alignment in handling this challenge. Nonetheless, existing works that utilize Spark to optimize BWA-MEM suffer from higher overhead. (2) Methods: In this paper, we presented PipeMEM, a framework to accelerate BWA-MEM with lower overhead with the help of the pipe operation in Spark. We additionally proposed to use a pipeline structure and in-memory-computation to accelerate PipeMEM. (3) Results: Our experiments showed that, on paired-end alignment tasks, our framework had low overhead. In a multi-node environment, our framework, on average, was 2.27× faster compared with BWASpark (an alignment tool in Genome Analysis Toolkit (GATK)), and 2.33× faster compared with SparkBWA. (4) Conclusions: PipeMEM could accelerate BWA-MEM in the Spark environment with high performance and low overhead.

Highlights

The development of next-generation sequencing (NGS) technique generates data faster compared to the previous techniques
Uploading data to HDFS is unavoidable for every Spark-based alignment tool
In this sub-section, we proved that the pipeline pattern could accelerate the pre-processing of PipeMEM

Summary

Introduction

The development of next-generation sequencing (NGS) technique generates data faster compared to the previous techniques. The increasing rate of sequencing data is even faster than the Moor Law in computer architecture This situation raised a challenge for genome data analysis, which further put the first step of it, sequence alignment, into the breach of overwhelming data. All of these call for proper alignment methods with high accuracy that can process such a large amount of data effectively. Under this background, many great single-node alignment tools have been developed, for example, BWA-SW [1], BWA-MEM [2], bowtie2 [3], cushaw [4], etc. Numerous genome analysis pipelines deploy it for their alignment step, for example, GenomeScope [5], PhyResSe [6], SpeedSeq [7], GPCG [8], and GATK (genome analysis toolkit) [9]

Methods

Results

Conclusion