Suffix Array Construction Research Articles

Fast, parallel, and cache-friendly suffix array construction

PurposeString indexes such as the suffix array (sa) and the closely related longest common prefix (lcp) array are fundamental objects in bioinformatics and have a wide variety of applications. Despite their importance in practice, few scalable parallel algorithms for constructing these are known, and the existing algorithms can be highly non-trivial to implement and parallelize.MethodsIn this paper we present caps-sa, a simple and scalable parallel algorithm for constructing these string indexes inspired by samplesort and utilizing an LCP-informed mergesort. Due to its design, caps-sa has excellent memory-locality and thus incurs fewer cache misses and achieves strong performance on modern multicore systems with deep cache hierarchies.ResultsWe show that despite its simple design, caps-sa outperforms existing state-of-the-art parallel sa and lcp-array construction algorithms on modern hardware. Finally, motivated by applications in modern aligners where the query strings have bounded lengths, we introduce the notion of a bounded-context sa and show that caps-sa can easily be extended to exploit this structure to obtain further speedups. We make our code publicly available at https://github.com/jamshed/CaPS-SA.

Generic Non-recursive Suffix Array Construction

The suffix array is arguably one of the most important data structures in sequence analysis and consequently there is a multitude of suffix sorting algorithms. However, to this date the GSACA algorithm introduced in 2015 is the only known non-recursive linear-time suffix array construction algorithm (SACA). Despite its interesting theoretical properties, there has been little effort in improving GSACA ’s non-competitive real-world performance. There is a super-linear algorithm DSH , which relies on the same sorting principle and is faster than DivSufSort , the fastest SACA for over a decade. The purpose of this article is twofold: We analyse the sorting principle used in GSACA and DSH and exploit its properties to give an optimised linear-time algorithm, and we show that it can be very elegantly used to compute both the original extended Burrows-Wheeler transform ( eBWT ) and a bijective version of the Burrows-Wheeler transform ( BBWT ) in linear time. We call the algorithm “generic,” since it can be used to compute the regular suffix array and the variants used for the BBWT and eBWT . Our suffix array construction algorithm is not only significantly faster than GSACA but also outperforms DivSufSort and DSH . Our BBWT -algorithm is faster than or competitive with all other tested BBWT construction implementations on large or repetitive data, and our eBWT -algorithm is faster than all other programs on data that is not extremely repetitive.

Reference-based genome compression using the longest matched substrings with parallelization consideration

BackgroundA large number of researchers have devoted to accelerating the speed of genome sequencing and reducing the cost of genome sequencing for decades, and they have made great strides in both areas, making it easier for researchers to study and analyze genome data. However, how to efficiently store and transmit the vast amount of genome data generated by high-throughput sequencing technologies has become a challenge for data compression researchers. Therefore, the research of genome data compression algorithms to facilitate the efficient representation of genome data has gradually attracted the attention of these researchers. Meanwhile, considering that the current computing devices have multiple cores, how to make full use of the advantages of the computing devices and improve the efficiency of parallel processing is also an important direction for designing genome compression algorithms.ResultsWe proposed an algorithm (LMSRGC) based on reference genome sequences, which uses the suffix array (SA) and the longest common prefix (LCP) array to find the longest matched substrings (LMS) for the compression of genome data in FASTA format. The proposed algorithm utilizes the characteristics of SA and the LCP array to select all appropriate LMSs between the genome sequence to be compressed and the reference genome sequence and then utilizes LMSs to compress the target genome sequence. To speed up the operation of the algorithm, we use GPUs to parallelize the construction of SA, while using multiple threads to parallelize the creation of the LCP array and the filtering of LMSs.ConclusionsExperiment results demonstrate that our algorithm is competitive with the current state-of-the-art algorithms in compression ratio and compression time.

Tunnel: Parallel-inducing sort for large string analytics

The suffix array is a crucial data structure for efficient string analysis. Over the course of twenty-six years, sequential suffix array construction algorithms have achieved O(n) time complexity and in-place sorting. In this paper, we present the Tunnel algorithm, the first large-scale parallel suffix array construction algorithm with a time complexity of Onp based on the parallel random access machine (PRAM) model. The Tunnel algorithm is built on three key ideas: dividing the problem of size O(n) into p sub-problems of reduced size Onp by replacing long suffixes with shorter prefixes of size at most a constant D; introducing a Tunnel mechanism to efficiently induce the order of a set of suffixes with long common prefixes; developing a strategy to transform a partially ordered suffix set into a total order relation by iteratively applying the Tunnel inducing method. We provide a detailed description of the algorithm, along with a thorough analysis of its time and space complexity, to demonstrate its correctness and efficiency. The proposed Tunnel algorithm exhibits scalable performance, making it suitable for large string analytics on large-scale parallel systems.

A fast algorithm for constructing suffix arrays for DNA alphabets

The continuous improvement of sequencing technologies has been paralleled by the development of efficient algorithms and data structures for sequencing data analysis and processing. Suffix array is one of data structures that are used to construct the Burrows-Wheeler transform (BWT) for long length genomes. Building a suffix array itself is an expensive-resource process since the computations are dominant by sorting suffixes in a lexical order. Most of the suffix array construction algorithms consider the general and integer alphabets without utilizing special cases for fixed-size ones such as DNA alphabets. In this paper, we exploit the nature of four-sized DNA alphabets and utilize their predefined lexicographical ordering in order to construct suffix arrays for genomic data correctly and efficiently. The suffix array construction algorithm for DNA alphabets is evaluated using three real data sets with different lengths ranging from small E-coli genome to long length Homo sapiens GRCh38.p13 chromosomes. For long length genomes, their corresponding sequence is divided into parts (i.e. reads) with a minimum overlap length, the suffix array is computed for each part separately, and finally all partially computed arrays are merged together into a single one. We studied the effects of varying the reads/overlap lengths on the running time of the proposed suffix array construction algorithm and conclude that the minimum overlap length should be equal to the average length of the longest common prefix between the adjacent parts.

An evaluation of fast segmented sorting implementations on GPUs

Many real-world problems require a sorting operation as part of their efficient solution. Some examples of this are real-time plasma diagnostic, image re-ranking, and suffix array construction. These problems usually involve a large amount of data, so their solutions need a particular application of the sorting procedure, consisting of sorting several arrays in matrix rows or array segments, an operation called segmented sorting. Previous studies showed that a merge sort-based strategy and a strategy called fix sort executed this operation on GPUs with good performance for different array sizes. In this work, we compare the fastest segmented sorting GPU implementations on seven different GPU models with various input data scenarios, including scenarios with varying numbers of segments, segment sizes, and considering segments of the same and different sizes. We first performed algorithm analysis to explain how the number of segments affects each implementation’s performance. Then, we perform an S-curve analysis and observe that, even though each strategy might be the fastest option for a subset of the sorting scenarios, some approaches may cause very high slowdowns on specific scenarios. We also compare the strategies using heat maps, show that their performance depends on the array size and number of segments, and propose a recommendation map to support selecting the best overall implementation based on the size and number of segments. Our experimental results show that choosing a strategy based on our recommendation map leads to the best strategy on 47.57% of the cases and a maximum slowdown of less than 1.5 times in 93.58% of the cases. Moreover, on average, the recommended strategy is only 1.11× worse than the optimum one.Finally, we evaluated how each strategy behaves when sorting arrays with different and equal segment sizes and showed that the fix sort-based approaches take roughly the same time to sort arrays with equal or different segment sizes, while the approach proposed by Hou et al. usually takes longer to sort arrays with different segment sizes than arrays with equal segment sizes.

Optimal in-place suffix sorting

The suffix array is a fundamental data structure for many applications that involve string searching and data compression. We obtain the first in-place suffix array construction algorithms that are optimal both in time and space for (read-only) integer alphabets. We make the following contributions:1.For integer alphabets, we obtain the first linear time suffix sorting algorithm which uses only O(1) workspace. The input string may be modified during the execution of the algorithm, but should be restored upon termination of the algorithm.2.We strengthen the first result by providing the first in-place linear time algorithm for read-only integer alphabets with |Σ|=O(n) (i.e., the input string cannot be modified). This algorithm settles the open problem posed by Franceschini and Muthukrishnan in ICALP 2007.3.Besides, for the read-only general alphabets (i.e., only comparisons are allowed), we present an optimal in-place O(nlog⁡n) time suffix sorting algorithm.

Full-text search engine with suffix index for massive heterogeneous data

Existing popular search engines like Elasticsearch (ES) commonly use inverted indices to quickly retrieve source data matching a given set of queries. However, an inverted index may not find all of the matching results from data, particularly those that are hard to be segmented into words, such as data logs and scientific signals. This article presents our innovative technique for a true full-text search system called SAES by replacing the inverted index in ES with the suffix index to guarantee a 100% recall ratio. We designed a distributed suffix index scheme with online building and offline merging capable of scaling with the architecture of ES. The suffix index is dynamically constructed by several suffix array construction tools which adapt to the data size and available computing resources such as CPU cores, RAM, and disk capacities. Furthermore, it can be compacted to provide a trade-off between searching speed and index storage space. An experimental study was conducted to test the functions and performance of single- and multi-node SAES on realistic datasets of texts, logs, genomes, and signals. The systems performed well for both exact and approximate search queries defined on units of bytes or half-bytes. This work provides a feasible reference design for extending ES with suffix index to support true full-text searches over massive heterogeneous data.

Universal reconstruction of a string

Many properties of a string can be viewed as sets of dependencies between substrings of the string expressed in terms of substring equality. We design a linear-time algorithm which finds a solution to an arbitrary system of such constraints: a generic string satisfying a system of substring equations. This provides a general tool for reconstructing a string from different kinds of repetitions or symmetries present in the string, in particular, from runs or from maximal palindromes. The recursive structure of our algorithm in some aspects resembles the suffix array construction by Kärkkäinen et al. (2006) [23].This is a full version of a paper presented at WADS 2015 [18].

Extended suffix array construction using Lyndon factors

In this paper, we are extending the novel approach of incremental suffix array construction using Lyndon factorization to the construction of extended suffix array where extended suffix array is the suffix array along with the corresponding longest common prefix (LCP) array. Main motive behind the incremental and simultaneous construction of suffix array and LCP array is that both involve in calculating the order information by considering the common prefixes of the suffixes. As local suffixes once sorted have the same sorted order when these are merged with sorted suffixes of another Lyndon factor. So, merging of Lyndon factors is simply merging of two sorted lists of suffixes of these Lyndon factors. Also, the two sorted orders coincide thus making the merging of Lyndon factors a simple merging of two sorted lists of suffixes. Incremental LCP construction simultaneously saves a lot of computation and hence time. The proposed approach has quadratic run time and the disk working space requirement is O(n). Experiments also show the performance gain of our approach in terms of time over the existing method of incremental construction.

Generalized enhanced suffix array construction in external memory

BackgroundSuffix arrays, augmented by additional data structures, allow solving efficiently many string processing problems. The external memory construction of the generalized suffix array for a string collection is a fundamental task when the size of the input collection or the data structure exceeds the available internal memory.ResultsIn this article we present and analyze mathsf {eGSA} [introduced in CPM (External memory generalized suffix and mathsf {LCP} arrays construction. In: Proceedings of CPM. pp 201–10, 2013)], the first external memory algorithm to construct generalized suffix arrays augmented with the longest common prefix array for a string collection. Our algorithm relies on a combination of buffers, induced sorting and a heap to avoid direct string comparisons. We performed experiments that covered different aspects of our algorithm, including running time, efficiency, external memory access, internal phases and the influence of different optimization strategies. On real datasets of size up to 24 GB and using 2 GB of internal memory, mathsf {eGSA} showed a competitive performance when compared to mathsf {eSAIS} and mathsf {SAscan}, which are efficient algorithms for a single string according to the related literature. We also show the effect of disk caching managed by the operating system on our algorithm.ConclusionsThe proposed algorithm was validated through performance tests using real datasets from different domains, in various combinations, and showed a competitive performance. Our algorithm can also construct the generalized Burrows-Wheeler transform of a string collection with no additional cost except by the output time.

Inducing enhanced suffix arrays for string collections

Constructing the suffix array for a string collection is an important task that may be performed by sorting the concatenation of all strings. In this article we present algorithms gSAIS and gSACA-K, which extend SAIS (Nong et al., 2011) [8] and SACA-K (Nong, 2013) [10] to construct the suffix array for a string collection maintaining their theoretical bounds, respecting the order among all suffixes, and improving their practical performance. gSACA-K is an optimal suffix array construction algorithm for string collections from constant alphabets. Moreover, we show how to modify gSAIS and gSACA-K to also compute the longest common prefix array and the document array as a byproduct of suffix sorting, with the same theoretical bounds. We performed experiments that showed that our algorithms have an outstanding practical performance.

Parallel lightweight wavelet tree, suffix array and FM-index construction

We present parallel lightweight algorithms to construct wavelet trees, rank and select structures, and suffix arrays in a shared-memory setting. The work and depth of our first parallel wavelet tree algorithm match those of the best existing parallel algorithm while requiring asymptotically less memory and our second algorithm achieves the same asymptotic bounds for small alphabet sizes. Our experiments show that they are both faster and more memory-efficient than existing parallel algorithms. We also present an experimental evaluation of the parallel construction of rank and select structures, which are used in wavelet trees. Next, we design the first parallel suffix array algorithm based on induced copying. Our induced copying requires linear work and polylogarithmic depth for constant alphabets sizes. When combined with a parallel prefix doubling algorithm, it is more efficient in practice both in terms of running time and memory usage compared to existing parallel implementations. As an application, we combine our algorithms to build the FM-index in parallel.

Engineering a Lightweight External Memory Suffix Array Construction Algorithm

We describe an external memory suffix array construction algorithm based on constructing suffix arrays for blocks of text and merging them into the full suffix array. The basic idea goes back over 20 years and there has been a couple of later improvements, but we describe several further improvements that make the algorithm much faster. In particular, we reduce the I/O volume of the algorithm by a factor \(\mathcal {O}\!\left( {\log _\sigma n} \right) \). Our experiments show that the algorithm is the fastest suffix array construction algorithm when the size of the text is within a factor of about five from the size of the RAM in either direction, which is a common situation in practice.

Optimal suffix sorting and LCP array construction for constant alphabets

We show how the longest common prefix (LCP) array can be generated as a by-product of the suffix array construction algorithm SACA-K (Nong, 2013). Our algorithm builds on Fischer's proposal (Fischer, WADS'11), and also runs in linear time, but uses only constant extra memory for constant alphabets.

Inducing Suffix and LCP Arrays in External Memory

We consider full text index construction in external memory (EM). Our first contribution is an inducing algorithm for suffix arrays in external memory, which runs in sorting complexity. Practical tests show that this algorithm outperforms the previous best EM suffix sorter [Dementiev et al., JEA 2008] by a factor of about two in time and I/O volume. Our second contribution is to augment the first algorithm to also construct the array of longest common prefixes (LCPs). This yields a new internal memory LCP array construction algorithm and the first EM construction algorithm for LCP arrays. The overhead in time and I/O volume for this extended algorithm over plain suffix array construction is roughly two. Our algorithms scale far beyond problem sizes previously considered in the literature (text size of 80GiB using only 4GiB of RAM in our experiments).

Special issue: Euro‐Par 2015

This special issue of Concurrency and Computation: Practice and Experience contains revised and extended versions of selected papers presented at the conference Euro-Par 2015. Euro-Par—the European Conference on Parallel Computing—is an annual series of international conferences dedicated to the promotion and advancement of all aspects of parallel and distributed computing. Euro-Par covers a wide spectrum of topics from algorithms and theory to software technology and hardware-related issues, with application areas ranging from scientific to mobile and cloud computing. The major part of the Euro-Par audience consists of researchers in academic institutions, government laboratories and industrial organisations. Euro-Par 2015, the 21st conference in the Euro-Par series, was held in Vienna, Austria. It was organised by the Research Group for Parallel Computing of the Vienna University of Technology (TU Wien). Thirteen broad topics were defined and advertised, covering a large variety of aspects of parallel and distributed computing. The call for papers attracted a total of 190 submissions. The submitted papers were reviewed at least three and, in most cases, four or even more times (four reviews on average). A total of 51 papers were finally accepted for publication. This makes a global acceptance rate of 27%. The authors of accepted papers came from 21 countries, with the four main contributing countries—the United States, France, Spain and Germany—accounting for a bit more than half of them. Based on the results of the reviews and a majority opinion of the respective topic programme committees, a number of papers were recommended for this special issue. The authors were contacted at the conference and invited to submit revised and extended versions of their papers. These new versions were reviewed independently by three reviewers; two had previously reviewed the conference version, the third had not. Eventually, four papers were accepted for publication. This year, two Euro-Par topics are represented—both covering methods of programming modern computer architectures. Topic 13 on Accelerator Computing is represented with three papers. The paper Performance optimization of sparse matrix-vector multiplication for multi-component PDE-based applications using GPUs, authored by Ahmad Abdelfattah, Hatem Ltaief, David Keyes and Jack Dongarra 1, describes the implementation of a single-GPU and multi-GPU kernel for block-sparse matrix-vector multiplication, a problem that appears in the discretisation of partial differential equations with many dependent variables. The performance of the kernel is measured on a subset of the Florida Sparse Matrix Collection. Especially noted by the reviewers was the uniform interface that applies to a wide range of problem sizes via tunable parameters. This makes it perform efficiently on a wide range of GPU architectures running CUDA. The paper Fast parallel skew and prefix-doubling suffix array construction on the GPU, authored by Leyuan Wang, Sean Baxter and John D. Owens 2, proposes a hybrid GPU implementation of known algorithms for constructing suffix arrays of a string that fits the given GPU architecture best. One highlight pointed out in the reviews is a highly efficient segmented sorting primitive, which is also valuable as independent result. The paper Performance and portability of accelerated lattice Boltzmann applications with OpenACC, authored by Enrico Calore, Jiri Kraus, Sebastiano Fabio Schifano and Raffaele Tripiccione 3, reports on a performance study based on a simple performance model of an OpenACC-based lattice Boltzmann implementation on three different architectures: an NVIDIA GPU, an AMD GPU and a multi-core CPU. The practical relevance of this work was particularly appreciated. Topic 09 on Multi- and Many-Core Programming is represented by the paper Continuous skyline queries on multicore architectures, authored by Tiziano De Matteis, Salvatore Di Girolamo and Gabriele Mencagli 4. Skyline queries result in all tuples whose attribute vector is not dominated (in the Pareto sense) by any other tuple. The authors provide a thorough description and experimental evaluation of a parallelization of the ‘eager’ algorithm for skyline computation of points in d-dimensional space in continuous data streams with expiration times defined by a sliding constant-sized time window. It was judged as a very nice case study of the use of FastFlow, a framework for high-level, pattern-based parallel programming in C++. Concluding this preface, we would like to thank Prof. Geoffrey Fox, editor-in-chief of Concurrency and Computation: Practice and Experience, for his support of this special issue. We would also like to thank our peers who assisted us in reviewing the papers and helped strengthen the final versions. Last, but not least, we also appreciate the support of Springer, who agreed to the publication of the extended versions of the articles that appeared originally in the Lecture Notes in Computer Science.

Fast parallel skew and prefix‐doubling suffix array construction on the GPU

SummarySuffix arrays are fundamental full‐text index data structures of importance to a broad spectrum of applications in such fields as bioinformatics, Burrows–Wheeler transform‐based lossless data compression, and information retrieval. In this work, we propose and implement two massively parallel approaches on the graphics processing unit (GPU) based on two classes of suffix array construction algorithms. The first, parallel skew, makes algorithmic improvements to the previous work of Deo and Keely to achieve a speedup of 1.45x over their work. The second, a hybrid skew and prefix‐doubling implementation, is the first of its kind on the GPU and achieves a speedup of 2.3–4.4x over Osipov's prefix‐doubling and 2.4–7.9x over our skew implementation on large datasets. Our implementations rely on two efficient parallel primitives, a merge and a segmented sort. We theoretically analyze the two formulations of suffix array construction algorithms and show performance comparisons on a large variety of practical inputs. We conclude that, with the novel use of our efficient segmented sort, prefix‐doubling is more competitive than skew on the GPU. We also demonstrate the effectiveness of our methods in our implementations of the Burrows‐Wheeler transform and in a parallel full‐text, minute‐space‐index for pattern searching. Copyright © 2016 John Wiley & Sons, Ltd.

Parallel and Space-Efficient Construction of Burrows-Wheeler Transform and Suffix Array for Big Genome Data.

Next-generation sequencing technologies have led to the sequencing of more and more genomes, propelling related research into the era of big data. In this paper, we present ParaBWT, a parallelized Burrows-Wheeler transform (BWT) and suffix array construction algorithm for big genome data. In ParaBWT, we have investigated a progressive construction approach to constructing the BWT of single genome sequences in linear space complexity, but with a small constant factor. This approach has been further parallelized using multi-threading based on a master-slave coprocessing model. After gaining the BWT, the suffix array is constructed in a memory-efficient manner. The performance of ParaBWT has been evaluated using two sequences generated from two human genome assemblies: the Ensembl Homo sapiens assembly and the human reference genome. Our performance comparison to FMD-index and Bwt-disk reveals that on 12 CPU cores, ParaBWT runs up to 2.2× faster than FMD-index and up to 99.0× faster than Bwt-disk. BWT construction algorithms for very long genomic sequences are time consuming and (due to their incremental nature) inherently difficult to parallelize. Thus, their parallelization is challenging and even relatively small speedups like the ones of our method over FMD-index are of high importance to research. ParaBWT is written in C++, and is freely available at http://parabwt.sourceforge.net.

LCP Array Construction in External Memory

One of the most important data structures for string processing—the suffix array—needs to be augmented with the longest-common-prefix (LCP) array in numerous applications. We describe the first external memory algorithm for constructing the LCP array given the suffix array as input. The only previous way to compute the LCP array for data that is bigger than the RAM is to use an external memory suffix array construction algorithm (SACA) with complex modifications to produce the LCP array as a by-product. Compared to the best prior method, our algorithm needs much less disk space (by more than a factor of three) and is significantly faster. Furthermore, our algorithm can be combined with any SACA, including a better one developed in the future.