Burrows-Wheeler Research Articles

BWT construction and search at the terabase scale.

Burrows-Wheeler Transform (BWT) is a common component in full-text indices. Initially developed for data compression, it is particularly powerful for encoding redundant sequences such as pangenome data. However, BWT construction is resource intensive and hard to be parallelized, and many methods for querying large full-text indices only report exact matches or their simple extensions. These limitations have hampered the biological applications of full-text indices. We developed ropebwt3 for efficient BWT construction and query. Ropebwt3 indexed 320 assembled human genomes in 65 hours and indexed 7.3 terabases of commonly studied bacterial assemblies in 26 days. This was achieved using up to 170 gigabytes of memory at the peak without working disk space. Ropebwt3 can find maximal exact matches and inexact alignments under affine-gap penalties, and can retrieve similar local haplotypes matching a query sequence. It demonstrates the feasibility of full-text indexing at the terabase scale. https://github.com/lh3/ropebwt3.

HAlign 4: A New Strategy for Rapidly Aligning Millions of Sequences.

HAlign is a high-performance multiple sequence alignment software based on the star alignment strategy, which is the preferred choice for rapidly aligning large numbers of sequences. HAlign3, implemented in Java, is the latest version capable of aligning an ultra-large number of similar DNA/RNA sequences. However, HAlign3 still struggles with long sequences and extremely large numbers of sequences. To address this issue, we have implemented HAlign4 in C ++. In this version, we replaced the original suffix tree with Burrows-Wheeler Transform (BWT) and introduced the wavefront alignment algorithm to further optimize both time and memory efficiency. Experiments show that HAlign4 significantly outperforms HAlign3 in runtime and memory usage in both single-threaded and multi-threaded configurations, while maintains high alignment accuracy comparable to MAFFT. HAlign4 can complete the alignment of 10 million COVID-19 sequences in about 12 minutes and 300GB of memory using 96 threads, demonstrating its efficiency and practicality for large-scale alignment on standard workstations. Source code is available at https://github.com/malabz/HAlign-4, dataset is available at https://zenodo.org/records/13934503. Supplementary data are available at Bioinformatics online.

Integration of BWT scrambling and data compression in an innovative system enhances protection and versatile management of sensor feeds (SEC)

The proposed Sensor Data Encryption with Compression (SEC) system is presented as an innovative solution for sensor data processing, aiming to achieve optimal efficiency while improving security and adaptability by integrating robust cryptography with advanced compression techniques. The system's objective is to create a reliable framework for managing sensitive sensor data by effectively tackling challenges in data processing through a balanced combination of strong cryptography and sophisticated compression methods. The SEC system utilizes advanced encryption and compression techniques, including the scrambling of Burrows-Wheeler Transform (BWT), Move-to-Front (MTF) encoding, Run-Length Encoding (RLE), and Huffman coding, to attain exceptional compression efficiency while maintaining data integrity. The practical implementation of the SEC system shows an impressive 85 % enhancement in data compression efficiency, as evidenced by reduced Bits per Character (BPC) across various sensor data inputs, along with improved resistance to cryptanalysis due to an increased unicity distance. Consequently, the SEC system demonstrates itself to be a robust and efficient solution, achieving superior data compression and enhanced cryptographic resilience, thereby addressing critical challenges in sensor data processing.

Prokrustean Graph: A substring index for rapid k-mer size analysis.

Despite the widespread adoption of -mer-based methods in bioinformatics, understanding the influence of -mer sizes remains a persistent challenge. Selecting an optimal -mer size or employing multiple -mer sizes is often arbitrary, application-specific, and fraught with computational complexities. Typically, the influence of -mer size is obscured by the outputs of complex bioinformatics tasks, such as genome analysis, comparison, assembly, alignment, and error correction. However, it is frequently overlooked that every method is built above a well-defined -mer-based object like Jaccard Similarity, de Bruijn graphs, -mer spectra, and Bray-Curtis Dissimilarity. Despite these objects offering a clearer perspective on the role of -mer sizes, the dynamics of -mer-based objects with respect to -mer sizes remain surprisingly elusive. This paper introduces a computational framework that generalizes the transition of -mer-based objects across -mer sizes, utilizing a novel substring index, the Prokrustean graph. The primary contribution of this framework is to compute quantities associated with -mer-based objects for all -mer sizes, where the computational complexity depends solely on the number of maximal repeats and is independent of the range of -mer sizes. For example, counting vertices of compacted de Bruijn graphs for can be accomplished in mere seconds with our substring index constructed on a gigabase-sized read set. Additionally, we derive a space-efficient algorithm to extract the Prokrustean graph from the Burrows-Wheeler Transform. It becomes evident that modern substring indices, mostly based on longest common prefixes of suffix arrays, inherently face difficulties at exploring varying -mer sizes due to their limitations at grouping co-occurring substrings. We have implemented four applications that utilize quantities critical in modern pangenomics and metagenomics. The code for these applications and the construction algorithm is available at https://github.com/KoslickiLab/prokrustean.

A survey of BWT variants for string collections.

In recent years, the focus of bioinformatics research has moved from individual sequences to collections of sequences. Given the fundamental role of the Burrows-Wheeler Transform (BWT) in string processing, a number of dedicated tools have been developed for computing the BWT of string collections. While the focus has been on improving efficiency, both in space and time, the exact definition of the BWT employed has not been at the center of attention. As we show in this paper, the different tools in use often compute non-equivalent BWT variants: the resulting transforms can differ from each other significantly, including the number r of runs, a central parameter of the BWT. Moreover, with many tools, the transform depends on the input order of the collection. In other words, on the same dataset, the same tool may output different transforms if the dataset is given in a different order. We studied 18 dedicated tools for computing the BWT of string collections and were able to identify 6 different BWT variants computed by these tools. We review the differences between these BWT variants, both from a theoretical and from a practical point of view, comparing them on 8 real-life biological datasets with different characteristics. We find that the differences can be extensive, depending on the datasets, and are largest on collections of many similar short sequences. The parameter r, the number of runs of the BWT, also shows notable variation between the different BWT variants; on our datasets, it varied by a multiplicative factor of up to 4.2. Source code and scripts to replicate the results and download the data used in the article are available at https://github.com/davidecenzato/BWT-variants-for-string-collections. Supplementary data are available at Bioinformatics online.

A Compression and Encryption Based Heart Disease Diagnosis with Deep Learning through ECG Signals

Electrocardiogram (ECG) monitoring models are commonly employed for diagnosing heart diseases. Since ECG signals are normally acquired for a longer time duration with high resolution, there is a need to compress the ECG signals for transmission and storage. So, a novel compression technique is essential in transmitting the signals to the telemedicine center to monitor and analyses the data. In addition, the protection of ECG signals poses a challenging issue, which encryption techniques can resolve. The existing Encryption-Then-Compression (ETC) models for multimedia data fail to properly maintain the trade- off between compression performance and signal quality. In this view, this study presents a new ETC with a diagnosis model for ECG data, called the ETC-ECG model. The proposed model involves four major processes, namely, pre-processing, encryption, compression, and classification. Once the ECG data of the patient are gathered, Discrete Wavelet Transform (DWT) with a Thresholding mechanism is used for noise removal. In addition, the chaotic map-based encryption technique is applied to encrypt the data. Moreover, the Burrows-Wheeler Transform (BWT) approach is employed for the compression of the encrypted data. Finally, a Deep Neural Network (DNN) is applied to the decrypted data to diagnose heart disease. The detailed experimental analysis takes place to ensure the effective performance of the presented model to assure data security, compression, and classification performance for ECG data

Design and Analysis of Reconfigurable Origami-Based Vacuum Pneumatic Artificial Muscles for Versatile Robotic System.

In this study, a vacuum-based modular actuator system named reconfigurable origami-based vacuum pneumatic artificial muscles (ROV-PAMs) is presented. The system consists of six types of actuating modules and three types of fluidic supporting modules each embedded with magnet-based connectors so that the modules can be assembled to modify the system behavior. The module can be used in a myriad of ways, including extending their working range, creating complex geometries upon deformation, and cooperating to improve overall performance. A simple analytical model for the actuating modules is derived based on the law of conservation of energy, and the model is verified experimentally which shows that this intuitive model can provide a reasonable prediction of performance. A block sorting robot is built using three different types of actuating modules with multiple fluidic supporting modules, and the robot shows that it is possible to flexibly and easily assemble modules to build a robot capable of completing diverse tasks. The ROV-PAM module and its concept can be applied to realize robotic designs, which can be altered on-the-fly to adjust its functionality to meet the evolving requirements required for truly flexible automation.

Constructing and indexing the bijective and extended Burrows–Wheeler transform

The Burrows–Wheeler transform (BWT) is a permutation whose applications are prevalent in data compression and text indexing. The bijective BWT is a bijective variant of it that has not yet been studied for text indexing applications. We fill this gap by proposing a self-index built on the bijective BWT. The self-index applies the backward search technique of the FM-index to find a pattern P with O(|P|lg⁡|P|) backward search steps. Additionally, we propose the first linear-time construction algorithm that is based on SAIS, improving the best known result of O(nlg⁡n/lg⁡lg⁡n) time to linear.

Reversible Data Hiding of JPEG Image Based on Adaptive Frequency Band Length

JPEG images are widely used on the Internet. Histogram shifting reversible data hiding (RDH) methods based on quantized DCT (discrete cosine transform) coefficients are a research focus for JPEG images. Among them, frequency band selection is a key step that affects the performance of JPEG image RDH. In the existing algorithm, frequency band selection is to evaluate the embedding performance of the whole frequency band. But after DCT block sorting, the distribution of expand AC (alternating current) coefficients (Coefficients that can carry data) is in the front of the frequency band, so the performance evaluation of the whole frequency band will produce errors. This paper proposed an adaptive frequency band length JPEG image RDH method, which determines the used length of the frequency band while selecting the frequency band, so that can effectively reduce the invalid shift. Instead of selecting the whole frequency band length in a fixed mode for performance evaluation, the optimal combination of frequency band lengths will be selected according to the embedding performance of different lengths of each frequency band. Then, a united solution mechanism is used to solve the optimal frequency band length, and the frequency band selection is also completed while solving the frequency band length. Experimental results show that our algorithm outperforms existing state-of-the-art methods in terms of marked image visual quality and file size increment.

Reversible data hiding of JPEG images based on block sorting and segmented embedding

Existing JPEG image reversible data hiding (RDH) schemes based on quantized DCT (Discrete cosine transform) modification usually use non-zero or zero AC (Alternating current) coefficients as peak points for data embedding. However, this approach tends to limit the embedding capacity or result in significant file size growth. This paper proposed an RDH scheme for JPEG images based on DCT block sorting and segmented embedding. This scheme can take into account the dual advantages of having a non-zero or zero AC coefficient as the peak point, so it has both high embedding capacity and low file growth, and good visual quality. Firstly, a new DCT block sorting method is designed, the smoothness of the block is estimated by the number of zero coefficients in the EOB (End of Block) and the fluctuation of the DC coefficient in the block, and the zero coefficients in the EOB in the block have not embedded data. Then, the sorted DCT block sequence is divided into three segments, the peak points of the front two segments are (−1, 1) and (−1, 0) respectively, and the third segment may be empty but it always remains unchanged. Finally, the dynamic allocation method is used to quickly solve the optimal segmentation point to achieve the optimal embedding performance. The experimental results show that the overall performance of this scheme is better than the existing state-of-the-art algorithms.

Concurrent encryption and lossless compression using inversion ranks

For secure and efficient transmission or storage, data files are commonly compressed and encrypted. In this work, we introduce a cost-effective encryption method of files as a built-in component of a lossless compression algorithm, thus avoiding the added cost of employing two separate processes. We have shown in earlier studies that preprocessing data with Burrows–Wheeler Transformation followed by Inversion Ranking transformation in advance of the utilization of an entropy coder resulted in an extremely effective general-purpose lossless compression technique. In our proposed algorithm, we isolate the inversion frequency vector obtained from the Inversion Ranking transformation and transmit the compressed file. Since the inversion frequency vector is necessary for decompression, we use it as our secret key and transmit it over a secure channel. Compared to the size of a data file, its frequency vector is rather small. Thus, encrypting only the inversion frequency vector, instead of the entire compressed file, results in substantial savings in computational cost. We demonstrate that the proposed technique is effective and resistant to common attacks on image and audio data sets.

μ- PBWT: a lightweight r-indexing of the PBWT for storing and querying UK Biobank data.

The Positional Burrows-Wheeler Transform (PBWT) is a data structure that indexes haplotype sequences in a manner that enables finding maximal haplotype matches in h sequences containing w variation sites in O(hw) time. This represents a significant improvement over classical quadratic-time approaches. However, the original PBWT data structure does not allow for queries over Biobank panels that consist of several millions of haplotypes, if an index of the haplotypes must be kept entirely in memory. In this article, we leverage the notion of r-index proposed for the BWT to present a memory-efficient method for constructing and storing the run-length encoded PBWT, and computing set maximal matches (SMEMs) queries in haplotype sequences. We implement our method, which we refer to as μ-PBWT, and evaluate it on datasets of 1000 Genome Project and UK Biobank data. Our experiments demonstrate that the μ-PBWT reduces the memory usage up to a factor of 20% compared to the best current PBWT-based indexing. In particular, μ-PBWT produces an index that stores high-coverage whole genome sequencing data of chromosome 20 in about a third of the space of its BCF file. μ-PBWT is an adaptation of techniques for the run-length compressed BWT for the PBWT (RLPBWT) and it is based on keeping in memory only a succinct representation of the RLPBWT that still allows the efficient computation of set maximal matches (SMEMs) over the original panel. Our implementation is open source and available at https://github.com/dlcgold/muPBWT. The binary is available at https://bioconda.github.io/recipes/mupbwt/README.html.

PhyBWT2: phylogeny reconstruction via eBWT positional clustering

BackgroundMolecular phylogenetics studies the evolutionary relationships among the individuals of a population through their biological sequences. It may provide insights about the origin and the evolution of viral diseases, or highlight complex evolutionary trajectories. A key task is inferring phylogenetic trees from any type of sequencing data, including raw short reads. Yet, several tools require pre-processed input data e.g. from complex computational pipelines based on de novo assembly or from mappings against a reference genome. As sequencing technologies keep becoming cheaper, this puts increasing pressure on designing methods that perform analysis directly on their outputs. From this viewpoint, there is a growing interest in alignment-, assembly-, and reference-free methods that could work on several data including raw reads data.ResultsWe present phyBWT2, a newly improved version of phyBWT (Guerrini et al. in 22nd International Workshop on Algorithms in Bioinformatics (WABI) 242:23–12319, 2022). Both of them directly reconstruct phylogenetic trees bypassing both the alignment against a reference genome and de novo assembly. They exploit the combinatorial properties of the extended Burrows-Wheeler Transform (eBWT) and the corresponding eBWT positional clustering framework to detect relevant blocks of the longest shared substrings of varying length (unlike the k-mer-based approaches that need to fix the length k a priori). As a result, they provide novel alignment-, assembly-, and reference-free methods that build partition trees without relying on the pairwise comparison of sequences, thus avoiding to use a distance matrix to infer phylogeny. In addition, phyBWT2 outperforms phyBWT in terms of running time, as the former reconstructs phylogenetic trees step-by-step by considering multiple partitions, instead of just one partition at a time, as previously done by the latter.ConclusionsBased on the results of the experiments on sequencing data, we conclude that our method can produce trees of quality comparable to the benchmark phylogeny by handling datasets of different types (short reads, contigs, or entire genomes). Overall, the experiments confirm the effectiveness of phyBWT2 that improves the performance of its previous version phyBWT, while preserving the accuracy of the results.

A new class of string transformations for compressed text indexing

Introduced about thirty years ago in the field of data compression, the Burrows-Wheeler Transform (BWT) is a string transformation that, besides being a booster of the performance of memoryless compressors, plays a fundamental role in the design of efficient self-indexing compressed data structures. Finding other string transformations with the same remarkable properties of BWT has been a challenge for many researchers for a long time. In this paper, we introduce a whole class of new string transformations, called local orderings-based transformations, which have all the “myriad virtues” of BWT. As a further result, we show that such new string transformations can be used for the construction of the recently introduced r-index, which makes them suitable also for highly repetitive collections. In this context, we consider the problem of finding, for a given string, the BWT variant that minimizes the number of runs in the transformed string.

Reversible data hiding for JPEG images based on block difference model and Laplacian distribution estimation

Reversible data hiding (RDH) for JPEG images has attracted extensive attentions in recent years. As far, a number of related schemes have been proposed, which enhance the embedding performance by improving the strategies such as frequency selection and block sorting to choose appropriate embedding coefficients with the histogram shifting mechanism. Nevertheless, the existing methods do not fully consider the characteristics of DCT coefficients and lack accurate measurement on the embedding distortion. To address the above problems, we propose an improved general framework of RDH for JPEG images based on the Laplacian distribution model. The block difference model is first established for the alternating current (AC) coefficients fitting. Subsequently, the scale parameter is calculated to estimate the embedding efficiency of each AC coefficient, and the coefficients with high embedding efficiency are preferentially selected for embedding. Furthermore, the proposed framework can be combined with other JPEG RDH methods based on quantized DCT coefficient modification. Experimental results demonstrate that better performance in terms of both visual quality and file size preservation can be achieved with the proposed scheme compared with the state-of-the-art RDH schemes for JPEG images.

An efficient and secure compression technique for data protection using burrows-wheeler transform algorithm

Data stored on physical storage devices and transmitted over communication channels often have a lot of redundant information, which can be reduced through compression techniques to conserve space and reduce the time it takes to transmit the data. The need for adequate security measures, such as secret key control in specific techniques, raises concerns about data exposure to potential attacks. Encryption plays a vital role in safeguarding information and maintaining its confidentiality by utilizing a secret key to make the data unreadable and unalterable. The focus of this paper is to tackle the challenge of simultaneously compressing and encrypting data without affecting the efficacy of either process. The authors propose an efficient and secure compression method incorporating a secret key to accomplish this goal. Encoding input data involves scrambling it with a generated key and then transforming it through the Burrows-Wheeler Transform (BWT). Subsequently, the output from the BWT is compressed through both Move-To-Front Transform and Run-Length Encoding. This method blends the cryptographic principles of confusion and diffusion into the compression process, enhancing its performance. The proposed technique is geared towards providing robust encryption and sufficient compression. Experimentation results show that it outperforms other techniques in terms of compression ratio. A security analysis of the technique has determined that it is susceptible to the secret key and plaintext, as measured by the unicity distance. Additionally, the results of the proposed technique showed a significant improvement with a compression ratio close to 90% after passing all the test text files.

Adaptive multi-predictor based reversible data hiding with superpixel irregular block sorting and optimization

ABSTRACT Reversible data hiding (RDH) is a special class of steganography that is capable of recovering the original cover image upon the extraction of the secret data. The main goal of this paper is to develop different adaptive predictors based on superpixel irregular block sorting. Firstly, a superpixel irregular block and sorting strategy is proposed which is applied to histogram shifting for the first time. Then, a multi-directional edge classification method is proposed, which divides pixels into strong edge pixels, normal edge pixels, and weak edge pixels. Moreover, strong edge pixels and normal edge pixels are further divided into four directions. According to edge classification, the most appropriate adaptive multi-predictor is proposed. Finally, an optimization-based data hiding strategy is proposed. The proposed scheme focuses on constructing a sharp enough histogram. The investigational results demonstrate that the proposed scheme achieves large capacity, high image quality, and low complexity.

Two-Cloud Private Read Alignment to a Public Reference Genome

The human genome is the ultimate identifier of an individual even though most of it is identical across human beings. Biological differences between two individuals are encoded in a set of base pair variations called Single Nucleotide Polymorphisms (SNPs), which may be indicative of an individual's personal information such as skin color and susceptibility to diseases. The large-scale nature of human genome necessitates outsourcing of genomic computations to public clouds. However, this raises some serious privacy concerns. The fact that the human reference template is public poses additional challenges. In this paper, we propose a two-cloud private read alignment algorithm using the Burrows-Wheeler Transform and the FM-Index. Our algorithm runs in the same order of complexity as the core FM-Index alignment algorithm without privacy. Our proposed scheme is able to achieve accuracy comparable to modern alignment algorithms such as Bowtie with complete privacy.

Weighted Burrows–Wheeler Compression

A weight-based dynamic compression method has recently been proposed, which is especially suitable for the encoding of files with locally skewed distributions. Its main idea is to assign larger weights to closer to be encoded symbols by means of an increasing weight function, rather than considering each position in the text evenly. A well known transformation that tends to convert input files into files with a more skewed distribution is the Burrows–Wheeler Transform (BWT). This paper proposes to apply the weighted approach on Burrows–Wheeler transformed files. While it is shown that the compression performance is not altered for static and adaptive arithmetic coding by any permutation of the symbols, hence in particular for BWT, empirical evidence of the efficiency of the combination of BWT with the weighted approach is provided.

Novel Block Sorting and Symbol Prediction Algorithm for PDE-Based Lossless Image Compression: A Comparative Study with JPEG and JPEG 2000

In this paper, we present a novel compression method based on partial differential equations complemented by block sorting and symbol prediction. Block sorting is performed using the Burrows–Wheeler transform, while symbol prediction is performed using the context mixing method. With these transformations, the range coder is used as a lossless compression method. The objective and subjective quality evaluation of the reconstructed image illustrates the efficiency of this new compression method and is compared with the current standards, JPEG and JPEG 2000.