General Purpose Compression Algorithms Research Articles

AFC: An adaptive lossless floating-point compression algorithm in time series database

The time series database is a specialized type of database specifically designed for storing and analyzing time series data. Compression of time series data is crucial for its performance. However, efficiently compressing time series data, particularly floating-point data, remains challenging. Existing compression algorithms are efficient for only a limited range of data patterns, indicating a lack of self-adaptation. In this paper, we propose an effective and Adaptive lossless Floating-point Compression algorithm AFC for time series databases. We devise four unique compression strategies, and based on the data patterns, AFC dynamically selects the appropriate strategy. These strategies handle data compression for diverse data patterns, enhancing the compression ratio and efficiency. The most suitable strategy is employed to achieve an optimal compression ratio. We compared our AFC algorithm with four state-of-the-art compression algorithms, namely Gorilla, FPC, TSXor, and Chimp, as well as various general-purpose compression algorithms such as LZ4 and Snappy. Experimental results demonstrate that our algorithm achieves an improvement of at least 20% in compression ratio and even up to 100% on certain datasets.

Chimp

Applications in diverse domains such as astronomy, economics and industrial monitoring, increasingly press the need for analyzing massive collections of time series data. The sheer size of the latter hinders our ability to efficiently store them and also yields significant storage costs. Applying general purpose compression algorithms would effectively reduce the size of the data, at the expense of introducing significant computational overhead. Time Series Management Systems that have emerged to address the challenge of handling this overwhelming amount of information, cannot suffer the ingestion rate restrictions that such compression algorithms would cause. Data points are usually encoded using faster, streaming compression approaches. However, the techniques that contemporary systems use do not fully utilize the compression potential of time series data, with implications in both storage requirements and access times. In this work, we propose a novel streaming compression algorithm, suitable for floating point time series data. We empirically establish properties exhibited by a diverse set of time series and harness these features in our proposed encodings. Our experimental evaluation demonstrates that our approach readily outperforms competing techniques, attaining compression ratios that are competitive with slower general purpose algorithms, and on average around 50% of the space required by state-of-the-art streaming approaches. Moreover, our algorithm outperforms all earlier techniques with regards to both compression and access time , offering a significantly improved trade-off between space and speed. The aforementioned benefits of our approach - in terms of all space requirements, compression time and read access - significantly improve the efficiency in which we can store and analyze time series data.

Second compression for pixelated images under edge-based compression algorithms: JPEG-LS as an example

This paper details the examination of a particular case of data compression, where the compression algorithm removes the redundancy from data, which occurs when edge-based compression algorithms compress (previously compressed) pixelated images. The newly created redundancy can be removed using another round of compression. This work utilized the JPEG-LS as an example of an edge-based compression algorithm for compressing pixelated images. The output of this process was subjected to another round of compression using a more robust but slower compressor (PAQ8f). The compression ratio of the second compression was, on average, 18%, which is high for random data. The results of the second compression were superior to the lossy JPEG. Under the used data set, lossy JPEG needs to sacrifice 10% on average to realize nearly total lossless compression ratios of the two-successive compressions. To generalize the results, fast general-purpose compression algorithms (7z, bz2, and Gzip) were used too.

Multi-threaded compression of Earth observation time-series data

ABSTRACTEarth observation data are typically compressed using general-purpose single-threaded compression algorithms that operate at a fraction of the bandwidth of modern storage and processing systems. We present evidence that recently developed multi-threaded compression codecs offer substantial benefits over widely used single-threaded codecs in terms of compression efficiency when applied to a selection of moderate resolution imaging spectroradiometer (MODIS) datasets stored in the HDF5 format. Compression codecs from the LZ77 and Rice families are shown to vary in efficacy when applied to different MODIS data products, highlighting the need for compression strategies to be tailored to different classes of data. We also introduce LPC-Rice, a new multi-threaded codec, that performs particularly well when applied to time-series data.

Using general-purpose compression algorithms for music analysis

General-purpose compression algorithms encode files as dictionaries of substrings with the positions of these strings’ occurrences. We hypothesized that such algorithms could be used for pattern discovery in music. We compared LZ77, LZ78, Burrows–Wheeler and COSIATEC on classifying folk song melodies. A novel method was used, combining multiple viewpoints, the k-nearest-neighbour algorithm and a novel distance metric, corpus compression distance. Using single viewpoints, COSIATEC outperformed the general-purpose compressors, with a classification success rate of 85% on this task. However, by combining 8 of the 10 best-performing viewpoints, including seven that used LZ77, the classification success rate rose to over 94%. In a second experiment, we compared LZ77 with COSIATEC on the task of discovering subject and countersubject entries in fugues by J.S. Bach. When voice information was absent in the input data, COSIATEC outperformed LZ77 with a mean score of 0.123, compared with 0.053 for LZ77. However, when the music was processed a voice at a time, the score for LZ77 more than doubled to 0.124. We also discovered a significant correlation between compression factor and score for all the algorithms, supporting the hypothesis that the best analyses are those represented by the shortest descriptions.

Resource efficient data compression algorithms for demanding, WSN based biomedical applications

During the last few years, medical research areas of critical importance such as Epilepsy monitoring and study, increasingly utilize wireless sensor network technologies in order to achieve better understanding and significant breakthroughs. However, the limited memory and communication bandwidth offered by WSN platforms comprise a significant shortcoming to such demanding application scenarios. Although, data compression can mitigate such deficiencies there is a lack of objective and comprehensive evaluation of relative approaches and even more on specialized approaches targeting specific demanding applications. The research work presented in this paper focuses on implementing and offering an in-depth experimental study regarding prominent, already existing as well as novel proposed compression algorithms. All algorithms have been implemented in a common Matlab framework. A major contribution of this paper, that differentiates it from similar research efforts, is the employment of real world Electroencephalography (EEG) and Electrocardiography (ECG) datasets comprising the two most demanding Epilepsy modalities. Emphasis is put on WSN applications, thus the respective metrics focus on compression rate and execution latency for the selected datasets. The evaluation results reveal significant performance and behavioral characteristics of the algorithms related to their complexity and the relative negative effect on compression latency as opposed to the increased compression rate. It is noted that the proposed schemes managed to offer considerable advantage especially aiming to achieve the optimum tradeoff between compression rate-latency. Specifically, proposed algorithm managed to combine highly completive level of compression while ensuring minimum latency thus exhibiting real-time capabilities. Additionally, one of the proposed schemes is compared against state-of-the-art general-purpose compression algorithms also exhibiting considerable advantages as far as the compression rate is concerned.

Light-weight reference-based compression of FASTQ data.

BackgroundThe exponential growth of next generation sequencing (NGS) data has posed big challenges to data storage, management and archive. Data compression is one of the effective solutions, where reference-based compression strategies can typically achieve superior compression ratios compared to the ones not relying on any reference.ResultsThis paper presents a lossless light-weight reference-based compression algorithm namely LW-FQZip to compress FASTQ data. The three components of any given input, i.e., metadata, short reads and quality score strings, are first parsed into three data streams in which the redundancy information are identified and eliminated independently. Particularly, well-designed incremental and run-length-limited encoding schemes are utilized to compress the metadata and quality score streams, respectively. To handle the short reads, LW-FQZip uses a novel light-weight mapping model to fast map them against external reference sequence(s) and produce concise alignment results for storage. The three processed data streams are then packed together with some general purpose compression algorithms like LZMA. LW-FQZip was evaluated on eight real-world NGS data sets and achieved compression ratios in the range of 0.111-0.201. This is comparable or superior to other state-of-the-art lossless NGS data compression algorithms.ConclusionsLW-FQZip is a program that enables efficient lossless FASTQ data compression. It contributes to the state of art applications for NGS data storage and transmission. LW-FQZip is freely available online at: http://csse.szu.edu.cn/staff/zhuzx/LWFQZip.

HydroZIP: How Hydrological Knowledge can Be Used to Improve Compression of Hydrological Data

From algorithmic information theory, which connects the information content of a data set to the shortest computer program that can produce it, it is known that there are strong analogies between compression, knowledge, inference and prediction. The more we know about a data generating process, the better we can predict and compress the data. A model that is inferred from data should ideally be a compact description of those data. In theory, this means that hydrological knowledge could be incorporated into compression algorithms to more efficiently compress hydrological data and to outperform general purpose compression algorithms. In this study, we develop such a hydrological data compressor, named HydroZIP, and test in practice whether it can outperform general purpose compression algorithms on hydrological data from 431 river basins from the Model Parameter Estimation Experiment (MOPEX) data set. HydroZIP compresses using temporal dependencies and parametric distributions. Resulting file sizes are interpreted as measures of information content, complexity and model adequacy. These results are discussed to illustrate points related to learning from data, overfitting and model complexity.

BIND – An algorithm for loss-less compression of nucleotide sequence data

Recent advances in DNA sequencing technologies have enabled the current generation of life science researchers to probe deeper into the genomic blueprint. The amount of data generated by these technologies has been increasing exponentially since the last decade. Storage, archival and dissemination of such huge data sets require efficient solutions, both from the hardware as well as software perspective. The present paper describes BIND-an algorithm specialized for compressing nucleotide sequence data. By adopting a unique 'block-length' encoding for representing binary data (as a key step), BIND achieves significant compression gains as compared to the widely used general purpose compression algorithms (gzip, bzip2 and lzma). Moreover, in contrast to implementations of existing specialized genomic compression approaches, the implementation of BIND is enabled to handle non-ATGC and lowercase characters. This makes BIND a loss-less compression approach that is suitable for practical use. More importantly, validation results of BIND (with real-world data sets) indicate reasonable speeds of compression and decompression that can be achieved with minimal processor/ memory usage. BIND is available for download at http://metagenomics.atc.tcs.com/compression/BIND. No license is required for academic or non-profit use.

DELIMINATE—a fast and efficient method for loss-less compression of genomic sequences

An unprecedented quantity of genome sequence data is currently being generated using next-generation sequencing platforms. This has necessitated the development of novel bioinformatics approaches and algorithms that not only facilitate a meaningful analysis of these data but also aid in efficient compression, storage, retrieval and transmission of huge volumes of the generated data. We present a novel compression algorithm (DELIMINATE) that can rapidly compress genomic sequence data in a loss-less fashion. Validation results indicate relatively higher compression efficiency of DELIMINATE when compared with popular general purpose compression algorithms, namely, gzip, bzip2 and lzma. Linux, Windows and Mac implementations (both 32 and 64-bit) of DELIMINATE are freely available for download at: http://metagenomics.atc.tcs.com/compression/DELIMINATE. sharmila@atc.tcs.com Supplementary data are available at Bioinformatics online.

Check image compression using a layered coding method

An emerging trend in the banking industry is to digitize checks for storage and transmission. An immediate requirement for efficient storage and transmission is check image compression. General purpose compression algorithms such as JPEG and wavelet-based methods produce annoying ringing or blocking artifacts at high compression ratios. In this paper, a layered approach to check image compression is proposed in which a check image is represented in several layers. The first layer describes the foreground map; the second layer specifies the gray levels of foreground pixels; the third layer is a lossy representation of the background image; and the fourth layer describes the error between the original and the reconstructed image of the first three layers. The layered coding approach produces images of better quality than traditional JPEG and wavelet coding methods, especially in the foreground, i.e., the text and graphics. In addition, this approach allows progressive retrieval or transmission of different image layers.

Let Sleeping Files Lie: Pattern Matching in Z-Compressed Files

The current explosion of stored information necessitates a new model of pattern matching, that ofcompressed matching. In this model one tries to find all occurrences of a pattern in a compressed text in time proportional to the compressed text size,i.e., without decompressing the text. The most effective general purpose compression algorithms areadaptive, in that the text represented by each compression symbol is determined dynamically by the data. As a result, the encoding of a substring depends on its location. Thus the same substring may “look different” every time it appears in the compressed text. In this paper we consider pattern matching without decompression in the UNIX Z-compression. This is a variant of the Lempel–Ziv adaptive compression scheme. Ifnis the length of thecompressedtext andmis the length of the pattern, our algorithms find the first pattern occurrence in timeO(n+m2) orO(nlogm+m). We also introduce a new criterion to measure compressed matching algorithms, that ofextra space. We show how to modify our algorithms to achieve a trade-off between the amount of extra space used and the algorithm's time complexity.