Lossless Data Compression Algorithm Research Articles

Performance Survey of Several Lossless Compression Algorithms for Telemetry Applications

This paper presents a tutorial overview of several classes of known algorithms for lossless data compression and compares them based on their efficiency for compressing telemetry data. Performance of the various algorithms are evaluated, based on parameters such as compression ratio and processing time, using different test telemetry data files. The lossless algorithms considered are categorized into the following seven groups: suppression, substitution, bit level, relative, statistical, combination, and prediction, which includes both classical and neural predictors. Among the algorithms considered in this paper, it is found that prediction-based adaptive methods are ideal for real-time applications as they achieve good compression at relatively higher speeds. However, for off-line processing where processing time is not very critical, combination methods such as sixpack and Lempel-Ziv with Arithmetic Coding (LZARI) are better suited as they have the highest compression ratios. A twostage compression method involving a predictor and an encoder can be considered a trade-off between the former two schemes as this method achieves good compression ratios with reasonably low processing times.

Universal lossless data compression with side information by using a conditional MPM grammar transform

A grammar transform is a transformation that converts any data sequence to be compressed into a grammar from which the original data sequence can be fully reconstructed. In a grammar-based code, a data sequence is first converted into a grammar by a grammar transform and then losslessly encoded. Among several previously proposed grammar transforms is the multilevel pattern matching (MPM) grammar transform. In this paper, the MPM grammar transform is first extended to the case of side information known to both the encoder and decoder, yielding a conditional MPM (CMPM) grammar transform. A new simple linear-time and space complexity algorithm is then proposed to implement the MPM and CMPM grammar transforms. Based on the CMPM grammar transform, a universal lossless data compression algorithm with side information is developed, which can achieve asymptotically the conditional entropy rate of any stationary, ergodic source pair. It is shown that the algorithm's worst case redundancy/sample against the k-contest conditional empirical entropy among all individual sequences of length n is upper-bounded by c(1/logn), where c is a constant. The proposed algorithm with side information is the first in the coming family of conditional grammar-based codes, whose expected high efficiency is due to the efficiency of the corresponding unconditional codes.

Renewal theory and source coding

Renewal theory provides a wavy to derive fundamental results about source coding and is useful in the analysis and design of many lossless data compression algorithms. We consider two very different applications of renewal theory to source coding. The first one results in a variable-length counterpart to the asymptotic equipartition property for unifilar Markov sources. The second application leads to the first analysis of variable-to-fixed length codes with plurally parsable dictionaries.

Efficient universal lossless data compression algorithms based on a greedy sequential grammar transform. I. Without context models

Efficient universal lossless data compression algorithms based on a greedy sequential grammar transform. I. Without context models

Improvements to Burrows-Wheeler compression algorithm

In 1994 Burrows and Wheeler presented a new algorithm for lossless data compression. The compression ratio that can be achieved using their algorithm is comparable with the best known other algorithms, whilst its complexity is relatively small. In this paper we explain the internals of this algorithm and discuss its various modifications that have been presented so far. Then we propose new improvements for its effectiveness. They allow us to obtain a compression ratio equal to 2.271 bpc for the Calgary Corpus files, which is the best result in the class of Burrows–Wheeler transform based algorithms. Copyright © 2000 John Wiley & Sons, Ltd.

A low-power CAM design for LZ data compression

Low-power and high-performance data compressors play an increasingly important role in the portable mobile computing and wireless communication markets. Among lossless data compression algorithms for hardware implementation, LZ77 is one of the most widely used. For real-time communication, some hardware LZ compressors/decompressors have been proposed in the past. Content addressable memory (CAM) is widely considered as the most efficient architecture for pattern matching required by the LZ77 compression process. In this paper, we propose a low-power CAM-based LZ77 data compressor. By shutting down the power for unnecessary comparisons between the CAM words and the input symbol, the proposed CAM architecture consumes much lower power than the conventional ones without noticeable performance penalty. Moreover, using the proposed conditional comparison mechanism and the novel CAM cell with the NAND-type matching logic, on average we have close to two orders of improvement on power consumption, i.e., a reduction of more than 98 percent for 8-bit words. Speed is sacrificed if we use the NAND-type matching logic, but the NAND-type logic and the NOR-type logic can be combined to provide the best solution that balances power and delay. Our approach also can be applied to general-purpose CAMs which use the valid bits, so far as the proposed design techniques are adopted.

A fast hardware data compression algorithm and some algorithmic extensions

This paper reports on work at IBM's Austin and Burlington laboratories concerning fast hardware implementations of general-purpose lossless data compression algorithms, particularly for use in enhancing the data capacity of computer storage devices or systems, and transmission data rates for networking or telecommunications channels. The distinctions between lossy and lossless compression and static and adaptive compression techniques are first reviewed. Then, two main classes of adaptive Lempel-Ziv algorithm, now known as LZ1 and LZ2, are introduced. An outline of early work comparing these two types of algorithm is presented, together with some fundamental distinctions which led to the choice and development of an IBM variant of the LZ1 algorithm, ALDC , and its implementation in hardware. The encoding format for ALDC is presented, together with details of IBM's current fast hardware CMOS compression engine designs, based on use of a content-addressable memory (CAM) array. Overall compression results are compared for ALDC and a number of other algorithms, using the CALGARY data compression benchmark file corpus. More recently, work using small hardware preprocessors to enhance the compression of ALDC on other types of data has shown promising results. Two such algorithmic extensions, BLDC and cLDC , are presented, with the results obtained on important data types for which significant improvement over ALDC alone is achieved.

The Ion Mass Imager on the Planet-B spacecraft

The Ion Mass Imager (IMI) is a light-weight ion mass composition instrument for the Japanese Planet-B mission to be launched to Mars in 1998. The objective of the Planet-B mission is to study the Martian environment with emphasis on the upper atmosphere interaction with the solar wind. IMI measures positive ions with energies between 10 eV/q and 35 keV/q and with a mass resolution high enough to resolve the most important ion species (H+, He++, He+, O++, O+, O2+). The instrument has a 360° field-of-view aperture and uses the spacecraft spin to cover almost the full unit sphere to obtain three-dimensional distribution functions every 4 s (half a spacecraft spin period). Particles are energy-filtered by a spherical electrostatic analyzer, and then mass-analysed by the magnetic separation method. The ions hit a microchannel plate assembly with a position sensitive anode divided into 32 mass channels. Together with 16 angular sectors, this system “images” the direction of motion and mass of ions. A pre-acceleration voltage of 0–4000 V is used to select the mass range, e.g., modes optimized for light ions (up to O+) and heavy ions (O+ to charged dust grains). A loss-less data compression algorithm is used in the in-flight processing software to optimize the amount of data that can be returned from Mars.

On the performance of data compression algorithms based upon string matching

Lossless and lossy data compression algorithms based on string matching are considered. In the lossless case, a result of Wyner and Ziv (1989) is extended. In the lossy case, a data compression algorithm based on approximate string matching is analyzed in the following two frameworks: (1) the database and the source together form a Markov chain of finite order; (2) the database and the source are independent with the database coming from a Markov model and the source from a general stationary, ergodic model. In either framework, it is shown that the resulting compression rate converges with probability one to a quantity computable as the infimum of an information theoretic functional over a set of auxiliary random variables; the quantity is strictly greater than the rate distortion function of the source except in some symmetric cases. In particular, this result implies that the lossy algorithm proposed by Steinberg and Gutman (1993) is not optimal, even for memoryless or Markov sources.

Dictionary Compression on the Pram

Parallel algorithms for lossless data compression via dictionary compression using optimal, longest fragment first (LFF), and greedy parsing strategies are described. Dictionary compression removes redundancy by replacing substrings of the input by references to strings stored in a dictionary. Given a static dictionary stored as a suffix tree, we present a CREW PRAM algorithm for optimal compression which runs in O(M + log M log n) time with O(nM2) processors, where it is assumed that M is the maximum length of any dictionary entry. Under the same model, we give an algorithm for LFF compression which runs in O( log 2 n) time with O(n/ log n) processors where it is assumed that the maximum dictionary entry is of length O( log n). We also describe an O(M + log n) time and O(n) processor algorithm for greedy parsing given a static or sliding-window dictionary. For sliding-window compression, a different approach finds the greedy parsing in O( log n) time using (O(nM log M/ log n) processors. Our algorithms are practical in the sense that their analysis elicits small constants.

Error-Resilient Optimal Data Compression

The problem of communication and computation in the presence of errors is difficult, and general solutions can be time consuming and inflexible (particularly when implemented with a prescribed error detection/correction). A reasonable approach is to investigate reliable communication in carefully selected areas of fundamental interest where specific solutions may be more practical than general purpose techniques. In this paper, we study the problem of error-resilient communication and computation in a particularly challenging area, adaptive lossless data compression, where the devastating effect of error propagation is a long-standing open problem that was posed in the papers of Lempel and Ziv in the late 1970s. In fact, the non-error resilience of adaptive data compression has been a practical drawback of its use in many applications. Protocols that require the receiver to request retransmission from the sender when an error is detected can be impractical for many applications where such two-way communication is not possible or is self-defeating (e.g., with data compression, retransmission may be tantamount to losing the data that could have been transmitted in the mean time). In addition, bits of encoded data that are corrupted while data is in storage will in general not be recoverable and may corrupt the entire decompressed file. By error resilience, we mean that even though errors may not be detected, there are strong guarantees that their effects will not propagate. Our main result is a provable error-resilient adaptive lossless data-compression algorithm which nevertheless maintains optimal compression over the usual input distributions (e.g., stationary ergodic sources). We state our result in the context of a more general model that we call dynamic dictionary communication, where a sender and receiver work in a "lock-step" cooperation to maintain identical copies of a dictionary D that is constantly changing. For lossless data compression, the dictionary stores a set of strings that have been seen in the past and data is compressed by sending only indices of strings over the channel. Other applications of our model include robotics (e.g., remote terrain mapping) and computational learning theory.

Lexical Permutation Sorting Algorithm

The Block Sorting Lossless Data Compression Algorithm (BWT) described by Burrows and Wheeler has received considerable attention. Its compression rates are simliar to context-based methods, such as PPM, but at execution speeds closer to Ziv-Lempel techniques. This paper describes the Lexical Permutation Sorting Algorithm (LPSA), its theoretical basis and delineates its relationship to BWT. In particular we describe how BWT can be reduced to LPSA and show how LPSA could give better results than BWT when transmitting permutations.

Stationary Stochastic Processes and Fractal Data Compression

It is shown that the invariant measure of a stationary nonatomic stochastic process yields an iterated function system with probabilities and an associated dynamical system that provide the basis for optimal lossless data compression algorithms. The theory is illustrated for the case of finite-order Markov processes: For a zero-order process, it produces the arithmetic compression method; while for higher order processes it yields dynamical systems, constructed from piecewise affine mappings from the interval [0, 1] into itself, that may be used to store information efficiently. The theory leads to a new geometrical approach to the development of compression algorithms.

On the design and implementation of a lossless data compression and decompression chip

A lossless data compression and decompression (LDCD) algorithm based on the notion of textural substitution has been implemented in silicon using a linear systolic array architecture. This algorithm employs a model in which the encoder and decoder each have a finite amount of memory which is referred to as the dictionary. Compression is achieved by finding matches between the dictionary and the input data stream whereby a substitution is made in the data stream by an index referencing the corresponding dictionary entry. The LDCD system is built using 30 application-specific integrated circuits (ASICs), each containing 126 identical processing elements (PEs) which perform both the encoding and decoding function at clock rates up to 20 MHz. >

A VLSI chip set for high-speed lossless data compression

A VLSI implementation of a lossless data compression algorithm is reported. This is the first implementation of an encoder/decoder chip set that uses the Rice (see JPL Publication 91-1, 1991) algorithm and provides an introduction to the algorithm and a description of the high-performance hardware. The algorithm is adaptive over a aide entropy range. Its performance on several 8-b test images exceeds other techniques employing differential pulse code modulation (DPCM) followed by arithmetic coding, adaptive Huffman coding, and a Lempel-Ziv-Welch (LZW) algorithm. A major feature of the algorithm is that it requires no look-up tables or external RAM. There are only 71000 transistors required to implement the encoder and decoder. Each chip was fabricated in a 1.0- mu m CMOS process and both are only 5 mm on a side. A comparison is made with other hardware realizations. Under laboratory conditions, the encoder compresses at a rate in excess of 50 Msamples/s and the decoder operates at 25 Msamples/s. The current implementation processes quantized data from 4 to 14 b/sample.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>