String Compression Research Articles

A New Case for Image Compression Using Logic Function Minimization

Sum of minterms is a canonical form for representing logic functions. There are classical methods such as Karnaugh map or Quine–McCluskey tabulation for minimizing a sum of products. This minimization reduces the minterms to smaller products called implicants. If minterms are represented by bit strings, the bit strings shrink through the minimization process. This can be considered as a kind of data compression provided that there is a way for retrieving the original bit strings from the compressed strings. This paper proposes implements and evaluates an image compression method called YALMIC (Yet Another Logic Minimization Based Image Compression) which depends on logic function minimization. This method considers adjacent pixels of the image as disjointed minterms constructing a logic function and compresses the 24-bit color images through minimizing the function. We compare the compression ratio of the proposed method to those of existing methods and show that YALMIC improves the compression ratio by about 25% on average.

Chaitin's halting probability and the compression of strings using oracles

If a computer is given access to an oracle—the characteristic function of a set whose membership relation may or may not be algorithmically calculable—this may dramatically affect its ability to compress information and to determine structure in strings, which might otherwise appear random. This leads to the basic question, ‘given an oracle A , how many oracles can compress information at most as well as A ?’ This question can be formalized using Kolmogorov complexity. We say that B ≤ LK A if there exists a constant c such that K A ( σ )< K B ( σ )+ c for all strings σ , where K X denotes the prefix-free Kolmogorov complexity relative to oracle X . The formal counterpart to the previous question now is, ‘what is the cardinality of the set of ≤ LK -predecessors of A ?’ We completely determine the number of oracles that compress at most as well as any given oracle A , by answering a question of Miller ( Notre Dame J. Formal Logic , 2010, 50 , 381–391), which also appears in Nies ( Computability and randomness . Oxford, UK: Oxford University Press, 2009. Problem 8.1.13); the class of ≤ LK -predecessors of a set A is countable if and only if Chaitin's halting probability Ω is Martin-Löf random relative to A .

COMPRESSED MEMBERSHIP PROBLEMS FOR REGULAR EXPRESSIONS AND HIERARCHICAL AUTOMATA

Membership problems for compressed strings in regular languages are investigated. Strings are represented by straight-line programs, i.e., context-free grammars that generate exactly one string. For the representation of regular languages, various formalisms with different degrees of succinctness (e.g., suitably extended regular expressions, hierarchical automata) are considered. Precise complexity bounds are derived. Among other results, it is shown that the compressed membership problem for regular expressions with intersection is PSPACE-complete. This solves an open problem of Plandowski and Rytter.

Move-to-Front, Distance Coding, and Inversion Frequencies revisited

Move-to-Front, Distance Coding and Inversion Frequencies are three simple and effective techniques used to process the output of the Burrows–Wheeler Transform. In this paper we provide the first complete comparative analyses of these techniques, establishing upper and lower bounds on their compression ratios. We describe simple variants of these three techniques that compress any string up to a constant factor of its k th-order empirical entropy for any k ≥ 0 . At the same time we prove lower bounds for the compression of arbitrary strings which show these variants to be nearly optimal. The bounds we establish are “entropy-only” bounds in the sense that they do not involve non-constant overheads. Our analyses provide new insights into the inner workings of these techniques, partially explain their good behavior in practice, and suggest strategies for improving their performance.

Why greed works for shortest common superstring problem

The shortest common superstring problem (SCS) has been extensively studied for its applications in string compression and DNA sequence assembly. Although the problem is known to be Max-SNP hard, the simple greedy algorithm performs extremely well in practice. To explain the good performance, previous researchers proved that the greedy algorithm is asymptotically optimal on random instances. Unfortunately, the practical instances in DNA sequence assembly are very different from the random instances. In this paper we explain the good performance of the greedy algorithm by using the smoothed analysis. We show that, for any given instance I of SCS, the average approximation ratio of the greedy algorithm on a small random perturbation of I is 1 + o ( 1 ) . The perturbation defined in the paper is small and naturally represents the mutations of the DNA sequence during evolution. Due to the existence of the uncertain nucleotides in the output of a DNA sequencing machine, we also proposed the shortest common superstring with wildcards problem (SCSW). We prove that in the worst case SCSW cannot be approximated within the ratio n 1 / 7 − ϵ , while the greedy algorithm still has 1 + o ( 1 ) smoothed approximation ratio.

Faster subsequence recognition in compressed strings

Computation on compressed strings is one of the key approaches to processing massive data sets. We consider local subsequence recognition problems on strings compressed by straight-line programs (SLP), which is closely related to Lempel–Ziv compression. For an SLP-compressed text of length $\overline{m}$ , and an uncompressed pattern of length n, Cegielski et al. gave an algorithm for local subsequence recognition running in time $O\left( {\overline{m} n^2 \log n}\right)$. We improve the running time to $O\left( {\overline{m} n^{1.5} }\right)$. Our algorithm can also be used to compute the longest common subsequence between a compressed text and an uncompressed pattern in time $O\left( {\overline{m} n^{1.5} }\right)$; the same problem with a compressed pattern is known to be NP-hard.

Lempel–Ziv Factorization Using Less Time & Space

For 30 years the Lempel–Ziv factorization LZ x of a string x = x[1..n] has been a fundamental data structure of string processing, especially valuable for string compression and for computing all the repetitions (runs) in x. Traditionally the standard method for computing LZ x was based on Θ(n)-time (or, depending on the measure used, O(n log n)-time) processing of the suffix tree ST x of x. Recently Abouelhoda et al. proposed an efficient Lempel–Ziv factorization algorithm based on an “enhanced” suffix array – that is, a suffix array SA x together with supporting data structures, principally an “interval tree”. In this paper we introduce a collection of fast space-efficient algorithms for LZ factorization, also based on suffix arrays, that in theory as well as in many practical circumstances are superior to those previously proposed; one family out of this collection achieves true Θ(n)-time alphabet-independent processing in the worst case by avoiding tree structures altogether.

REDUCING SIMPLE GRAMMARS: EXPONENTIAL AGAINST HIGHLY-POLYNOMIAL TIME IN PRACTICE

Simple grammar reduction is an important component in the implementation of Concatenation State Machines (a hardware version of stateless push-down automata designed for wire-speed network packet classification). We present a comparison and experimental analysis of the best-known algorithms for grammar reduction. There are two approaches to this problem: one processing compressed strings without decompression and another one which processes strings explicitly. It turns out that the second approach is more efficient in the considered practical scenario despite having worst-case exponential time complexity (while the first one is polynomial). The study has been conducted in the context of network packet classification, where simple grammars are used for representing the classification policies.

Using sequence compression to speedup probabilistic profile matching

Matching a biological sequence against a probabilistic pattern (or profile) is a common task in computational biology. A probabilistic profile, represented as a scoring matrix, is more suitable than a deterministic pattern to retain the peculiarities of a given segment of a family of biological sequences. Brute-force algorithms take O(NP) to match a sequence of N characters against a profile of length P << N. In this work, we exploit string compression techniques to speedup brute-force profile matching. We present two algorithms, based on run-length and LZ78 encodings, that reduce computational complexity by the compression factor of the encoding.

Algorithmic Clustering of Music Based on String Compression

December 01 2004 Algorithmic Clustering of Music Based on String Compression Rudi Cilibrasi, Rudi Cilibrasi Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Rudi.Cilibrasi@cwi.nl Search for other works by this author on: This Site Google Scholar Paul VitÁnyi, Paul VitÁnyi Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Paul.Vitanyi@cwi.nlInstitute for Logic, Language, and Computation, University of Amsterdam, Plantage Muidergracht 24, 1018 TV Amsterdam, The Netherlands Search for other works by this author on: This Site Google Scholar Ronald de Wolf Ronald de Wolf Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Ronald.de.Wolf@cwi.nl Search for other works by this author on: This Site Google Scholar Author and Article Information Rudi Cilibrasi Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Rudi.Cilibrasi@cwi.nl Paul VitÁnyi Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Paul.Vitanyi@cwi.nlInstitute for Logic, Language, and Computation, University of Amsterdam, Plantage Muidergracht 24, 1018 TV Amsterdam, The Netherlands Ronald de Wolf Centrum voor Wiskunde en Informatica, Kruislaan 413, 1098 SJ Amsterdam, The Netherlands, Ronald.de.Wolf@cwi.nl Online ISSN: 1531-5169 Print ISSN: 0148-9267 © 2004 Massachusetts Institute of Technology2004 Computer Music Journal (2004) 28 (4): 49–67. https://doi.org/10.1162/0148926042728449 Cite Icon Cite Permissions Share Icon Share Facebook Twitter LinkedIn MailTo Views Icon Views Article contents Figures & tables Video Audio Supplementary Data Peer Review Search Site Citation Rudi Cilibrasi, Paul VitÁnyi, Ronald de Wolf; Algorithmic Clustering of Music Based on String Compression. Computer Music Journal 2004; 28 (4): 49–67. doi: https://doi.org/10.1162/0148926042728449 Download citation file: Ris (Zotero) Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentAll JournalsComputer Music Journal Search Advanced Search This content is only available as a PDF. © 2004 Massachusetts Institute of Technology2004 Article PDF first page preview Close Modal You do not currently have access to this content.

A lower bound on compression of unknown alphabets

Many applications call for universal compression of strings over large, possibly infinite, alphabets. However, it has long been known that the resulting redundancy is infinite even for i.i.d. distributions. It was recently shown that the redundancy of the strings’ patterns, which abstract the values of the symbols, retaining only their relative precedence, is sublinear in the blocklength n, hence the per-symbol redundancy diminishes to zero. In this paper we show that pattern redundancy is at least ( 1.5 log 2 e ) n 1 / 3 bits. To do so, we construct a generating function whose coefficients lower bound the redundancy, and use Hayman's saddle-point approximation technique to determine the coefficients’ asymptotic behavior.

Compactly representing parallel program executions

Collecting a program's execution profile is important for many reasons: code optimization, memory layout, program debugging and program comprehension. Path based execution profiles are more detailed than count based execution profiles, since they present the order of execution of the various blocks in a program: modules, procedures, basic blocks etc. Recently, online string compression techniques have been employed for collecting compact representations of sequential program executions. In this paper, we show how a similar approach can be taken for shared memory parallel programs. Our compaction scheme yields one to two orders of magnitude compression compared to the uncompressed parallel program trace on some of the SPLASH benchmarks. Our compressed execution traces contain detailed information about synchronization and control/data flow which can be exploited for post-mortem analysis. In particular, information in our compact execution traces are useful for accurate data race detection (detecting unsynchronized shared variable accesses that occurred in the execution).

Optimal encoding of non-stationary sources

The usual assumption for proofs of the optimality of lossless encoding is a stationary ergodic source. Dynamic sources with non-stationary probability distributions occur in many practical situations where the data source is formed from a composition of distinct sources, for example, a document with multiple authors, a multimedia document, or the composition of distinct packets sent over a communication channel. There is a vast literature of adaptive methods used to tailor the compression to dynamic sources. However, little is known about optimal or near optimal methods for lossless compression of strings generated by sources that are not stationary ergodic. Here, we do not assume the source is stationary. Instead, we assume that the source produces an infinite sequence of concatenated finite strings s1…sn, where: (i) Each finite string si is generated by a sampling of a (possibly distinct) stationary ergodic source Si, and (ii) the length of each of the si is lower bounded by a function L(n) such that L(n)/log(n) grows unboundedly with the length n of all the text within s1…si. Thus each input string is a sequence of substrings generated by possibly distinct and unknown stationary ergodic sources. The optimal expected length of a compressed coding of a finite prefix s1…sk is ∑i=1kniHi, where ni is the length of si and Hi is the entropy of Si. We give a window-based LZ77-type method for compression that we prove gives an encoding with asymptotically optimal expected length. We give another LZ77-type method for compression where the expected time for encoding and decoding is nearly linear (approaching arbitrarily close to linear O(n) for large n). We also prove that this later method gives an encoding with asymptotically optimal expected length. In addition, give a dictionary-based LZ78-type method for compression, which takes linear time with small constant factors. This final algorithm also gives an encoding with asymptotically optimal expected length, assuming the Si are stationary ergodic sources that satisfy certain mixing conditions and L(n)⩾ne for some e>0.

Query optimization in compressed database systems

Over the last decades, improvements in CPU speed have outpaced improvements in main memory and disk access rates by orders of magnitude, enabling the use of data compression techniques to improve the performance of database systems. Previous work describes the benefits of compression for numerical attributes, where data is stored in compressed format on disk. Despite the abundance of string-valued attributes in relational schemas there is little work on compression for string attributes in a database context. Moreover, none of the previous work suitably addresses the role of the query optimizer: During query execution, data is either eagerly decompressed when it is read into main memory, or data lazily stays compressed in main memory and is decompressed on demand only In this paper, we present an effective approach for database compression based on lightweight, attribute-level compression techniques. We propose a IIierarchical Dictionary Encoding strategy that intelligently selects the most effective compression method for string-valued attributes. We show that eager and lazy decompression strategies produce sub-optimal plans for queries involving compressed string attributes. We then formalize the problem of compression-aware query optimization and propose one provably optimal and two fast heuristic algorithms for selecting a query plan for relational schemas with compressed attributes; our algorithms can easily be integrated into existing cost-based query optimizers. Experiments using TPC-H data demonstrate the impact of our string compression methods and show the importance of compression-aware query optimization. Our approach results in up to an order speed up over existing approaches.

Compressibility and Resource Bounded Measure

We give a new definition of resource bounded measure based on compressibility of infinite binary strings. We prove that the new definition is equivalent to the one commonly used. This new characterization offers us a different way to look at resource bounded measure, shedding more light on the meaning of measure zero results and providing one more tool to prove such results. The main contribution of the paper is the new definition and the proofs leading to the equivalence result. We then show how this new characterization can be used to prove that the class of linear autoreducible sets has p- measure 0. We also prove that the class of sets that are truth-table reducible to a p-selective set has p-measure 0 and that the class of sets that Turing reduce to a subpolynomial dense set has p-measure 0. This strengthens various results.

Compressed and fully compressed pattern matching in one and two dimensions

We survey the complexity issues related to several algorithmic problems for compressed and fully compressed pattern matching in one- and two-dimensional texts without explicit decompression. Several related problems for compressed strings and arrays are considered: equality testing, computation of regularities, subsegment extraction, language membership, and solvability of word equations.

String Matching in Lempel—Ziv Compressed Strings

String matching and compression are two widely studied areas of computer science. The theory of string matching has a long association with compression algorithms. Data structures from string matching can be used to derive fast implementations of many important compression schemes, most notably the Lempel—Ziv (LZ77) algorithm. Intuitively, once a string has been compressed—and therefore its repetitive nature has been elucidated—one might be tempted to exploit this knowledge to speed up string matching. The Compressed Matching Problem is that of performing string matching in a compressed text, without uncompressing it. More formally, let T be a text, let Z be the compressed string representing T , and let P be a pattern. The Compressed Matching Problem is that of deciding if P occurs in T , given only P and Z . Compressed matching algorithms have been given for several compression schemes such as LZW. In this paper we give the first nontrivial compressed matching algorithm for the classic adaptive compression scheme, the LZ77 algorithm. In practice, the LZ77 algorithm is known to compress more than other dictionary compression schemes, such as LZ78 and LZW, though for strings with constant per bit entropy, all these schemes compress optimally in the limit. However, for strings with o(1) per bit entropy, while it was recently shown that the LZ77 gives compression to within a constant factor of optimal, schemes such as LZ78 and LZW may deviate from optimality by an exponential factor. Asymptotically, compressed matching is only relevant if |Z|=o(|T|) , i.e., if the compression ratio |T|/|Z| is more than a constant. These results show that LZ77 is the appropriate compression method in such settings. We present an LZ77 compressed matching algorithm which runs in time O(n log 2 u/n + p) where n=|Z| , u=|T| , and p=|P| . Compare with the naive ``decompresion'' algorithm, which takes time Θ(u+p) to decide if P occurs in T . Writing u+p as (n u)/n+p , we see that we have improved the complexity, replacing the compression factor u/n by a factor log 2 u/n . Our algorithm is competitive in the sense that O(n log 2 u/n + p)=O(u+p) , and opportunistic in the sense that O(n log 2 u/n + p)=o(u+p) if n=o(u) and p=o(u) .

Diagnosis of ECG diagnosable diseases—syntactic pattern recognition approach

A method is presented for the analysis of electrocardiogram (ECG) diagnosable diseases using syntactic pattern recognition techniques. The ECG analysis described here will determine right-bundle branch block, left-bundle branch block, left ventricular hypertrophy, left-anterior hemiblock, and left-atnal hypertrophy. The 'patient data matrix’, obtained through the already developed data acquisition, string generation program, string compression program, wave recognition programs, wave measurement programs, the axis analysis programs and the arrhythmia programs, is used to classify ECG diagnosable diseases of a standard patient ECG. The ECG analysis program together with the above-specified programs can be connected to standard three-channel ECG chart to provide immediate on-site ECG interpretation.

The use and usefulness of numeration systems

The proper choice of a counting system may solve mathematical problems or lead to improved algorithms. This is illustrated by a problem in combinatorial group theory, compression of sparse binary strings, encoding of contiguous binary strings of unknown lengths, ranking of permutations and combinations, strategies of games, and other examples. Two abstract counting systems are given from which the concrete ones used for the applications can be derived in an easy and transparent manner.

Compression of character strings by an adaptive dictionary

A new technique for compression of character strings is presented. The technique is based on the use of a dictionary forest which is built simultaneously with the encoding and decoding. Codes representing substrings are addresses in the dictionary forest. Experimental results show that the length of the text can be reduced more than 50% with no a priori knowledge of the nature of the text.