String Compression Research Articles

Through the compression glass: language complexity and the linguistic structure of compressed strings

Abstract Against the backdrop of the sociolinguistic-typological complexity debate which is all about measuring, comparing and explaining language complexity, this article investigates how Kolmogorov-based information theoretic complexity relates to linguistic structures. Specifically, the linguistic structure of text which has been compressed with the text compression algorithm gzip will be analysed. One implementation of Kolmogorov-based language complexity is the compression technique (Ehret, Katharina. 2021. An information-theoretic view on language complexity and register variation: Compressing naturalistic corpus data. Corpus Linguistics and Linguistic Theory (2). 383–410) which employs gzip to measure language complexity in naturalistic text samples. In order to determine what type of structures compression algorithms like gzip capture, and how these compressed strings relate to linguistically meaningful structures, gzip’s lexicon output is retrieved and subjected to an in-depth analysis. As a case study, the compression technique is applied to the English version of Lewis Carroll’s Alice’s Adventures in Wonderland and its lexicon output is extracted. The results show that gzip-like algorithms sometimes capture linguistically meaningful structures which coincide, for instance, with lexical words or suffixes. However, many compressed sequences are linguistically unintelligible or simply do not coincide with any linguistically meaningful structures. Compression algorithms like gzip thus crucially capture purely formal structural regularities. As a consequence, information theoretic complexity, in this context, is a linguistically agnostic, purely structural measure of regularity and redundancy in texts.

A novel approach to T-cell receptor beta chain (TCRB) repertoire encoding using lossless string compression.

T-cell receptor beta chain (TCRB) repertoires are crucial for understanding immune responses. However, their high diversity and complexity present significant challenges in representation and analysis. The main motivation of this study is to develop a unified and compact representation of a TCRB repertoire that can efficiently capture its inherent complexity and diversity and allow for direct inference. We introduce a novel approach to TCRB repertoire encoding and analysis, leveraging the Lempel-Ziv 76 algorithm. This approach allows us to create a graph-like model, identify-specific sequence features, and produce a new encoding approach for an individual's repertoire. The proposed representation enables various applications, including generation probability inference, informative feature vector derivation, sequence generation, a new measure for diversity estimation, and a new sequence centrality measure. The approach was applied to four large-scale public TCRB sequencing datasets, demonstrating its potential for a wide range of applications in big biological sequencing data. Python package for implementation is available https://github.com/MuteJester/LZGraphs.

Compression and information entropy of binary strings from the collision history of three hard balls

We investigate how to measure and define the entropy of a simple chaotic system, three hard spheres on a ring. A novel approach is presented, which does not assume the ergodic hypothesis. It consists of transforming the particles’ collision history into a sequence of binary digits. We then investigate three approaches which should demonstrate the non-randomness of these collision-generated strings compared with random number generator created strings: Shannon entropy, diehard randomness tests and compression percentage. We show that the Shannon information entropy is unable to distinguish random from deterministic strings. The Diehard test performs better, but for certain mass-ratios the collision-generated strings are misidentified as random with high confidence. The zlib and bz2 compression algorithms are efficient at detecting non-randomness and low information content, with compression efficiencies that tend to 100% in the limit of infinite strings. Thus ‘compression algorithm entropy’ is non-extensive for this chaotic system, in marked contrast to the extensive entropy determined from phase-space integrals by assuming ergodicity.

String compression in FA–presentable structures

We construct a FA–presentation ψ:L→N of the structure (N;S) for which a numerical characteristic r(n) defined as the maximum number ψ(w) for all strings w∈L of length less than or equal to n grows faster than any tower of exponents of a fixed height. This result leads us to a more general notion of a compressibility rate defined for FA–presentations of any FA–presentable structure. We show the existence of FA–presentations for the configuration space of a Turing machine and Cayley graphs of some groups for which it grows faster than any tower of exponents of a fixed height. For FA–presentations of the Presburger arithmetic (N;+) we show that it is bounded from above by a linear function.

Sensitivity of string compressors and repetitiveness measures

The sensitivity of a string compression algorithm C asks how much the output size C(T) for an input string T can increase when a single character edit operation is performed on T. This notion enables one to measure the robustness of compression algorithms in terms of errors and/or dynamic changes occurring in the input string. In this paper, we analyze the worst-case multiplicative sensitivity of string compression algorithms, which is defined by maxT∈Σn⁡{C(T′)/C(T):ed(T,T′)=1}, where ed(T,T′) denotes the edit distance between T and T′. In particular, for the most common versions of the Lempel-Ziv 77 compressors, we prove that the worst-case multiplicative sensitivity is only a small constant (2 or 3, depending on the version of the Lempel-Ziv 77 and the edit operation type), i.e., the size of the Lempel-Ziv 77 factorizations can be larger by only a small constant factor. We strengthen our upper bound results by presenting matching lower bounds on the worst-case sensitivity for all these major versions of the Lempel-Ziv 77 factorizations. We generalize these results to the smallest bidirectional scheme b. In addition, we show that the sensitivity of a grammar-based compressor called GCIS (Grammar Compression by Induced Sorting) is also a small constant. Further, we extend the notion of the worst-case sensitivity to string repetitiveness measures such as the smallest string attractor size γ and the substring complexity δ, and show that the worst-case sensitivity of δ is also a small constant. These results contrast with the previously known related results such that the size z78 of the Lempel-Ziv 78 factorization can increase by a factor of Ω(n1/4) (shown by Lagarde and Perifel), and the number r of runs in the Burrows-Wheeler transform can increase by a factor of Ω(log⁡n) (shown by Giuliani et al.) when a character is prepended to an input string of length n. By applying our sensitivity bounds of δ or the smallest grammar to known results (cf. Navarro's survey) some non-trivial upper bounds for the sensitivities of important string compressors and repetitiveness measures including γ, r, LZ-End, RePair, LongestMatch, and AVL-grammar, are derived. We also exhibit the worst-case additive sensitivity maxT∈Σn⁡{C(T′)−C(T):ed(T,T′)=1}, which allows one to observe more details in the changes of the output sizes.

Comparisonal Analysis Of Even-Rodeh Algorithm Code And Fibonacci Code Algorithm For Text File Compression

The growing requirement towards bigger storage space is the main cause of new compression techniques being developed. By doing compression, big chunks of data will be lowered in size to save storage space. In this research we will be using the Even-Rodeh Code and Fibonacci Code Algorithm, the performance will be benchmarked with bitrate, compression ratio, space saving, compression and decompression time in text files. Compression is done by reading strings on text files, and then the Even-Rodeh Code and Fibonacci Code Algorithm makes a string code and performs compression. The compression results are *.erc and *.fib files containing character information and string bit which can be decompressed. The decompression result is the original text file which can be saved using the extensions *.doc, *.docx, *.pdf. In this system test the sample used are strings that contains one type of character (Homogen String) and strings that contain several types of character (heterogeneous string) saved in text files *.doc,*.docx, *.pdf. In homogeneous string compressions we can observe that the Fibonacci Code is more efficient in compression ratio, averaging at 0.25 and faster decompression time at 0.011 millisecond average compared to the Even-Rodeh Code Algorithm. In heterogeneous string compression the Fibonacci Code performs better than the Even-Rodeh Code on an average compression ratio of 0.65 and a shorter average decompression time of 0.293 milliseconds.

Entropy Bounds for Grammar-Based Tree Compressors

The definition of $\text {k}^{th}$ -order empirical entropy of strings is extended to node-labelled binary trees: A notion of $\text {k}^{th}$ -order empirical entropy for node-labelled binary trees is proposed that is able to capture regularities in both labels and structure of a tree. A suitable binary encoding of tree straight-line programs (that have been used for grammar-based tree compression before) is shown to yield binary tree encodings of size bounded by the $\text {k}^{th}$ -order empirical entropy plus some lower order terms. This result is then extended from node-labelled binary trees to node-labelled unranked trees. This generalizes recent results for grammar-based string compression to grammar-based tree compression. Additionally, experimental results with real XML document trees are presented, in which the proposed notion of $\text {k}^{th}$ -order empirical tree entropy is computed and compared to the performance of grammar-based tree compressors for those XML document trees.

Indexing Highly Repetitive String Collections, Part I

Two decades ago, a breakthrough in indexing string collections made it possible to represent them within their compressed space while at the same time offering indexed search functionalities. As this new technology permeated through applications like bioinformatics, the string collections experienced a growth that outperforms Moore’s Law and challenges our ability to handle them even in compressed form. It turns out, fortunately, that many of these rapidly growing string collections are highly repetitive, so that their information content is orders of magnitude lower than their plain size. The statistical compression methods used for classical collections, however, are blind to this repetitiveness, and therefore a new set of techniques has been developed to properly exploit it. The resulting indexes form a new generation of data structures able to handle the huge repetitive string collections that we are facing. In this survey, formed by two parts, we cover the algorithmic developments that have led to these data structures. In this first part, we describe the distinct compression paradigms that have been used to exploit repetitiveness, and the algorithmic techniques that provide direct access to the compressed strings. In the quest for an ideal measure of repetitiveness, we uncover a fascinating web of relations between those measures, as well as the limits up to which the data can be recovered, and up to which direct access to the compressed data can be provided. This is the basic aspect of indexability, which is covered in the second part of this survey.

Performance Analysis of Start-Step-Stop Codes and Gopala-Hemachandra Codes2 (GH-2(n)) Ascompression Algorithms on Text Files

Data compression was born as a solution to human needs for digital data. This research compares two compression algorithms, namely the Start-Step-Stop code algorithm and Gopala-Hemachandra codes 2 (GH-2 (n)) algorithm in the case of text file compression. compression ratio (CR), space savings (SS), bit rate and running time are calculated in the test as consideration to compare the advantages of both algorithms. Tests carried out using homogeneous strings, heterogeneous strings and artificial corpus. In homogeneous string compression, both algorithms have the same advantages because the value of CR, SS and bit rate are the same for each number of strings used. From the time of compression, Start-Step-Stop algorithm faster than the Gopala-Hemachandra algorithm. Testing of heterogeneous strings found that the Start-Step-Stop algorithm is superior to the Gopala-Hemachandra algorithm because it can save more storage space.

Stress-Strain State of the Bottom of the Casing String in the Horizontal Well

The purpose of the work is to develop the method for calculating the stress-strain state parameters of the bottom of the casing which is lowered into a horizontal well during its construction. When pushed into a well, a long casing string on the supports experiences cross bendings and bucklings which are caused by its own gravity and frictional forces created by axial forces of string compression. The latter are variables in length and depend on the reactions at the supports. The problem is solved by integrating the differential equation of the long rod buckling caused by its own gravity. In a first approximation, the reactions at the supports are found out disregard-ing the axial forces of friction. The author finds the general solution of the basic differential equation of horizontal pipe column deformations taking into account friction and axial forces acting during its lowering. It forms the basis for calculating bucklings and bendings, angles of rod rotation, its inner bending moments and cross forces at the sections between the supports. Geometric and power parameters of pipe deformations at the sections between the supports can be determined using the system of equations of the compatibleness of the angles of rotation and the support moments equilibrium. These equations contain axial force parameters, which can also be derived using the suggested system of algebraic equations. The solution of the problem becomes possible due to adding the transverse force equations to this system. These equations are obtained on the basis of the established relation between transverse and axial forces in the column and reactions and friction at the supports. It allows determining the axial compressive forces simultaneously. The method of linearization of the algebraic equations system and its iterative solution with high accuracy is developed. Due to the value of axial forces, calculated in a first approxi-mation, the system becomes linear, and its iterative solution allows finding the desired parameters with high accu-racy. The obtained results take into account the requirements to the construction technology of a horizontal well. The author considers the deviations of the well direction from the horizontal on the change of the stress-strain state of the casing. The paper offers the formulae to calculate the optimal distance between the casing centralizers as well as solves the problem of its reverse motion. The author takes into consideration additional moments of frictional forces affecting the casing centralizers. The obtained results might serve for the analysis of the stress-strain state of the casing column in the technological process of the horizontal well construction which makes it possible to increase its reliability and lifelength.

Optimality in quantum data compression using dynamical entropy

In this article we study lossless compression of strings of pure quantum states of indeterminate-length quantum codes which were introduced by Schumacher and Westmoreland. Past work has assumed that the strings of quantum data are prepared to be encoded in an independent and identically distributed way. We discuss the notion of quantum stochastic ensembles, allowing us to consider strings of quantum states prepared in a general way. For any quantum stochastic ensemble we define an associated quantum dynamical system and prove that the optimal average codeword length via lossless coding is equal to the quantum dynamical entropy of the associated quantum dynamical system.

Compressed range minimum queries

Given a string S of n integers in [0,σ), a range minimum query RMQ(i,j) asks for the index of the smallest integer in S[i…j]. It is well known that the problem can be solved with a succinct data structure of size 2n+o(n) and constant query-time. In this paper we show how to preprocess S into a compressed representation that allows fast range minimum queries. This allows for sublinear size data structures with logarithmic query time. The most natural approach is to use string compression and construct a data structure for answering range minimum queries directly on the compressed string. We investigate this approach in the context of grammar compression. We then consider an alternative approach. Instead of compressing S using string compression, we compress the Cartesian tree of S using tree compression. We show that this approach can be exponentially better than the former, is never worse by more than an O(σ) factor (i.e. for constant alphabets it is never asymptotically worse), and can in fact be worse by an Ω(σ) factor.

Efficient regular expression matching on LZ77 compressed strings using negative factors

The state-of-the-art approaches for regular expression matching on LZ78 compressed strings do not perform efficiently. Moreover, LZ78 compression has some shortcomings, such as higher compression ratio and slower decompression speed than LZ77 (a variant of LZ78). In this paper, we study regular expression matching on LZ77 compressed strings. To address this problem, we propose an efficient algorithm, namely, RELZ, utilizing the positive factors, i.e., the prefix and the suffix, and negative factors (Negative factors are substrings that cannot appear in an answer.) of the regular expression to prune the candidates. For the sake of quickly locating these two kinds of factors on the compressed string without decompression, we design a variant of suffix trie index, called SSLZ. We construct bitmaps for factors of regular expression to detect candidates. Moreover, due to the high space cost of SSLZ, we propose a variant index that partially maintain suffixes of the phrases with high frequency and develop an efficient regular expression algorithm based on the novel index, namely, RELZ+. In addition, two optimization strategies employing block filtering and LZ filtering are proposed to prune false negative candidates. At last, we conduct a comprehensive performance evaluation depending on four real data sets to validate our ideas and the proposed algorithms. The experimental results show that our RELZ and RELZ+ algorithms significantly outperform the existing algorithms.

Algorithmic clustering based on string compression to extract P300 structure in EEG signals

Background and objectives: P300 is an Event Related Potential control signal widely used in Brain Computer Interfaces. Using the oddball paradigm, a P300 speller allows a human to spell letters through P300 events produced by his/her brain. One of the most common issues in the detection of this event is that its structure may differ between different subjects and over time for a specific subject. The main purpose of this work is to deal with this inherent variability and identify the main structure of P300 using algorithmic clustering based on string compression.Methods: In this work, we make use of the Normalized Compression Distance (NCD) to extract the main structure of the signal regardless of its inherent variability. In order to apply compression distances, we carry out a novel signal-to-ASCII process that transforms and merges different events into suitable objects to be used by a compression algorithm. Once the ASCII objects are created, we use NCD-driven clustering as a tool to analyze if our object creation method suitably represents the information contained in the signals and to explore if compression distances are a valid tool for identifying P300 structure. With the purpose of increasing the level of generalization of our study, we apply two different clustering methods: a hierarchical clustering algorithm based on the minimum quartet tree method and a multidimensional projection method.Results:Our experimental results show good clustering performance over different experiments, showing the structure extraction capabilities of our procedure. Two datasets with recordings in different scenarios were used to analyze the problem and validate our results, respectively. It has to be pointed out that when the clustering performance over individual electrodes is analyzed, higher P300 activity is found in similar regions to other articles using the same datasets. This suggests that our approach might be used as an electrode-selection criteria.Conclusions: The proposed NCD-driven clustering methodology can be used to discover the structural characteristics of EEG and thereby, it is suitable as a complementary methodology for the P300 analysis.

A novel recursive backtracking genetic programming-based algorithm for 12-lead ECG compression

ECG signal is among medical signals used to diagnose heart problems. A large volume of medical signal’s data in telemedicine systems causes problems in storing and sending tasks. In the present paper, a recursive algorithm with backtracking approach is used for ECG signal compression. This recursive algorithm constructs a mathematical estimator function for each segment of the signal using genetic programming algorithm. When all estimator functions of different segments of the signal are determined and put together, a piecewise-defined function is constructed. This function is utilized to generate a reconstructed signal in the receiver. The compression result is a set of compressed strings representing the piecewise-defined function which is coded through a text compression method. In order to improve the compression results in this method, the input signal is smoothed. MIT-BIH arrhythmia database is employed to test and evaluate the proposed algorithm. The results of this algorithm include the average of compression ratio that equals 30.97 and the percent root-mean-square difference that is equal to 2.38%, suggesting its better efficiency in comparison with other state-of-the-art methods.

A Detection Method for Plagiarism Reports of Students

In recent years, plagiarism that uses the sentences or phrases of others without permission has become a social problem. It is widely spread from very familiar student reports to novels and worldwide academic papers.In this paper, we deal with plagiarism in student reports, and explain the plagiarism patterns often found there. Then we propose a method to detect them efficiently and accurately.This method is based on the way of making two texts to be compared with into one-dimension string respectively, repeating shift and mutual comparison, and checking the matching section of words. This method can accurately detect perfect matches of any size, regardless of its placement in the text.To this basic detection method, we combine some heuristics which are estimation of detection possibility and compression of strings, to improve both detection accuracy of the plagiarism sections and the reduction of calculation time, and propose it as a plagiarism detection method.This method has only one parameter for plagiarism judgement, and it is also intuitive and easy to set.Finally, we show the effectiveness of the proposed method, especially the introduced heuristics, using real data.

Finger Search in Grammar-Compressed Strings

Grammar-based compression, where one replaces a long string by a small context-free grammar that generates the string, is a simple and powerful paradigm that captures many popular compression schemes. Given a grammar, the random access problem is to compactly represent the grammar while supporting random access, that is, given a position in the original uncompressed string report the character at that position. In this paper we study the random access problem with the finger search property, that is, the time for a random access query should depend on the distance between a specified index f, called the finger, and the query index i. We consider both a static variant, where we first place a finger and subsequently access indices near the finger efficiently, and a dynamic variant where also moving the finger such that the time depends on the distance moved is supported. Let n be the size the grammar, and let N be the size of the string. For the static variant we give a linear space representation that supports placing the finger in O(log N) time and subsequently accessing in O(log D) time, where D is the distance between the finger and the accessed index. For the dynamic variant we give a linear space representation that supports placing the finger in O(log N) time and accessing and moving the finger in O(log D + log log N) time. Compared to the best linear space solution to random access, we improve a O(log N) query bound to O(log D) for the static variant and to O(log D + log log N) for the dynamic variant, while maintaining linear space. As an application of our results we obtain an improved solution to the longest common extension problem in grammar compressed strings. To obtain our results, we introduce several new techniques of independent interest, including a novel van Emde Boas style decomposition of grammars.

Fingerprints in compressed strings

In this paper we show how to construct a data structure for a string S of size N compressed into a context-free grammar of size n that supports efficient Karp–Rabin fingerprint queries to any substring of S. That is, given indices i and j, the answer to a query is the fingerprint of the substring S[i,j]. We present the first O(n) space data structures that answer fingerprint queries without decompressing any characters. For Straight Line Programs (SLP) we get O(log⁡N) query time, and for Linear SLPs (an SLP derivative that captures LZ78 compression and its variations) we get O(log⁡log⁡N) query time. We extend the result to solve the longest common extension problem in query time O(log⁡Nlog⁡ℓ) and O(log⁡ℓlog⁡log⁡ℓ+log⁡log⁡N) for SLPs and Linear SLPs, respectively. Here, ℓ denotes the length of the LCE.

Recompression

In this article, we present an application of a simple technique of local recompression, previously developed by the author in the context algorithms for compressed strings [Jeż 2014a, 2015b, 2015a], to word equations. The technique is based on local modification of variables (replacing X by aX or Xa ) and iterative replacement of pairs of letters occurring in the equation by a “fresh” letter, which can be seen as a bottom-up compression of the solution of the given word equation, or, to be more specific, building a Straight-Line Programme for the solution of the word equation. Using this technique, we give new, independent, and self-contained proofs of many known results for word equations. To be more specific, the presented (nondeterministic) algorithm runs in O ( n log n space and in time polynomial in n and log N , where n is the size of the input equation and N the size of the length-minimal solution of the word equation. Furthermore, for O (1) variables, the bound on the space consumption is in fact linear, that is, O ( m ), where m is the size of the space used by the input. This yields that for each k the set of satisfiable word equations with k variables is context sensitive. The presented algorithm can be easily generalised to a generator of all solutions of the given word equation (without increasing the space usage). Furthermore, a further analysis of the algorithm yields an independent proof of doubly exponential upper bound on the size of the length-minimal solution. The presented algorithm does not use exponential bound on the exponent of periodicity. Conversely, the analysis of the algorithm yields an independent proof of the exponential bound on exponent of periodicity.

Compressed Subsequence Matching and Packed Tree Coloring

We present a new algorithm for subsequence matching in grammar compressed strings. Given a grammar of size n compressing a string of size N and a pattern string of size m over an alphabet of size $$\sigma $$ , our algorithm uses $$O(n+\frac{n\sigma }{w})$$ space and $$O(n+\frac{n\sigma }{w}+m\log N\log w\cdot occ)$$ or $$O(n+\frac{n\sigma }{w}\log w+m\log N\cdot occ)$$ time. Here w is the word size and occ is the number of minimal occurrences of the pattern. Our algorithm uses less space than previous algorithms and is also faster for $$occ=o(\frac{n}{\log N})$$ occurrences. The algorithm uses a new data structure that allows us to efficiently find the next occurrence of a given character after a given position in a compressed string. This data structure in turn is based on a new data structure for the tree color problem, where the node colors are packed in bit strings.