Compressed Text Research Articles

An approach for fast compressed text matching and to avoid false matching using WBTC and wavelet tree

Text matching is a process of finding the frequency of occurrences of text pattern in a corpus. It's very costly to store, process, and retrieve a vast volume of text data. In this paper, we present a method to keep the massive text corpus in lesser memory space by using text compression and to retrieve the results by matching directly on this compressed corpus without decompression using compressed pattern matching (CPM). The proposed approach also helps to minimize the time taken to perform matching without compromising the false matching results. We used word-based tagged coding to perform text compression and Wavelet Trees for representing the compressed text in memory. The proposed Text Matching in Compressed text using Parallel Wavelet Tree (TMC_PWT) method is quite fast in comparison to other existing text matching algorithms that support CPM. In the context of CPM, the proposed method provides a good compression ratio and does not suffer from the problem of false matching.

Urbanonyms are one of the most important elements of communication

The article deals with communication problems. The communicative functions of urban-legitimate texts are defined and described, namely: the function of preserving and transmitting national identity, traditions and culture; playback function, the function of reporting events or people; the function of summarizing information and presenting sustainable public knowledge about the world; evaluation function, the function of reference to another geographic object. The expediency of the use of urbanonyms in classes in the Ukrainian language as a foreign language has been substantiated. Urbanonyms can be both a condition and a means of communication revealing not only the essence of the language, but also performing informative and communicative functions. Urbanonomy is an important linguistic and historical-cultural component that accumulates and transmits information from one generation to another. At the same time, the urban-type text reflects the thoughts, the spiritual world of its creators, the peculiarities of a certain historical period, thus fulfilling the function of preservation and transfer of national identity, traditions, and culture. The exploration of urban units allows one to describe the aspect of the language and culture interaction, in particular, how the lexical system reflects the needs, interests and characteristics of human behavior as a carrier of a particular culture. Spatial reality, presented in the linguistic consciousness and formalized as a concept “place”, can be considered as a toponymic picture of the world, which is an integral part of the linguistic picture of the world as a whole. The toponymic picture of the world reflects the character of the culture of every ethnic group and every person. In speech, an urbanization is implemented as a compressed text and is a variant of the text of culture, since it contains information about associations, representations, background knowledge of the object. The acquaintance of foreign students with urban areas contributes to the formation of their communicative competence, provides students with the information necessary for linguocultural adaptation. Urbanonyms are one of the forms of obtaining regional knowledge.

Fast Pattern Matching in Compressed Text using Wavelet Tree

ABSTRACTWith the recent increase in the size of data, compression has become essential tool for handling massive data. Compression is usually more helpful in handling huge data over the networks, large text collections, and storing massive biological data. It is also useful in searching a pattern (of any kind) inside the huge data set. Compressed pattern matching (CPM) is the task of performing string matching in a compressed text without decompressing it. In this type of matching, pattern may or may not be compressed. This paper presents an efficient algorithm (WBTC_WT) for matching a pattern directly inside the compressed text. The proposed algorithm uses the tagged sub optimal code category of algorithm called word-based tagged code (WBTC) and a self-indexed data structure called wavelet tree. WBTC is used for encoding the text and wavelet tree provides fast searching over the compressed text. WBTC_WT removes the problem of false matches, encountered in some of the previous approaches and is able to match arbitrary portion of text without decompressing it. The proposed algorithm is implemented, analyzed, and compared with existing approaches on huge data set. From the simulation results, it is found that our algorithm outperforms the existing algorithms in most of the cases.

On optimal parsing for LZ78-like compressors

Flexible parsing algorithm, a two-steps-greedy parsing algorithm for text factorisation, has been proved to be an optimal parsing for LZ78-like compressors in the case of constant-cost phrases [1,2]. Whilst in early implementations of LZ78-like compressors the phrases have constant cost, in common modern implementations the cost of the k-th phrase is ⌈log2⁡k+C⌉ where C is a real constant [3,4]. Indeed we show examples where Flexible parsing is not optimal under the above more realistic setting. In this paper we prove that, under the assumption that the cost of a phrase is block-wise constant and non-decreasing, the Flexible parsing is almost optimal. For almost optimal we mean that, for any text T, the difference between the sizes of the compressed text obtained by using a Flexible parsing and an optimal parsing is bounded by the maximal cost of a phrase in T, i.e. it is logarithmic in practical cases.Furthermore we investigate how an optimal parsing, and hence an almost optimal parsing, affects the rate of convergence to the entropy of LZ78-like compressors. We discuss some experimental results considering the ratio between the speed of convergence to the entropy of compressors with and without an optimal parsing. This ratio presents a kind of wave effect that increases as the entropy of a memoryless source decreases but it seems always to slowly converge to one. According to the theory, this wave can be a tsunami for some families of highly compressible strings and, although the optimal (and the almost optimal) parsing does not improve the asymptotical speed of convergence to the entropy, it can improve compression ratio, and hence the decoding speed, in many practical cases.

Quality assessment of structure and language elements of written responses given by seven Scandinavian drug information centres

PurposeThe aim of this study was to identify structure and language elements affecting the quality of responses from Scandinavian drug information centres (DICs).MethodsSix different fictitious drug-related queries were sent to each of seven Scandinavian DICs. The centres were blinded for which queries were part of the study. The responses were assessed qualitatively by six clinical pharmacologists (internal experts) and six general practitioners (GPs, external experts). In addition, linguistic aspects of the responses were evaluated by a plain language expert.ResultsThe quality of responses was generally judged as satisfactory to good. Presenting specific advice and conclusions were considered to improve the quality of the responses. However, small nuances in language formulations could affect the individual judgments of the experts, e.g. on whether or not advice was given. Some experts preferred the use of primary sources to the use of secondary and tertiary sources. Both internal and external experts criticised the use of abbreviations, professional terminology and study findings that was left unexplained. The plain language expert emphasised the importance of defining and explaining pharmacological terms to ensure that enquirers understand the response as intended. In addition, more use of active voice and less compressed text structure would be desirable.ConclusionsThis evaluation of responses to DIC queries may give some indications on how to improve written responses on drug-related queries with respect to language and text structure. Giving specific advice and precise conclusions and avoiding too compressed language and non-standard abbreviations may aid to reach this goal.

A review on compressed pattern matching

Summary Compressed pattern matching (CPM) refers to the task of locating all the occurrences of a pattern (or set of patterns) inside the body of compressed text. In this type of matching, pattern may or may not be compressed. CPM is very useful in handling large volume of data especially over the network. It has many applications in computational biology, where it is useful in finding similar trends in DNA sequences; intrusion detection over the networks, big data analytics etc. Various solutions have been provided by researchers where pattern is matched directly over the uncompressed text. Such solution requires lot of space and consumes lot of time when handling the big data. Various researchers have proposed the efficient solutions for compression but very few exist for pattern matching over the compressed text. Considering the future trend where data size is increasing exponentially day-by-day, CPM has become a desirable task. This paper presents a critical review on the recent techniques on the compressed pattern matching. The covered techniques includes: Word based Huffman codes, Word Based Tagged Codes; Wavelet Tree Based Indexing. We have presented a comparative analysis of all the techniques mentioned above and highlighted their advantages and disadvantages.

Compressed parameterized pattern matching

Pattern matching between traditional strings is well-defined for both uncompressed and compressed sequences. Prior to this work, parameterized pattern matching (p-matching) was defined predominately by the matching between uncompressed parameterized strings (p-strings) from the constant alphabet Σ and the parameter alphabet Π. In this work, we define the compressed parameterized pattern matching (compressed p-matching) problem to find all of the p-matches between a pattern P and text T, using only P and the compressed text Tc. Initially, we present parameterized compression (p-compression) as a new way to losslessly compress data. Experimentally, we show that p-compression is competitive with various other standard compression schemes. Subsequently, we provide the compression and decompression algorithms. Next, two different approaches are developed to address the compressed p-matching problem: (1) using the recently proposed parameterized arithmetic codes (pAC) and (2) using the parameterized border array (p-border). Our general solution is independent of the underlying compression scheme. The results are further examined for catenate, Tunstall codes, Huffman codes, and LZSS.

Improved and extended locating functionality on compressed suffix arrays

Compressed Suffix Arrays (CSAs) offer the same functionality as classical suffix arrays (SAs), and more, within space close to that of the compressed text, and in addition they can reproduce any text fragment. Furthermore, their pattern search times are comparable to those of SAs. This combination has made CSAs extremely successful substitutes for SAs in space-demanding applications. Their weakest point is that they are orders of magnitude slower when retrieving the precise positions of pattern occurrences. SAs have other well-known shortcomings, inherited by CSAs, such as not retrieving those positions in a useful order. In this paper we present new techniques that, on the one hand, improve the current space/time tradeoffs for retrieving pattern occurrences in CSAs, and on the other, efficiently support extended pattern locating functionalities, such as retrieving occurrences in text order or limiting the occurrences to within a text window. Our experimental results display considerable savings with respect to the baseline techniques in many cases of interest: in some cases we outperform their time by a factor of two or three.

Locally Compressed Suffix Arrays

We introduce a compression technique for suffix arrays. It is sensitive to the compressibility of the text and local , meaning that random portions of the suffix array can be decompressed by accessing mostly contiguous memory areas. This makes decompression very fast, especially when various contiguous cells must be accessed. Our main technical contributions are the following. First, we show that runs of consecutive values that are known to appear in function Ψ( i ) = A −1 [ A [ i ] + 1] of suffix arrays A of compressible texts also show up as repetitions in the differential suffix array A '[ i ] = A [ i ] − A [ i −1]. Second, we use Re-Pair, a grammar-based compressor, to compress the differential suffix array, and upper bound its compression ratio in terms of the number of runs. Third, we show how to compact the space used by the grammar rules by up to 50%, while still permitting direct access to the rules. Fourth, we develop specific variants of Re-Pair that work using knowledge of Ψ, and use much less space than the general Re-Pair compressor, while achieving almost the same compression ratios. Fifth, we implement the scheme and compare it exhaustively with previous work, including the first implementations of previous theoretical proposals.

An efficient Variable-to-Fixed length encoding using multiplexed parse trees

We discuss an improved method of Variable-to-Fixed length code (VF code) encoding. A VF code is a coding scheme that splits an input text into a consecutive sequence of substrings, and then assigns a fixed length codeword to each substring. Since all the codewords have the same length, they are easily extracted, and thus, VF code is suitable for processing compressed texts directly. Almost Instantaneous VF code (AIVF code), which was proposed by Yamamoto and Yokoo in 2001, achieves a good compression ratio by using a set of parse trees. However, it requires more time and space for both encoding and decoding than does a classical VF code. In this paper, we prove that the set of parse trees of AIVF code can be multiplexed into a compact single tree and the original encoding and decoding procedures can be simulated using the compacted tree. We also give the upper and lower bounds of the number of nodes in the multiplexed parse tree, in addition to those of the number of nodes reduced from the original trees. The experimental results showed that the proposed method can encode natural language texts much faster than AIVF coding.

Improved Space-Time Tradeoffs for Approximate Full-Text Indexing with One Edit Error

In this paper we are interested in indexing texts for substring matching queries with one edit error. That is, given a text T of n characters over an alphabet of size σ, we are asked to build a data structure that answers the following query: find all the occ substrings of the text that are at edit distance at most 1 from a given string q of length m. In this paper we show two new results for this problem. The first result, suitable for an unbounded alphabet, uses O(nlog ε n) (where ε is any constant such that 0<ε<1) words of space and answers to queries in time O(m+occ). This improves simultaneously in space and time over the result of Cole et al. The second result, suitable only for a constant alphabet, relies on compressed text indices and comes in two variants: the first variant uses O(nlog ε n) bits of space and answers to queries in time O(m+occ), while the second variant uses O(nloglogn) bits of space and answers to queries in time O((m+occ)loglogn). This second result improves on the previously best results for constant alphabets achieved in Lam et al. and Chan et al.

From Talk to Power: Application ofTogether towards Lifein Local Churches

While Together towards Life (TTL) offers a fresh and compelling statement of the meaning of mission and evangelism, it is delivered a dense and highly compressed text. The author of this article argues that order for TTL's rich propositional statements to be received and enacted by churches and agents of mission, they must be expressed familiar firms such as priers and songs, Bible studies and stories, pictures, and action points--Through the creation of toolkits that would offer a range of materials to be deployed as required. TTL's particular call for spiritual engagement accords with the changing landscapes of world Christianity, where new movements, especially the global South, have critiqued the cerebral and institutional forms that Christian faith has taken the West. The article concludes that the affirmation invites us to be in the to become part of the movement of the Spirit of God the life of the world, inspired by the passion that arises through the mission of God. the kingdom of God does not consist talk but power. 1 Corinthians 4:20 A pregnant moment The adoption, by the Central Committee of the WCC, of the new affirmation on mission and evangelism, Together towards Life: Mission and Evangelism Changing Landscapes (TTL), September 2012 marks a wonderful moment the story of the worldwide church. (1) From across a very wide spectrum of world Christianity we have discovered that there is so much that we can say together about mission and evangelism. The importance of this achievement is highlighted if we consider what might have happened. The collusion of Christian mission with Western colonialism could have led to mission being completely discredited. The perception that mission involves an aggressive and oppositional stance, threatening peace and causing conflict, could have led to the quiet side-lining of mission and evangelism, with the WCC statement of 1982 gradually receding into distant memory. Instead we have a fresh and compelling statement that places mission and evangelism at the heart of the church's self-understanding. It frankly acknowledges the human error that has caused mission to be misunderstood or misused on occasion the past. It finds new direction and energy for mission by tracing it back to its origins the triune life of God and, particular, to the engagement of the Spirit of God with the life of the world. Rather than mission being about the aggrandizement of any community or interest group, its true nature is understood terms of the outpouring of the Holy Spirit, which brings the transforming love of God into the life of the world. Our calling is to discern where and how the Holy Spirit is working--and to join in. The evident reality that this Spirit-centred understanding of mission has struck a chord across such a wide spectrum of a still highly fragmented world church brings us truly to a moment of grace. Formed as we are by different histories, different theologies, and different spiritualities, we have found a point of convergence the Spirit-oriented vision of mission, which is set out TTL. For those who have mission and evangelism at heart it is a moment pregnant with possibility. This is cause for much thankfulness, but it is also a distinctly challenging situation--Will we take the opportunity it presents and make the most of it? Or will we fail to realize its rich possibilities? Specifically, will it remain the preserve of specialist scholars, professional mission functionaries, and ecumenical activists, or will it become a motivating power for the other 2,354,523,000 people who make up the world church? (2) It is safe to say that the great majority of the worldwide membership of the church have not heard of TTL. Nor are they likely to if it remains a text exclusively aimed at the specialist. Grassroots reception of the new affirmation In my own context I am responsible for a small network of churches on the west coast of Scotland. …

Computing the number of cubic runs in standard Sturmian words

The standard Sturmian words (standard words in short) are extensively studied in combinatorics of words. They are complicated enough to have many interesting properties, and, at the same time, due to their recurrent structure, they are highly compressible. In this paper, we present compact formulas for the number of cubic runs in any standard word w (denoted by ρ(3)(w)). We show also that lim sup|w|→∞ρ(3)(w)|w|=3Φ+29Φ+4≈0.36924841, where Φ=5+12 is the golden ratio, and present a sequence of strictly growing standard words achieving this limit. The exact asymptotic ratio here is irrational, contrary to the situation of squares and runs in the same class of words. Furthermore, we design an efficient algorithm for computing the number of cubic runs in standard words in linear time with respect to the size of a directive sequence, i.e., the compressed representation describing the word (recurrences). The explicit size of a word can be exponential with respect to this representation, and hence this is yet another example of a very fast computation on highly compressible texts.

Lostology: Transmedia storytelling and expansion/compression strategies

Abstract The objective of this article is to analyze Lost from the perspective of transmedia storytelling and to propose a taxonomy of transmedia expansion/compression strategies. In the first section, the article presents the basic components of transmedia storytelling from a theoretical point of view that combines narratology and semiotics. After describing the most important components of Lost's transmedia fictional universe in the second section, the article presents a general description and taxonomy of expansion/compression narrative strategies based on traditional rhetorical categories. The article also analyzes “compressed texts” – like recapitulations – and their role inside the expansive strategies.

Binary jumbled string matching for highly run-length compressible texts

The Binary Jumbled String Matching Problem is defined as follows: Given a string s over {a,b} of length n and a query (x,y), with x,y non-negative integers, decide whether s has a substring t with exactly x aʼs and y bʼs. Previous solutions created an index of size O(n) in a pre-processing step, which was then used to answer queries in constant time. The fastest algorithms for construction of this index have running time O(n2/logn) (Burcsi et al., 2010 [1]; Moosa and Rahman, 2010 [7]), or O(n2/log2n) in the word-RAM model (Moosa and Rahman, 2012 [8]). We propose an index constructed directly from the run-length encoding of s. The construction time of our index is O(n+ρ2logρ), where O(n) is the time for computing the run-length encoding of s and ρ is the length of this encoding—this is no worse than previous solutions if ρ=O(n/logn) and better if ρ=o(n/logn). Our index L can be queried in O(logρ) time. While |L|=O(min(n,ρ2)) in the worst case, preliminary investigations have indicated that |L| may often be close to ρ. Furthermore, the algorithm for constructing the index is conceptually simple and easy to implement. In an attempt to shed light on the structure and size of our index, we characterize it in terms of the prefix normal forms of s introduced in Fici and Lipták (2011) [6].1

Geometric BWT: Compressed Text Indexing via Sparse Suffixes and Range Searching

We introduce a new variant of the popular Burrows-Wheeler transform (BWT), called Geometric Burrows-Wheeler Transform (GBWT), which converts a text into a set of points in 2-dimensional geometry. We also introduce a reverse transform, called Points2Text, which converts a set of points into text. Using these two transforms, we show strong equivalence between data structural problems in geometric range searching and text pattern matching. This allows us to apply the lower bounds known in the field of orthogonal range searching to the problems in compressed text indexing. In addition, we give the first succinct (compact) index for I/O-efficient pattern matching in external memory, and show how this index can be further improved to achieve higher-order entropy compressed space.

Hide Compressed Texts in an Audio File

Hide Compressed Texts in an Audio File

Compressed text indexing with wildcards

Let T=T[email protected]^k^^1T[email protected]^k^^[email protected]^k^^dTd+1 be a text of total length n, where characters of each Ti are chosen from an alphabet @S of size @s, and @f denotes a wildcard symbol. The text indexing with wildcards problem is to index T such that when we are given a query pattern P, we can locate the occurrences of P in T efficiently. This problem has been applied in indexing genomic sequences that contain single-nucleotide polymorphisms (SNP) because SNP can be modeled as wildcards. Recently Tam et al. (2009) and Thachuk (2011) have proposed succinct indexes for this problem. In this paper, we present the first compressed index for this problem, which takes only nHh+o([email protected])+O(dlogn) bits of space, where Hh is the hth-order empirical entropy (h=o(log@sn)) of T.

On compressing and indexing repetitive sequences

We introduce LZ-End, a new member of the Lempel–Ziv family of text compressors, which achieves compression ratios close to those of LZ77 but is much faster at extracting arbitrary text substrings. We then build the first self-index based on LZ77 (or LZ-End) compression, which in addition to text extraction offers fast indexed searches on the compressed text. This self-index is particularly effective for representing highly repetitive sequence collections, which arise for example when storing versioned documents, software repositories, periodic publications, and biological sequence databases.

Implicit indexing of natural language text by reorganizing bytecodes

Word-based byte-oriented compression has succeeded on large natural language text databases, by providing competitive compression ratios, fast random access, and direct sequential searching. We show that by just rearranging the target symbols of the compressed text into a tree-shaped structure, and using negligible additional space, we obtain a new implicitly indexed representation of the compressed text, where search times are drastically improved. The occurrences of a word can be listed directly, without any text scanning, and in general any inverted-index-like capability, such as efficient phrase searches, can be emulated without storing any inverted list information. We experimentally show that our proposal performs not only much more efficiently than sequential searches over compressed text, but also than explicit inverted indexes and other types of indexes, when using little extra space. Our representation is especially successful when searching for single words and short phrases.