Substrings Of Length Research Articles

Improving Parse Trees for Efficient Variable-to-Fixed Length Codes

We address the problem of improving variable-length-to-fixed-length codes (VF codes). A VF code that we deal here with is an encoding scheme that parses an input text into variable length substrings and then assigns a fixed length codeword to each parsed substring. VF codes have favourable properties for fast decoding and fast compressed pattern matching, but they are worse in compression ratio than the latest compression methods. The compression ratio of a VF code depends on the parse tree used as a dictionary. To gain a better compression ratio we present several improvement methods for constructing parse trees. All of them are heuristical solutions since it is intractable to construct the optimal parse tree. We compared our methods with the previous VF codes, and showed experimentally that their compression ratios reach to the level of state-of-the-art compression methods.

Improving Parse Trees for Efficient Variable-to-Fixed Length Codes

We address the problem of improving variable-length-to-fixed-length codes (VF codes). A VF code that we deal here with is an encoding scheme that parses an input text into variable length substrings and then assigns a fixed length codeword to each parsed substring. VF codes have favourable properties for fast decoding and fast compressed pattern matching, but they are worse in compression ratio than the latest compression methods. The compression ratio of a VF code depends on the parse tree used as a dictionary. To gain a better compression ratio we present several improvement methods for constructing parse trees. All of them are heuristical solutions since it is intractable to construct the optimal parse tree. We compared our methods with the previous VF codes, and showed experimentally that their compression ratios reach to the level of state-of-the-art compression methods.

Rejection of Irrelevant Human Actions in Real-time Hidden Markov Model based Recognition Systems for Wearable Computers.

Hidden Markov Model (HMM) is a well established technique for detecting patterns in a stream of observations. It performs well when the observation sequence does not contain unseen patterns that were not part of the training set. In an unconstrained environment, the observation sequence might contain new patterns that the HMM model is not familiar with. In such cases, HMM will match the test pattern to some trained pattern, which is most similar to the test pattern. This increases the false positives in the system. In this paper, we are describing a threshold based technique to detect such irrelevant patterns in a continuous stream of observations, and classify them as unwanted or bad patterns. The novelty of our approach is that it allows early detection of unwanted patterns. Test patterns are validated on a fixed length substring of observation sequence, rather than on the whole observation sequence corresponding to the test pattern. The substrings are validated based on its similarity with a valid pattern using a threshold value. This reduces the latency of detection of unwanted movement, and makes the detection process independent of duration of the various pattern classes. We evaluated this technique in the context of body sensor networks based human action recognition, and have achieved about 93 percent accuracy in detecting unwanted actions, while maintaining a 94 percent accuracy of recognizing valid actions.

Efficient counting of k-mers in DNA sequences using a bloom filter

BackgroundCounting k-mers (substrings of length k in DNA sequence data) is an essential component of many methods in bioinformatics, including for genome and transcriptome assembly, for metagenomic sequencing, and for error correction of sequence reads. Although simple in principle, counting k-mers in large modern sequence data sets can easily overwhelm the memory capacity of standard computers. In current data sets, a large fraction-often more than 50%-of the storage capacity may be spent on storing k-mers that contain sequencing errors and which are typically observed only a single time in the data. These singleton k-mers are uninformative for many algorithms without some kind of error correction.ResultsWe present a new method that identifies all the k-mers that occur more than once in a DNA sequence data set. Our method does this using a Bloom filter, a probabilistic data structure that stores all the observed k-mers implicitly in memory with greatly reduced memory requirements. We then make a second sweep through the data to provide exact counts of all nonunique k-mers. For example data sets, we report up to 50% savings in memory usage compared to current software, with modest costs in computational speed. This approach may reduce memory requirements for any algorithm that starts by counting k-mers in sequence data with errors.ConclusionsA reference implementation for this methodology, BFCounter, is written in C++ and is GPL licensed. It is available for free download at http://pritch.bsd.uchicago.edu/bfcounter.html

Shorthand Universal Cycles for Permutations

The set of permutations of 〈n〉={1,…,n} in one-line notation is Π(n). The shorthand encoding of a 1⋯a n ∈Π(n) is a 1⋯a n−1. A shorthand universal cycle for permutations (SP-cycle) is a circular string of length n! whose substrings of length n−1 are the shorthand encodings of Π(n). When an SP-cycle is decoded, the order of Π(n) is a Gray code in which successive permutations differ by the prefix-rotation σ i =(1 2 ⋯ i) for i∈{n−1,n}. Thus, SP-cycles can be represented by n! bits. We investigate SP-cycles with maximum and minimum ‘weight’ (number of σ n−1s in the Gray code). An SP-cycle n a n b⋯n z is ‘periodic’ if its ‘sub-permutations’ a,b,…,z equal Π(n−1). We prove that periodic min-weight SP-cycles correspond to spanning trees of the (n−1)-permutohedron. We provide two constructions: B(n) and C(n). In B(n) the spanning trees use ‘half-hunts’ from bell-ringing, and in C(n) the sub-permutations use cool-lex order by Williams (SODA, 987–996, 2009). Algorithmic results are: (1) memoryless decoding of B(n) and C(n), (2) O((n−1)!)-time generation of B(n) and C(n) using sub-permutations, (3) loopless generation of B(n)’s binary representation n bits at a time, and (4) O(n+ν(n))-time ranking of B(n)’s permutations where ν(n) is the cost of computing a permutation’s inversion vector. Results (1)–(4) improve on those for the previous SP-cycle construction D(n) by Ruskey and Williams (ACM Trans. Algorithms 6(3):Art. 45, 2010), which we characterize here using ‘recycling’.

The convergence classes of Collatz function

The Collatz conjecture, also known as the 3 x + 1 conjecture, can be stated in terms of the reduced Collatz function R ( x ) = ( 3 x + 1 ) / 2 h (where 2 h is the larger power of 2 that divides 3 x + 1 ). The conjecture is: Starting from any odd positive integer and repeating R ( x ) we eventually get to 1. G k , the k -th convergence class, is the set of odd positive integers x such that R k ( x ) = 1 . In this paper an infinite sequence of binary strings s h of length 2 ⋅ 3 h − 1 (the seeds) are defined and it is shown that the binary representation of all x ∈ G k is the concatenation of k periodic strings whose periods are s k , … , s 1 . More precisely x = s k , d k , 1 [ n 1 ) … s 1 , d k , k [ n k ) where s k , d k , i [ n i ) is the substring of length n i that starts in position d k , i in a sufficiently long repetition of the seed s i . Finally, starting positions d k , i and lengths n i for which s k , d k , 1 [ n 1 ) … s 1 , d k , k [ n k ) ∈ G k are defined, thus giving a complete characterization of classes G k .

A fast, lock-free approach for efficient parallel counting of occurrences of k-mers

Counting the number of occurrences of every k-mer (substring of length k) in a long string is a central subproblem in many applications, including genome assembly, error correction of sequencing reads, fast multiple sequence alignment and repeat detection. Recently, the deep sequence coverage generated by next-generation sequencing technologies has caused the amount of sequence to be processed during a genome project to grow rapidly, and has rendered current k-mer counting tools too slow and memory intensive. At the same time, large multicore computers have become commonplace in research facilities allowing for a new parallel computational paradigm. We propose a new k-mer counting algorithm and associated implementation, called Jellyfish, which is fast and memory efficient. It is based on a multithreaded, lock-free hash table optimized for counting k-mers up to 31 bases in length. Due to their flexibility, suffix arrays have been the data structure of choice for solving many string problems. For the task of k-mer counting, important in many biological applications, Jellyfish offers a much faster and more memory-efficient solution. The Jellyfish software is written in C++ and is GPL licensed. It is available for download at http://www.cbcb.umd.edu/software/jellyfish.

Self-Indexed Grammar-Based Compression

Self-indexes aim at representing text collections in a compressed format that allows extracting arbitrary portions and also offers indexed searching on the collection. Current self-indexes are unable of fully exploiting the redundancy of highly repetitive text collections that arise in several applications. Grammar-based compression is well suited to exploit such repetitiveness. We introduce the first grammar-based self-index. It builds on Straight-Line Programs (SLPs), a rather general kind of context-free grammars. If an SLP of n rules represents a text T[1, u], then an SLP-compressed representation of T requires 2n log 2 n bits. For that same SLP, our self-index takes O(n log n) + n log 2 u bits. It extracts any text substring of length m in time O((m + h) log n), and finds occ occurrences of a pattern string of length m in time O((m(m + h) + h occ) log n), where h is the height of the parse tree of the SLP. No previous grammar representation had achieved o(n) search time. As byproducts we introduce (i) a representation of SLPs that takes 2n log 2 n(1 + o(1)) bits and efficiently supports more operations than a plain array of rules; (ii) a representation for binary relations with labels supporting various extended queries; (iii) a generalization of our self-index to grammar compressors that reduce T to a sequence of terminals and nonterminals, such as Re-Pair and LZ78.

Negative selection algorithms on strings with efficient training and linear-time classification

A string-based negative selection algorithm is an immune-inspired classifier that infers a partitioning of a string space Σ ℓ into “normal” and “anomalous” partitions from a training set S containing only samples from the “normal” partition. The algorithm generates a set of patterns, called “detectors”, to cover regions of the string space containing none of the training samples. Strings that match at least one of these detectors are then classified as “anomalous”. A major problem with existing implementations of this approach is that the detector generating step needs exponential time in the worst case. Here we show that for the two most widely used kinds of detectors, the r -chunk and r -contiguous detectors based on partial matching to substrings of length r , negative selection can be implemented more efficiently by avoiding generating detectors altogether: for each detector type, training set S ⊆ Σ ℓ and parameter r ≤ ℓ one can construct an automaton whose acceptance behaviour is equivalent to the algorithm’s classification outcome. The resulting runtime is O ( | S | ℓ r | Σ | ) for constructing the automaton in the training phase and O ( ℓ ) for classifying a string.

Stronger Lempel-Ziv Based Compressed Text Indexing

Given a text T[1..u] over an alphabet of size σ, the full-text search problem consists in finding the occ occurrences of a given pattern P[1..m] in T. In indexed text searching we build an index on T to improve the search time, yet increasing the space requirement. The current trend in indexed text searching is that of compressed full-text self-indices, which replace the text with a more space-efficient representation of it, at the same time providing indexed access to the text. Thus, we can provide efficient access within compressed space. The Lempel-Ziv index (LZ-index) of Navarro is a compressed full-text self-index able to represent T using 4uH k (T)+o(ulog σ) bits of space, where H k (T) denotes the k-th order empirical entropy of T, for any k=o(log σ u). This space is about four times the compressed text size. The index can locate all the occ occurrences of a pattern P in T in O(m 3log σ+(m+occ)log u) worst-case time. Although this index has proven very competitive in practice, the O(m 3log σ) term can be excessive for long patterns. Also, the factor 4 in its space complexity makes it larger than other state-of-the-art alternatives. In this paper we present stronger Lempel-Ziv based indices (LZ-indices), improving the overall performance of the original LZ-index. We achieve indices requiring (2+e)uH k (T)+o(ulog σ) bits of space, for any constant e>0, which makes them the smallest existing LZ-indices. We simultaneously improve the search time to O(m 2+(m+occ)log u), which makes our indices very competitive with state-of-the-art alternatives. Our indices support displaying any text substring of length l in optimal O(l/log σ u) time. In addition, we show how the space can be squeezed to (1+e)uH k (T)+o(ulog σ) to obtain a structure with O(m 2) average search time for m≥2log σ u. Alternatively, the search time of LZ-indices can be improved to O((m+occ)log u) with (3+e)uH k (T)+o(ulog σ) bits of space, which is much less than the space needed by other Lempel-Ziv-based indices achieving the same search time. Overall our indices stand out as a very attractive alternative for space-efficient indexed text searching.

Efficient tools for comparative substring analysis

This paper introduces an efficient implementation of approaches to alignment-free comparative genome analysis and genome-based phylogeny relying on substring composition. Distances derived from substring statistics have been proposed recently as a meaningful alternative to distances derived from sequence alignment. In particular, procaryote phylogenies based on comparative 5- and 6-mer analysis of whole proteomes have successfully been worked out. The present implementation extends the computation of composition-based distances so as to involve all k-mers for any k up to any preset m aximum length K (including K = ∞). Remarkably, although there may be Θ( L 2) distinct strings that occur in a given sequence of length L (and Θ( KL) of length k ≤ K), it is shown that composition-based distances as well as many other details of interest in comparative genome analysis can be computed in O( L) time and space (with a constant that is independent of the size of K, that is, the same constant works for all K). A typical run with 2 sequences of altogether 1.5 million characters computes their composition-based distance in about 2 s, a performance to be contrasted with the several hours needed, even when restricting attention to substrings of length at most 6, by the direct method in use. This paper • describes the details of this implementation—an implementation that allows the user to compute composition-based distances for a wide range of instances on data sets of unprecedented size which may be useful in assessing the validity of the approach and to fine-tune the identification of those values of k (or K) yielding the best separators and descriptors in correspondence with different inputs, • indicates how the proposed algorithm can also be used for other tasks related to the identification and comparative analysis of highly over- or under-represented (sub)strings in given genomes, meta-genomes, or any other sequence families of interest (e.g., all proteins encoded by a given genome, all strings of non-coding or regulatory RNA, all introns, etc.), • and thus conforms with the increasing need for radically new, fast, and massive techniques for comparative genome analysis.

MiR-192 Mediates TGF-β/Smad3-Driven Renal Fibrosis

TGF-beta/Smad3 promotes renal fibrosis, but the mechanisms that regulate profibrotic genes remain unclear. We hypothesized that miR-192, a microRNA expressed in the kidney may mediate renal fibrosis in a Smad3-dependent manner. Microarray and real-time PCR demonstrated a tight association between upregulation of miR-192 in the fibrotic kidney and activation of TGF-beta/Smad signaling. Deletion of Smad7 promoted miR-192 expression and enhanced Smad signaling and fibrosis in obstructive kidney disease. In contrast, overexpression of Smad7 to block TGF-beta/Smad signaling inhibited miR-192 expression and renal fibrosis in the rat 5/6 nephrectomy model; in vitro, overexpression of Smad7 in tubular epithelial cells abolished TGF-beta1-induced miR-192 expression. Furthermore, Smad3 but not Smad2 mediated TGF-beta1-induced miR-192 expression by binding to the miR-192 promoter. Last, overexpression of a miR-192 mimic promoted and addition of a miR-192 inhibitor blocked TGF-beta1-induced collagen matrix expression. Taken together, miR-192 may be a critical downstream mediator of TGF-beta/Smad3 signaling in the development of renal fibrosis.

Unsupervised binning of environmental genomic fragments based on an error robust selection of l-mers

BackgroundWith the rapid development of genome sequencing techniques, traditional research methods based on the isolation and cultivation of microorganisms are being gradually replaced by metagenomics, which is also known as environmental genomics. The first step, which is still a major bottleneck, of metagenomics is the taxonomic characterization of DNA fragments (reads) resulting from sequencing a sample of mixed species. This step is usually referred as “binning”. Existing binning methods are based on supervised or semi-supervised approaches which rely heavily on reference genomes of known microorganisms and phylogenetic marker genes. Due to the limited availability of reference genomes and the bias and instability of marker genes, existing binning methods may not be applicable in many cases.ResultsIn this paper, we present an unsupervised binning method based on the distribution of a carefully selected set of l-mers (substrings of length l in DNA fragments). From our experiments, we show that our method can accurately bin DNA fragments with various lengths and relative species abundance ratios without using any reference and training datasets.Another feature of our method is its error robustness. The binning accuracy decreases by less than 1% when the sequencing error rate increases from 0% to 5%. Note that the typical sequencing error rate of existing commercial sequencing platforms is less than 2%.ConclusionsWe provide a new and effective tool to solve the metagenome binning problem without using any reference datasets or markers information of any known reference genomes (species). The source code of our software tool, the reference genomes of the species for generating the test datasets and the corresponding test datasets are available at http://i.cs.hku.hk/~alse/MetaCluster/.

A promiseBQP-complete string rewriting problem

We consider the following combinatorial problem. We are given three strings s, t, and t' of length L over some fixed finite alphabet and an integer $m$ that is polylogarithmic in L. We have a symmetric relation on substrings of constant length that specifies which substrings are allowed to be replaced with each other. Let $\Delta (n)$ denote the difference between the numbers of possibilities to obtain $t$ from $s$ and $t'$ from $s$ after $n \in\N$ replacements. The problem is to determine the sign of $\Delta(m)$. As promises we have a gap condition and a growth condition. The former states that $|\Delta (m)| \geq \epsilon\,c^m$ where $\epsilon$ is inverse polylogarithmic in $L$ and $c>0$ is a constant. The latter is given by $\Delta (n) \leq c^n$ for all $n$. We show that this problem is PromiseBQP-complete, i.e., it represents the class of problems that can be solved efficiently on a quantum computer.

Practical approaches to reduce the space requirement of lempel-ziv--based compressed text indices

Given a text T [1¨ n ] over an alphabet of size σ, the full-text search problem consists in locating the occ occurrences of a given pattern P [1¨ m ] in T . Compressed full-text self-indices are space-efficient representations of the text that provide direct access to and indexed search on it. The LZ-index of Navarro is a compressed full-text self-index based on the LZ78 compression algorithm. This index requires about 5 times the size of the compressed text (in theory, 4 nH k ( T )+ o ( n logσ) bits of space, where H k ( T ) is the k -th order empirical entropy of T ). In practice, the average locating complexity of the LZ-index is O (σ m log σ n + occ σ m /2), where occ is the number of occurrences of P . It can extract text substrings of length ℓ in O (ℓ) time. This index outperforms competing schemes both to locate short patterns and to extract text snippets. However, the LZ-index can be up to 4 times larger than the smallest existing indices (which use nH k ( T )+ o ( n logσ) bits in theory), and it does not offer space/time tuning options. This limits its applicability. In this article, we study practical ways to reduce the space of the LZ-index. We obtain new LZ-index variants that require 2(1+ϵ) nH k ( T ) + o ( n logσ) bits of space, for any 0<ϵ <1. They have an average locating time of O (1/ϵ( m log n + occ σ m /2 )), while extracting takes O (ℓ) time. We perform extensive experimentation and conclude that our schemes are able to reduce the space of the original LZ-index by a factor of 2/3, that is, around 3 times the compressed text size. Our schemes are able to extract about 1 to 2 MB of the text per second, being twice as fast as the most competitive alternatives. Pattern occurrences are located at a rate of up to 1 to 4 million per second. This constitutes the best space/time trade-off when indices are allowed to use 4 times the size of the compressed text or more.

An example of mathematical authorship attribution

In this paper we discuss a novel mathematical approach to authorship attribution which we implemented recently to face a concrete problem of author recognition. The fundamental ideas for our methods came from statistical mechanics and information theory. We combine two approaches. Both of them use similarity measures between couples of texts as indicators of stylistic closeness: the first one is based on the comparison of frequencies of fixed length substrings (n-grams) throughout the texts; the second one relies on a suitable use of compression algorithms as relative entropy approximators, in the spirit of the so-called Ziv–Merhav theorem. The two methods were separately developed and then combined, together with a suitable and theoretically founded ranking analysis, to produce an original authorship attribution procedure that yielded very successful results on the specific problem to which it was applied. This ranking analysis could be of interest also in other application fields.

Closest Substring Problems with Small Distances

We study two pattern matching problems that are motivated by applications in computational biology. In the Closest Substring problem k strings $s_1,\dots, s_k$ are given, and the task is to find a string s of length L such that each string $s_i$ has a consecutive substring of length L whose distance is at most d from s. We present two algorithms that aim to be efficient for small fixed values of d and k: for some functions f and g, the algorithms have running time $f(d)\cdot n^{O(\log d)}$ and $g(d,k)\cdot n^{O(\log\log k)}$, respectively. The second algorithm is based on connections with the extremal combinatorics of hypergraphs. The Closest Substring problem is also investigated from the parameterized complexity point of view. Answering an open question from [P. A. Evans, A. D. Smith, and H. T. Wareham, Theoret. Comput. Sci., 306 (2003), pp. 407–430, M. R. Fellows, J. Gramm, and R. Niedermeier, Combinatorica, 26 (2006), pp. 141–167, J. Gramm, J. Guo, and R. Niedermeier, Lecture Notes in Comput. Sci. 2751, Springer, Berlin, 2003, pp. 195–209, J. Gramm, R. Niedermeier, and P. Rossmanith, Algorithmica, 37 (2003), pp. 25–42], we show that the problem is W[1]-hard even if both d and k are parameters. It follows as a consequence of this hardness result that our algorithms are optimal in the sense that the exponent of n in the running time cannot be improved to $o(\log d)$ or to $o(\log \log k)$ (modulo some complexity-theoretic assumptions). Consensus Patterns is the variant of the problem where, instead of the requirement that each $s_i$ has a substring that is of distance at most d from s, we have to select the substrings in such a way that the average of these k distances is at most $\delta$. By giving an $f(\delta)\cdot n^9$ time algorithm, we show that the problem is fixed-parameter tractable. This answers an open question from [M. R. Fellows, J. Gramm, and R. Niedermeier, Combinatorica, 26 (2006), pp. 141–167].

Finite automata for testing composition-based reconstructibility of sequences

Symbolic sequences uniquely reconstructible from all their substrings of length k compose a regular factorial language. We thoroughly characterize this language by its minimal forbidden words, and explicitly build up a deterministic finite automaton that accepts it. This provides an efficient on-line algorithm for testing the unique reconstructibility of the sequences.

Fast and Practical Algorithms for Planted (l, d) Motif Search

We consider the planted (l, d) motif search problem, which consists of finding a substring of length l that occurs in a set of input sequences {s1, . . . , sn} with up to d errors, a problem that arises from the need to find transcription factor-binding sites in genomic information. We propose a sequence of practical algorithms, which start based on the ideas considered in PMS1. These algorithms are exact, have little space requirements, and are able to tackle challenging instances with bigger d, taking less time in the instances reported solved by exact algorithms. In particular, one of the proposed algorithms, PMSprune, is able to solve the challenging instances, such as (17, 6) and (19, 7), which were not previously reported as solved in the literature.

Approximate parameterized matching

Two equal length strings s and s ′, over alphabets Σ s and Σ s ′, parameterize match if there exists a bijection π : Σ s → Σ s ′ such that π ( s ) = s ′, where π ( s ) is the renaming of each character of s via π. Parameterized matching is the problem of finding all parameterized matches of a pattern string p in a text t , and approximate parameterized matching is the problem of finding at each location a bijection π that maximizes the number of characters that are mapped from p to the appropriate | p |-length substring of t . Parameterized matching was introduced as a model for software duplication detection in software maintenance systems and also has applications in image processing and computational biology. For example, approximate parameterized matching models image searching with variable color maps in the presence of errors. We consider the problem for which an error threshold, k , is given, and the goal is to find all locations in t for which there exists a bijection π which maps p into the appropriate | p |-length substring of t with at most k mismatched mapped elements. Our main result is an algorithm for this problem with O ( nk 1.5 + mk log m ) time complexity, where m = | p | and n =| t |. We also show that when | p | = | t | = m , the problem is equivalent to the maximum matching problem on graphs, yielding a O ( m + k 1.5 ) solution.