Longest Common Prefix Research Articles

Better External Memory LCP Array Construction

The suffix array, perhaps the most important data structure in modern string processing, needs to be augmented with the longest-common-prefix (LCP) array in many applications. Their construction is often a major bottleneck, especially when the data is too big for internal memory. We describe two new algorithms for computing the LCP array from the suffix array in external memory. Experiments demonstrate that the new algorithms are about a factor of two faster than the fastest previous algorithm. We then further engineer the two new algorithms and improve them in three ways. First, we speed up the algorithms by up to a factor of two through parallelism. Eight threads is sufficient for making the algorithms essentially I/O bound. Second, we reduce the disk space usage of the algorithms making them in-place: the input (text and suffix array) is treated as read-only, and the working disk space never exceeds the size of the final output (the LCP array). Third, we add support for large alphabets. All previous implementations assume the byte alphabet.

The query complexity of a permutation-based variant of Mastermind

We study the query complexity of a permutation-based variant of the guessing game Mastermind. In this variant, the secret is a pair (z,π) which consists of a binary string z∈{0,1}n and a permutation π of [n]. The secret must be unveiled by asking queries of the form x∈{0,1}n. For each such query, we are returned the score fz,π(x)≔max{i∈[0..n]∣∀j≤i:zπ(j)=xπ(j)};i.e., the score of x is the length of the longest common prefix of x and z with respect to the order imposed by π. The goal is to minimize the number of queries needed to identify (z,π). This problem originates from the study of black-box optimization heuristics, where it is known as the LeadingOnes problem.In this work, we prove matching upper and lower bounds for the deterministic and randomized query complexity of this game, which are Θ(nlogn) and Θ(nloglogn), respectively.

Extended suffix array construction using Lyndon factors

In this paper, we are extending the novel approach of incremental suffix array construction using Lyndon factorization to the construction of extended suffix array where extended suffix array is the suffix array along with the corresponding longest common prefix (LCP) array. Main motive behind the incremental and simultaneous construction of suffix array and LCP array is that both involve in calculating the order information by considering the common prefixes of the suffixes. As local suffixes once sorted have the same sorted order when these are merged with sorted suffixes of another Lyndon factor. So, merging of Lyndon factors is simply merging of two sorted lists of suffixes of these Lyndon factors. Also, the two sorted orders coincide thus making the merging of Lyndon factors a simple merging of two sorted lists of suffixes. Incremental LCP construction simultaneously saves a lot of computation and hence time. The proposed approach has quadratic run time and the disk working space requirement is O(n). Experiments also show the performance gain of our approach in terms of time over the existing method of incremental construction.

A Simplified Description of Child Tables for Sequence Similarity Search.

Finding related nucleotide or protein sequences is a fundamental, diverse, and incompletely-solved problem in bioinformatics. It is often tackled by seed-and-extend methods, which first find "seed" matches of diverse types, such as spaced seeds, subset seeds, or minimizers. Seeds are usually found using an index of the reference sequence(s), which stores seed positions in a suffix array or related data structure. A child table is a fundamental way to achieve fast lookup in an index, but previous descriptions have been overly complex. This paper aims to provide a more accessible description of child tables, and demonstrate their generality: they apply equally to all the above-mentioned seed types and more. We also show that child tables can be used without LCP (longest common prefix) tables, reducing the memory requirement.

Checking Big Suffix and LCP Arrays by Probabilistic Methods

For full-text indexing of massive data, the suffix and LCP (longest common prefix) arrays have been recognized as fundamental data structures, and there are at least two needs in practice for checking their correctness, i.e., program debugging and verifying the arrays constructed by probabilistic algorithms. Two probabilistic methods are proposed to check the suffix and LCP arrays of constant or integer alphabets in external memory using a Karp-Rabin fingerprinting technique, where the checking is wrong only with a negligible error probability. The first method checks the lexicographical order and the LCP-value of two suffixes by computing and comparing the fingerprints of their LCPs. This method is general in terms of that it can verify any full or sparse suffix/LCP array of any order. The second method uses less space, it first employs the fingerprinting technique to verify a subset of the given suffix and LCP arrays, from which two new suffix and LCP arrays are induced and compared with the given arrays for verification, where the induced suffix and LCP arrays can be removed for constant alphabets to save space.

Learning robot tasks with loops from experiences to enhance robot adaptability

Learning robot task models with loops helps to increase both the applicability and the compactness of task knowledge. In the framework of Experience-Based Planning Domains (EBPDs), previously formalized by the authors, an approach was developed for learning and exploiting high-level robot task models (the so-called activity schemata) with loops. The paper focuses on the development of: (i) a method—Contiguous Non-overlapping Longest Common Subsequence (CNLCS)—based on the Longest Common Prefix (LCP) array for detecting loops of actions in a robot experience; and (ii) an abstract planner to instanciate a learned task model with loops for solving particular instances of the same task with varying numbers of objects. Demonstrations of this system in both real and simulated environments prove its potentialities.

Burrows–Wheeler transform and LCP array construction in constant space

In this article we extend the elegant in-place Burrows–Wheeler transform (BWT) algorithm proposed by Crochemore et al. [12]. Our extension is twofold: we first show how to compute simultaneously the longest common prefix (LCP) array as well as the BWT, using constant additional space; we then show how to build the LCP array directly in compressed representation using Elias coding, still using constant additional space and with no asymptotic slowdown. Furthermore, we provide a time/space tradeoff for our algorithm when additional memory is allowed. Our algorithm runs in quadratic time, as does Crochemore et al.'s, and is supported by interesting properties of the BWT and of the LCP array, contributing to our understanding of the time/space tradeoff curve for building indexing structures.

Optimal suffix sorting and LCP array construction for constant alphabets

We show how the longest common prefix (LCP) array can be generated as a by-product of the suffix array construction algorithm SACA-K (Nong, 2013). Our algorithm builds on Fischer's proposal (Fischer, WADS'11), and also runs in linear time, but uses only constant extra memory for constant alphabets.

Inducing Suffix and LCP Arrays in External Memory

We consider full text index construction in external memory (EM). Our first contribution is an inducing algorithm for suffix arrays in external memory, which runs in sorting complexity. Practical tests show that this algorithm outperforms the previous best EM suffix sorter [Dementiev et al., JEA 2008] by a factor of about two in time and I/O volume. Our second contribution is to augment the first algorithm to also construct the array of longest common prefixes (LCPs). This yields a new internal memory LCP array construction algorithm and the first EM construction algorithm for LCP arrays. The overhead in time and I/O volume for this extended algorithm over plain suffix array construction is roughly two. Our algorithms scale far beyond problem sizes previously considered in the literature (text size of 80GiB using only 4GiB of RAM in our experiments).

Compressed Cache-Oblivious String B-Tree

In this article, we study three variants of the well-known prefix-search problem for strings, and we design solutions for the cache-oblivious model which improve the best known results. Among these contributions, we close (asymptotically) the classic problem, which asks for the detection of the set of strings that share the longest common prefix with a queried pattern by providing an I/O-optimal solution that matches the space lower bound for tries up to a constant multiplicative factor of the form (1 + ϵ), for ϵ > 0. Our solutions hinge upon a novel compressed storage scheme that adds the ability to decompress prefixes of the stored strings I/O-optimally to the elegant locality-preserving front coding (Bender et al. 2006) still preserving its space bounds.

State complexity of prefix distance

The prefix distance between strings x and y is the number of symbol occurrences in the strings that do not belong to the longest common prefix of x and y. The suffix and the substring distances are defined analogously in terms of the longest common suffix and longest common substring, respectively, of two strings. We show that the set of strings within prefix distance k from an n state DFA (deterministic finite automaton) language can be recognized by a DFA with (k+1)⋅n−k(k+1)2 states and that this number of states is needed in the worst case. Also we give tight bounds for the nondeterministic state complexity of the set of strings within prefix, suffix or substring distance k from a regular language.

Bitpacking techniques for indexing genomes: II.Enhanced suffix arrays.

BackgroundSuffix arrays and their variants are used widely for representing genomes in search applications. Enhanced suffix arrays (ESAs) provide fast search speed, but require large auxiliary data structures for storing longest common prefix and child interval information. We explore techniques for compressing ESAs to accelerate genomic search and reduce memory requirements.ResultsWe evaluate various bitpacking techniques that store integers in fewer than 32 bits each, as well as bytecoding methods that reserve a single byte per integer whenever possible. Our results on the fly, chicken, and human genomes show that bytecoding with an exception guide array is the fastest method for retrieving auxiliary information. Genomic searching can be further accelerated using a data structure called a discriminating character array, which reduces memory accesses to the suffix array and the genome string. Finally, integrating storage of the auxiliary and discriminating character arrays further speeds up genomic search.ConclusionsThe combination of exception guide arrays, a discriminating character array, and integrated data storage provide a 2- to 3-fold increase in speed for genomic searching compared with using bytecoding alone, and is 20 % faster and 40 % more space-efficient than an uncompressed ESA.Electronic supplementary materialThe online version of this article (doi:10.1186/s13015-016-0068-6) contains supplementary material, which is available to authorized users.

LCP Array Construction in External Memory

One of the most important data structures for string processing—the suffix array—needs to be augmented with the longest-common-prefix (LCP) array in numerous applications. We describe the first external memory algorithm for constructing the LCP array given the suffix array as input. The only previous way to compute the LCP array for data that is bigger than the RAM is to use an external memory suffix array construction algorithm (SACA) with complex modifications to produce the LCP array as a by-product. Compared to the best prior method, our algorithm needs much less disk space (by more than a factor of three) and is significantly faster. Furthermore, our algorithm can be combined with any SACA, including a better one developed in the future.

Tighter bounds for the sum of irreducible LCP values

The suffix array is frequently augmented with the longest-common-prefix (LCP) array that stores the lengths of the longest common prefixes between lexicographically adjacent suffixes of a text. While the sum of the values in the LCP array can be Ω(n2) for a text of length n, the sum of so-called irreducible LCP values was shown to be O(nlg⁡n) a few years ago. In this paper, we improve the bound to O(nlg⁡r), where r≤n is the number of runs in the Burrows–Wheeler transform of the text. We also show that our bound is tight up to lower order terms (unlike the previous bound). Our results and the techniques used in proving them provide new insights into the combinatorics of text indexing and compression, and have immediate applications to LCP array construction algorithms.

A scalable anomaly detection and mitigation architecture for legacy networks via an OpenFlow middlebox

AbstractIn this paper, we investigate the applicability of inserting an OpenFlow middlebox to enhance the remotely triggered black hole routing mechanism, to mitigate distributed denial of service (DDoS) attacks in legacy networks. Specifically, we propose a modular architecture that exploits the network programmability of software‐defined networking within the context of network functions virtualization, deploying on‐demand virtualized network functions (VNFs) capable to manipulate and filter malicious traffic. Leveraging on the OpenFlow control functionality, we match and handle traffic on a per‐flow level, preserving connectivity to/from the victim while pushing the mitigation process upstream, towards the edge of the affected network. To that end, a multilevel anomaly detection and identification mechanism was developed, pinpointing the victim in case an attack is detected. Subsequently, a virtualized network function instructs the edge router to forward all traffic destined to the victim to an OpenFlow switch, acting as a middlebox capable to filter malicious traffic identified by an OpenFlow controller, while preserving benign flows. The proposed architecture was implemented and evaluated based on the combination of datasets containing traces of real DDoS attacks and normal background traffic from our university campus network. Our analysis illustrated a clear clustering of Internet protocol prefixes used by malicious sources; thus, we implemented a longest common prefix aggregation algorithm to enable scaling of the proposed mitigation process, overcoming constraints due to hardware limitations of OpenFlow devices. Our analysis verifies that the proposed modular and scalable schema can efficiently identify DDoS attack victims and filter malicious traffic, without exhausting system and network resources. Copyright © 2015 John Wiley & Sons, Ltd.

Longest common extensions in trees

The longest common extension (LCE) of two indices in a string is the length of the longest identical substrings starting at these two indices. The LCE problem asks to preprocess a string into a compact data structure that supports fast LCE queries.In this paper we generalize the LCE problem to trees and suggest a few applications of LCE in trees to tries and XML databases. Given a labeled and rooted tree T of size n, the goal is to preprocess T into a compact data structure that support the following LCE queries between subpaths and subtrees in T. Let v1, v2, w1, and w2 be nodes of T such that w1 and w2 are descendants of v1 and v2 respectively.•LCEPP(v1,w1,v2,w2): (path–path LCE) return the longest common prefix of the paths v1↝w1 and v2↝w2.•LCEPT(v1,w1,v2): (path–tree LCE) return maximal path–path LCE of the path v1↝w1 and any path from v2 to a descendant leaf.•LCETT(v1,v2): (tree–tree LCE) return a maximal path–path LCE of any pair of paths from v1 and v2 to descendant leaves. We present the first non-trivial bounds for supporting these queries. For LCEPP queries, we present a linear-space solution with O(log⁎⁡n) query time. For LCEPT queries, we present a linear-space solution with O((log⁡log⁡n)2) query time, and complement this with a lower bound showing that any path–tree LCE structure of size O(npolylog(n)) must necessarily use Ω(log⁡log⁡n) time to answer queries. For LCETT queries, we present a time-space trade-off, that given any parameter τ, 1≤τ≤n, leads to an O(nτ) space and O(n/τ) query-time solution (all of these bounds hold on a standard unit-cost RAM model with logarithmic word size). This is complemented with a reduction from the set intersection problem implying that a fast linear space solution is not likely to exist.

Temporal logics on strings with prefix relation

We show that linear-time temporal logic over concrete domains made of finite strings and the prefix relation admits a PS pace -complete satisfiability problem. Actually, we extend a known result with the concrete domain made of the set of natural numbers and the greater than relation (corresponding to the singleton alphabet case) and we solve an open problem mentioned in several publications. Since the prefix relation is not a total ordering, it is not possible to take advantage of existing techniques dedicated to temporal logics with concrete domains that are essentially linearly ordered structures. Instead, we introduce an adequate encoding of string constraints into length constraints that allows us to reduce the problem on strings to the problem on natural numbers. To do so, we also propose an extended version of the logic on strings that is able to compare lengths of longest common prefixes and for which the satisfiability problem is shown in PS pace . Finally, we show how to lift the result for the branching-time case in order to get decidability when the underlying temporal logic is CTL*.

Short pulse groups searching based on isomorphic sequences

Conventional pulse sorting algorithms have difficulty in extracting short pulse groups which are dispersed in massive pulse streams. An isomorphic sequence search algorithm, based on a suffix array and the longest common prefix, is used to extract short pulse repetition interval-repeated pulse groups.

A simple parallel cartesian tree algorithm and its application to parallel suffix tree construction

We present a simple linear work and space, and polylogarithmic time parallel algorithm for generating multiway Cartesian trees. We show that bottom-up traversals of the multiway Cartesian tree on the interleaved suffix array and longest common prefix array of a string can be used to answer certain string queries. By adding downward pointers in the tree (e.g. using a hash table), we can also generate suffix trees from suffix arrays on arbitrary alphabets in the same bounds. In conjunction with parallel suffix array algorithms, such as the skew algorithm, this gives a rather simple linear work parallel, O ( n ε ) time (0< ε <1), algorithm for generating suffix trees over an integer alphabet Σ {1, ..., n }, where n is the length of the input string. It also gives a linear work parallel algorithm requiring O (log2 n ) time with high probability for constant-sized alphabets. More generally, given a sorted sequence of strings and the longest common prefix lengths between adjacent elements, the algorithm will generate a patricia tree (compacted trie) over the strings. Of independent interest, we describe a work-efficient parallel algorithm for solving the all nearest smaller values problem using Cartesian trees, which is much simpler than the work-efficient parallel algorithm described in previous work. We also present experimental results comparing the performance of the algorithm to existing sequential implementations and a second parallel algorithm that we implement. We present comparisons for the Cartesian tree algorithm on its own and for constructing a suffix tree. The results show that on a variety of strings our algorithm is competitive with the sequential version on a single processor and achieves good speedup on multiple processors. We present experiments for three applications that require only the Cartesian tree, and also for searching using the suffix tree.

Range LCP

In this paper, we define the Range LCP problem as follows. Preprocess a string S, of length n, to enable efficient solutions of the following query: Given [i,j], 0<i⩽j⩽n, compute maxℓ,k∈[i..j]LCP(Sℓ,Sk), where LCP(Sℓ,Sk) is the length of the longest common prefix of the suffixes of S starting at locations ℓ and k. This is a natural generalization of the classical LCP problem. We provide algorithms with the following complexities:1.Preprocessing Time:O(|S|), Space:O(|S|), Query Time:O(|j−i|loglogn).2.Preprocessing Time: none, Space:O(|j−i|log|j−i|), Query Time:O(|j−i|log|j−i|). However, the query just gives the pairs with the longest LCP, not the LCP itself.3.Preprocessing Time:O(|S|log2|S|), Space:O(|S|log1+ε|S|) for arbitrary small constant ε, Query Time:O(loglog|S|).