Pattern Matching Queries Research Articles

Elastic founder graphs improved and enhanced

Indexing labeled graphs for pattern matching is a central challenge of pangenomics. Equi et al. (2022) [14] developed the Elastic Founder Graph (EFG) representing an alignment of m sequences of length n, drawn from alphabet Σ plus the special gap character: the paths spell the original sequences or their recombination. By enforcing the semi-repeat-free property, the EFG admits a polynomial-space index for linear-time pattern matching, breaking through the conditional lower bounds on indexing labeled graphs (Equi et al. [13]). In this work, we improve the space of the EFG index answering pattern matching queries in linear time, from linear in the length of all strings spelled by three consecutive node labels, to linear in the size of the edge labels. Then, we develop linear-time construction algorithms optimizing for different metrics: we improve the existing linearithmic construction algorithms to O(mn), by solving the novel exclusive ancestor set problem on trees; we propose, for the simplified gapless setting, an O(mn)-time solution minimizing the maximum block height, that we generalize by substituting block height with prefix-aware height. Finally, to show the versatility of the framework, we develop a BWT-based EFG index and study how to encode and perform document listing queries on a set of paths of the graphs, reporting which paths present a given pattern as a substring. We propose the EFG framework as an improved and enhanced version of the framework for the gapless setting, along with construction methods that are valid in any setting concerned with the segmentation of aligned sequences.

An efficient approach for faster matching of approximate patterns in graphs

Graphs have proven to be an efficient problem representation scheme in many real-world applications and can serve to address mining of patterns in large volumes of data. This work addresses the data management issue in graph databases for shortest path queries, verification of reachability, and pattern matching queries. The key issue is to match all possible patterns in a large data graph that matches a user-provided pattern, known as the query graph. The large size of the graph makes the problem complex when searching for all patterns that are similar to the user’s query. For a given query, a match occurs when a graph with n vertices has the same label as the corresponding vertices in the query graph. This work addresses such pattern matching query problems on large graphs and presents an efficient algorithm to mine all patterns that satisfies the given query. The current work presents a novel Fast Graph Pattern (FGP) mining algorithm that performs a label-specific walk on a graph and stores all vertices that are visited during the walk and hashes them on a labeled root. Experiments are performed using five benchmark datasets, and the proposed approach is compared with three state-of-the-art methods. The obtained results suggest better performance of the current proposal by achieving five times quicker results.

Text Indexing for Long Patterns: Anchors are All you Need

In many real-world database systems, a large fraction of the data is represented by strings: sequences of letters over some alphabet. This is because strings can easily encode data arising from different sources. It is often crucial to represent such string datasets in a compact form but also to simultaneously enable fast pattern matching queries. This is the classic text indexing problem. The four absolute measures anyone should pay attention to when designing or implementing a text index are: (i) index space; (ii) query time; (iii) construction space; and (iv) construction time. Unfortunately, however, most (if not all) widely-used indexes (e.g., suffix tree, suffix array, or their compressed counterparts) are not optimized for all four measures simultaneously, as it is difficult to have the best of all four worlds. Here, we take an important step in this direction by showing that text indexing with locally consistent anchors (lc-anchors) offers remarkably good performance in all four measures, when we have at hand a lower bound l on the length of the queried patterns --- which is arguably a quite reasonable assumption in practical applications. Specifically, we improve on the construction of the index proposed by Loukides and Pissis, which is based on bidirectional string anchors (bd-anchors), a new type of lc-anchors, by: (i) designing an average-case linear-time algorithm to compute bd-anchors; and (ii) developing a semi-external-memory implementation to construct the index in small space using near-optimal work. We then present an extensive experimental evaluation, based on the four measures, using real benchmark datasets. The results show that, for long patterns, the index constructed using our improved algorithms compares favorably to all classic indexes: (compressed) suffix tree; (compressed) suffix array; and the FM-index.

Gapped Indexing for Consecutive Occurrences

The classic string indexing problem is to preprocess a string S into a compact data structure that supports efficient pattern matching queries. Typical queries include existential queries (decide if the pattern occurs in S), reporting queries (return all positions where the pattern occurs), and counting queries (return the number of occurrences of the pattern). In this paper we consider a variant of string indexing, where the goal is to compactly represent the string such that given two patterns \(P_1\) and \(P_2\) and a gap range \({[}\alpha , \beta ]\) we can quickly find the consecutive occurrences of \(P_1\) and \(P_2\) with distance in \({[}\alpha , \beta ]\), i.e., pairs of subsequent occurrences with distance within the range. We present data structures that use linear space and query time \({\widetilde{O}}(|P_1|+|P_2|+n^{2/3})\) for existence and counting and \({\widetilde{O}}(|P_1|+|P_2|+n^{2/3}\hbox {occ}^{1/3})\) for reporting. We complement this with a conditional lower bound based on the set intersection problem showing that any solution using \({\widetilde{O}}(n)\) space must use \({\widetilde{\Omega }}(|P_1| + |P_2| + \sqrt{n})\) query time. To obtain our results we develop new techniques and ideas of independent interest including a new suffix tree decomposition and hardness of a variant of the set intersection problem.

Example-based spatial pattern matching

The prevalence of GPS-enabled mobile devices and location-based services yield massive volume of spatial objects where each object contains information including geographical location, name, address, category and other attributes. This paper introduces a novel type of query termed example-based spatial pattern matching (EPM) query. It takes as input a set of spatial objects, each of which is associated with one or more keywords and a location. These objects serve as an example that depicts the spatial pattern that users want to retrieve. The EPM query returns all sets of objects that match the spatial pattern. The EPM query can be used for applications like urban planning, scene recognition and similar region search. We propose an efficient algorithm and three pruning techniques to answer EPM queries. Furthermore, we provide an approximation guarantee for intermediate results of the algorithm. Our experimental evaluations on four real-world datasets demonstrate the effectiveness and efficiency of our proposed algorithm and techniques.

String indexing for top-k close consecutive occurrences

The classic string indexing problem is to preprocess a string S into a compact data structure that supports efficient subsequent pattern matching queries, that is, given a pattern string P, report all occurrences of P within S. In this paper, we study a basic and natural extension of string indexing called the string indexing for top-k close consecutive occurrences problem (Sitcco). Here, a consecutive occurrence is a pair (i,j), i<j, such that P occurs at positions i and j in S and there is no occurrence of P between i and j, and their distance is defined as j−i. Given a pattern P and a parameter k, the goal is to report the top-k consecutive occurrences of P in S of minimal distance. The challenge is to compactly represent S while supporting queries in time close to the length of P and k. We give three time-space trade-offs for the problem. Let n be the length of S, m the length of P, and ϵ∈(0,1]. Our first result achieves O(nlog⁡n) space and optimal query time of O(m+k). Our second and third results achieve linear space and query times either O(m+k1+ϵ) or O(m+klog1+ϵ⁡n). Along the way, we develop several techniques of independent interest, including a new translation of the problem into a line segment intersection problem and a new recursive clustering technique for trees.

MFPMiner: Mining Meaningful Frequent Patterns from Spatio-textual Trajectories

In the second decade of this century, technical progress has led to a worldwide proliferation of devices for tracking the movement behavior of a person, a vehicle, or another kind of entity. One of the consequences of this development is a massive and still growing amount of movement and movement-related data recorded by cellphones, automobiles, vessels, aircraft, and further GPS-enabled entities. As a result, the requirements for managing and analyzing movement records also increase, serving commercial, administrative, or private purposes. Since the development of hardware components cannot keep pace with the data growth, exploring methods of analyzing such trajectory datasets has become a very active and influential research field. For many application scenarios, besides the spatial trajectory of an entity, it is desirable to take additional semantic information into consideration. These descriptions also change with time and may represent, e.g., the course of streets passed by a bus, the sequence of region names traversed by an aircraft, or the points of interest in proximity of the positions of a taxi. Such data may be directly recorded by a sensor (such as the altitude of an aircraft) or computed from the spatial trajectory combined with some underlying information (for example, street names). It is often helpful or even necessary to focus on such semantic information for efficient analyses, as changes usually occur less frequently than it is the case for the spatial trajectory, where data points usually arrive in very close temporal distances. However, any kind of querying requires a deep semantic knowledge of the dataset at hand, particularly for retrieving the set of trajectories that match a certain mobility pattern, that is, a sequence of temporal, spatial, and semantic specifications. In this article, we introduce a framework named MFPMiner 1 for retrieving all mobility patterns fulfilling a user-specified frequency threshold from a spatio-textual trajectory dataset. The resulting patterns and their relative frequency can be regarded as a knowledge base of the considered data. They may be directly visualized or applied for a pattern matching query yielding the set of matching trajectories. We demonstrate the functionality of our approach in an application scenario and provide an experimental evaluation of its performance on real and synthetic datasets by comparing it to three competitive methods. The framework has been fully implemented in a DBMS environment and is freely available open source software.

Reverse-Safe Text Indexing

We introduce the notion of reverse-safe data structures. These are data structures that prevent the reconstruction of the data they encode (i.e., they cannot be easily reversed). A data structure D is called z - reverse-safe when there exist at least z datasets with the same set of answers as the ones stored by D . The main challenge is to ensure that D stores as many answers to useful queries as possible, is constructed efficiently, and has size close to the size of the original dataset it encodes. Given a text of length n and an integer z , we propose an algorithm that constructs a z -reverse-safe data structure ( z -RSDS) that has size O(n) and answers decision and counting pattern matching queries of length at most d optimally, where d is maximal for any such z -RSDS. The construction algorithm takes O(nɷ log d) time, where ɷ is the matrix multiplication exponent. We show that, despite the nɷ factor, our engineered implementation takes only a few minutes to finish for million-letter texts. We also show that plugging our method in data analysis applications gives insignificant or no data utility loss. Furthermore, we show how our technique can be extended to support applications under realistic adversary models. Finally, we show a z -RSDS for decision pattern matching queries, whose size can be sublinear in n . A preliminary version of this article appeared in ALENEX 2020.

Towards Dynamic Verifiable Pattern Matching

Verifiable pattern matching enables users to obtain authenticated query results over outsourced data on an untrusted remote server. It is a fundamental problem in many security-critical big data applications, including big database search, human genome data search, text search, etc., especially when these applications are outsourced to third-party clouds. However, the state-of-the-art schemes do not yet support efficient data updates. In this work, we propose the first dynamic verifiable pattern matching scheme to support efficient data updates. The proposed scheme is built on two ideas: one is to embed unique randomness to decouple the character and its index in the outsourced data, enabling efficient data updates; the other is to reduce the verifiable pattern matching problem to a discrete set membership testing problem, which relies on the decoupling introduced in the first idea. Based on these two ideas, the proposed scheme first employs the suffix array index structure to search pattern matching queries. The scheme then authenticates the outsourced text using a newly designed authenticated data structure based on the RSA accumulator, which guarantees the verifiability of pattern matching query results. Data update is naturally supported using the RSA accumulator working on discrete sets. Based on the proposed design, we have prototyped a proof-of-concept for the proposed scheme and have conducted an extensive experimental evaluation. In addition to supporting efficient data update, our experimental results show that the proposed scheme incurs reduced verification cost in comparison with the baseline state-of-the-art scheme.

Towards a Complete Perspective on Labeled Tree Indexing: New Size Bounds, Efficient Constructions, and Beyond

A labeled tree (or a trie) is a natural generalization of a string, which can also be seen as a compact representation of a set of strings. This paper considers the labeled tree indexing problem, and provides a number of new results on space bound analysis, and on algorithms for efficient construction and pattern matching queries. Kosaraju [FOCS 1989] was the first to consider the labeled tree indexing problem, and he proposed the suffix tree for a backward trie, where the strings in the trie are read in the leaf-to-root direction. In contrast to a backward trie, we call a usual trie as a forward trie. Despite a few follow-up works after Kosaraju's paper, indexing forward/backward tries is not well understood yet. In this paper, we show a full perspective on the sizes of indexing structures such as suffix trees, DAWGs, CDAWGs, suffix arrays, affix trees, affix arrays for forward and backward tries. Some of them take $O(n)$ space in the size $n$ of the input trie, while the others can occupy $O(n^2)$ space in the worst case. In particular, we show that the size of the DAWG for a forward trie with $n$ nodes is $\Omega(\sigma n)$, where $\sigma$ is the number of distinct characters in the trie. This becomes $\Omega(n^2)$ for an alphabet of size $\sigma = \Theta(n)$. Still, we show that there is a compact $O(n)$-space implicit representation of the DAWG for a forward trie, whose space requirement is independent of the alphabet size. This compact representation allows for simulating each DAWG edge traversal in $O(\log \sigma)$ time, and can be constructed in $O(n)$ time and space over any integer alphabet of size $O(n)$. In addition, this readily extends to the first indexing structure that permits bidirectional pattern searches over a trie within linear space in the input trie size.

Astrid

Accurate selectivity estimation for string predicates is a long-standing research challenge in databases. Supporting pattern matching on strings (such as prefix, substring, and suffix) makes this problem much more challenging, thereby necessitating a dedicated study. Traditional approaches often build pruned summary data structures such as tries followed by selectivity estimation using statistical correlations. However, this produces insufficiently accurate cardinality estimates resulting in the selection of sub-optimal plans by the query optimizer. Recently proposed deep learning based approaches leverage techniques from natural language processing such as embeddings to encode the strings and use it to train a model. While this is an improvement over traditional approaches, there is a large scope for improvement. We propose Astrid, a framework for string selectivity estimation that synthesizes ideas from traditional and deep learning based approaches. We make two complementary contributions. First, we propose an embedding algorithm that is query-type (prefix, substring, and suffix) and selectivity aware. Consider three strings 'ab', 'abc' and 'abd' whose prefix frequencies are 1000, 800 and 100 respectively. Our approach would ensure that the embedding for 'ab' is closer to 'abc' than 'abd'. Second, we describe how neural language models could be used for selectivity estimation. While they work well for prefix queries, their performance for substring queries is sub-optimal. We modify the objective function of the neural language model so that it could be used for estimating selectivities of pattern matching queries. We also propose a novel and efficient algorithm for optimizing the new objective function. We conduct extensive experiments over benchmark datasets and show that our proposed approaches achieve state-of-the-art results.

FeatureFlow: exploring feature evolution for time-varying volume data

Time-varying volume data generated from scientific simulations are generally temporal and contain dynamic and complex features. The evolution of these features is important to understand the phenomena hidden in the data. In this paper, we introduce FeatureFlow, which is a novel visualization technique revealing feature evolution based on a hierarchical river metaphor. FeatureFlow decomposes the entire feature evolution into multiple levels and exploits an evolution measure to quantify the changes of the features. FeatureFlow visually summarizes the hierarchical evolution, the evolution value, and associated attributes to intuitively display the complex 4D spatial-temporal feature evolution in 2D. In addition, FeatureFlow converts each river into a string based on the serial ordering of evolutionary events and supports evolutionary pattern-matching queries. Experiments on three time-varying volume data sets and feedback from two domain experts demonstrate the utility of FeatureFlow in effectively helping users understand and explore feature evolution in time-varying volume data.

Spatial pattern matching

In this paper, we study the spatial pattern matching (SPM) query. Given a set D of spatial objects (e.g., houses and shops), each with a textual description, we aim at finding all combinations of objects from D that match a user-defined spatial pattern P. A pattern P is a graph whose vertices represent spatial objects, and edges denote distance relationships between them. The SPM query returns the instances that satisfy P. An example of P can be "a house within 10-minute walk from a school, which is at least 2km away from a hospital". The SPM query can benefit users such as house buyers, urban planners, and archaeologists. We first formally formulate the SPM problem, and then propose efficient query algorithms. We also develop an online system, called SpaceKey, which is based on the SPM query, to support some real applications such as property searching. Finally, we point out a list of possible research directions for future work.

Evaluating pattern matching queries for spatial databases

In this paper, we study the spatial pattern matching (SPM) query. Given a set D of spatial objects (e.g., houses and shops), each with a textual description, we aim at finding all combinations of objects from D that match a user-defined spatial patternP. A pattern P is a graph whose vertices represent spatial objects, and edges denote distance relationships between them. The SPM query returns the instances that satisfy P. An example of P can be “a house within 10-min walk from a school, which is at least 2 km away from a hospital.” The SPM query can benefit users such as house buyers, urban planners, and archeologists. We prove that answering such queries is computationally intractable and propose two efficient algorithms for their evaluation. Moreover, we study efficient solutions to address two related problems of the SPM: (1) find top-k matches that are close to a query location and (2) return partial matches for a query pattern. Experiments and case studies on real datasets show that our proposed solutions are highly effective and efficient.

Secure Hashing-Based Verifiable Pattern Matching

Verifiable pattern matching is the problem of finding a given pattern verifiably from the outsourced textual data, which is resident in an untrusted remote server. This problem has drawn much attention due to a large number of applications. The state-of-the-art method for this problem suffers from low efficiency. To enable fast verifiable pattern matching, we propose a novel scheme in this paper. Our scheme is based on an ordered set accumulator data structure and a newly developed verifiable suffix array structure, which only involves fast cryptographic hash computations. Our scheme also supports fast multiple-occurrence pattern matching. A striking feature of our proposed scheme is that our scheme works even with no secret keys, which ensures public verifiability. We conduct extensive experiments to evaluate the proposed scheme using Java. The results show that our scheme is orders of magnitude faster than the state-of-the-art work. Specifically, our scheme with public verifiability only costs a preprocessing time of 47 s (merely one-time off-line cost during outsourcing), a search time of $30~\mu \text{s}$ , a verification time of $149~\mu \text{s}$ , and a proof size of 2760 bytes for a verifiable pattern matching query with pattern length 200 on 10-million long textual data which consists of sequences of two-byte, Unicode characters in Java.

Fixed Block Compression Boosting in FM-Indexes: Theory and Practice

The FM index (Ferragina and Manzini in J ACM 52(4):552–581, 2005) is a widely-used compressed data structure that stores a string T in a compressed form and also supports fast pattern matching queries. In this paper, we describe new FM-index variants that combine nice theoretical properties, simple implementation and improved practical performance. Our main theoretical result is a new technique called fixed block compression boosting, which is a simpler and faster alternative to optimal compression boosting and implicit compression boosting used in previous FM-indexes. We also describe several new techniques for implementing fixed-block boosting efficiently, including a new, fast, and space-efficient implementation of wavelet trees. Our extensive experiments show the new indexes to be consistently fast and small relative to the state-of-the-art, and thus they make a good “off-the-shelf” choice for many applications.

Digree

In this work we present Digree, a system prototype that enables distributed execution of graph pattern matching queries in a cloud of interconnected graph databases. We explain how a graph query can be decomposed into independent sub-patterns that are processed in parallel by the distributed independent graph database systems and how the results are finally synthesized at a master node. We experimentally compare a prototype of our system against a popular big data engine and show that Digree provides significantly faster query execution.

ODG: Omics database generator - a tool for generating, querying, and analyzing multi-omics comparative databases to facilitate biological understanding

BackgroundRapid generation of omics data in recent years have resulted in vast amounts of disconnected datasets without systemic integration and knowledge building, while individual groups have made customized, annotated datasets available on the web with few ways to link them to in-lab datasets. With so many research groups generating their own data, the ability to relate it to the larger genomic and comparative genomic context is becoming increasingly crucial to make full use of the data.ResultsThe Omics Database Generator (ODG) allows users to create customized databases that utilize published genomics data integrated with experimental data which can be queried using a flexible graph database. When provided with omics and experimental data, ODG will create a comparative, multi-dimensional graph database. ODG can import definitions and annotations from other sources such as InterProScan, the Gene Ontology, ENZYME, UniPathway, and others. This annotation data can be especially useful for studying new or understudied species for which transcripts have only been predicted, and rapidly give additional layers of annotation to predicted genes. In better studied species, ODG can perform syntenic annotation translations or rapidly identify characteristics of a set of genes or nucleotide locations, such as hits from an association study. ODG provides a web-based user-interface for configuring the data import and for querying the database. Queries can also be run from the command-line and the database can be queried directly through programming language hooks available for most languages. ODG supports most common genomic formats as well as generic, easy to use tab-separated value format for user-provided annotations.ConclusionsODG is a user-friendly database generation and query tool that adapts to the supplied data to produce a comparative genomic database or multi-layered annotation database. ODG provides rapid comparative genomic annotation and is therefore particularly useful for non-model or understudied species. For species for which more data are available, ODG can be used to conduct complex multi-omics, pattern-matching queries.

Packed Compact Tries: A Fast and Efficient Data Structure for Online String Processing

In this paper, we present a new data structure called the packed compact trie (packed c-trie) which stores a set $S$ of $k$ strings of total length $n$ in $n \log\sigma + O(k \log n)$ bits of space and supports fast pattern matching queries and updates, where $\sigma$ is the size of an alphabet. Assume that $\alpha = \log_\sigma n$ letters are packed in a single machine word on the standard word RAM model, and let $f(k,n)$ denote the query and update times of the dynamic predecessor/successor data structure of our choice which stores $k$ integers from universe $[1,n]$ in $O(k \log n)$ bits of space. Then, given a string of length $m$, our packed c-tries support pattern matching queries and insert/delete operations in $O(\frac{m}{\alpha} f(k,n))$ worst-case time and in $O(\frac{m}{\alpha} + f(k,n))$ expected time. Our experiments show that our packed c-tries are faster than the standard compact tries (a.k.a. Patricia trees) on real data sets. As an application of our packed c-trie, we show that the sparse suffix tree for a string of length $n$ over prefix codes with $k$ sampled positions, such as evenly-spaced and word delimited sparse suffix trees, can be constructed online in $O((\frac{n}{\alpha} + k) f(k,n))$ worst-case time and $O(\frac{n}{\alpha} + k f(k,n))$ expected time with $n \log \sigma + O(k \log n)$ bits of space. When $k = O(\frac{n}{\alpha})$, by using the state-of-the-art dynamic predecessor/successor data structures, we obtain sub-linear time construction algorithms using only $O(\frac{n}{\alpha})$ bits of space in both cases. We also discuss an application of our packed c-tries to online LZD factorization.

Using domain-specific languages for analytic graph databases

Recently graph has been drawing lots of attention both as a natural data model that captures fine-grained relationships between data entities and as a tool for powerful data analysis that considers such relationships. In this paper, we present a new graph database system that integrates a robust graph storage with an efficient graph analytics engine. Primarily, our system adopts two domain-specific languages (DSLs), one for describing graph analysis algorithms and the other for graph pattern matching queries. Compared to the API-based approaches in conventional graph processing systems, the DSL-based approach provides users with more flexible and intuitive ways of expressing algorithms and queries. Moreover, the DSL-based approach has significant performance benefits as well, (1) by skipping (remote) API invocation overhead and (2) by applying high-level optimization from the compiler.