Compact Data Structure Research Articles

Clustering-based compression for raster time series

Abstract A raster time series is a sequence of independent rasters arranged chronologically covering the same geographical area. These are commonly used to depict the temporal evolution of represented variables. The $T$-$k^{2}$-raster is a compact data structure that performs very well in practice for compact representations for raster time series. This structure classifies each raster as a snapshot or a log and encodes logs concerning their reference snapshots, which are the immediately preceding selected snapshots. An enhanced version of the $T$-$k^{2}$-raster, called Heuristic $T$-$k^{2}$-raster, incorporates a heuristic for automating the selection of snapshots. In this study, we investigate the optimality of the heuristic employed in Heuristic $T$-$k^{2}$-raster by comparing it with a dynamic programming (DP) approach. Our experimental evaluation demonstrates that Heuristic $T$-$k^{2}$-raster is a near-optimal solution, achieving compression performance almost identical to the DP method. These results indicate that variations of the structure that maintain the temporal order of the rasters are unlikely to significantly improve compression. Consequently, we explore an alternative approach based on clustering, where rasters are grouped according to their similarity, regardless of their temporal order. Our experimental evaluation reveals that this clustering-based strategy can enhance compression in scenarios characterized by cyclic behaviour.

Space‐efficient data structures for the inference of subsumption and disjointness relations

AbstractConventional database systems function as static data repositories, storing vast amounts of facts and offering efficient query processing capabilities. The sheer volume of data these systems store has a direct impact on their scalability, both in terms of storage space and query processing time. Deductive database systems, on the other hand, require far less storage space since they derive new knowledge by applying inference rules. The challenge is how to efficiently obtain the required derivations, compared to having them in explicit form. In this study, we concentrate on a set of predefined inference rules for subsumption and disjointness relations, including their negations. We use compact data structures to store the facts and provide algorithms to support each type of relation, minimizing even further the storage space requirements. Our experimental findings demonstrate the feasibility of this approach, which not only saves space but is often faster than a baseline that uses well‐known graph traversal algorithms implemented on top of a traditional adjacency list representation to derive the relations.

Genotype Representation Graphs: Enabling Efficient Analysis of Biobank-Scale Data.

Computational analysis of a large number of genomes requires a data structure that can represent the dataset compactly while also enabling efficient operations on variants and samples. Current practice is to store large-scale genetic polymorphism data using tabular data structures and file formats, where rows and columns represent samples and genetic variants. However, encoding genetic data in such formats has become unsustainable. For example, the UK Biobank polymorphism data of 200,000 phased whole genomes has exceeded 350 terabytes (TB) in Variant Call Format (VCF), cumbersome and inefficient to work with. To mitigate the computational burden, we introduce the Genotype Representation Graph (GRG), an extremely compact data structure to losslessly present phased whole-genome polymorphisms. A GRG is a fully connected hierarchical graph that exploits variant-sharing across samples, leveraging ideas inspired by Ancestral Recombination Graphs. Capturing variant-sharing in a multitree structure compresses biobank-scale human data to the point where it can fit in a typical server's RAM (5-26 gigabytes (GB) per chromosome), and enables graph-traversal algorithms to trivially reuse computed values, both of which can significantly reduce computation time. We have developed a command-line tool and a library usable via both C++ and Python for constructing and processing GRG files which scales to a million whole genomes. It takes 160GB disk space to encode the information in 200,000 UK Biobank phased whole genomes as a GRG, more than 13 times smaller than the size of compressed VCF. We show that summaries of genetic variants such as allele frequency and association effect can be computed on GRG via graph traversal that runs significantly faster than all tested alternatives, including vcf.gz, PLINK BED, tree sequence, XSI, and Savvy. Furthermore, GRG is particularly suitable for doing repeated calculations and interactive data analysis. We anticipate that GRG-based algorithms will improve the scalability of various types of computation and generally lower the cost of analyzing large genomic datasets.

Exploring the Potential of k^n-tree for Efficient Representation of n-ary Relations

The objective of this experimental study was to investigate the scalability of -trees, a compact data structure designed for representing -ary relations, compared to a baseline based on a plain representation of adjacency lists. A literature review of compact data structures was conducted, focusing on -trees and their potential for efficient -ary data representation. To assess scalability, experiments comparing -trees performance against the baseline using set intersection as a benchmark were conducted. Results demonstrated superior -trees scalability in terms of time and memory, especially for high-dimensional and clustered datasets. On average, -trees were eight times faster and consumed times less memory than the baseline. The study also analyzed the impact of the order parameter on performance, revealing a trade-off between space efficiency and query time. This study provides valuable insights into the practical applicability of -trees for managing and querying high-dimensional data.

Dynamic compact data structure for temporal reachability with unsorted contact insertions

Abstract Temporal graphs represent interactions between entities over time. Deciding whether entities can reach each other through temporal paths is useful for various applications such as in communication networks and epidemiology. Previous works have studied the scenario in which addition of new interactions can happen at any point in time. A known strategy maintains, incrementally, a Timed Transitive Closure using a dynamic data structure composed of $O(n^{2})$ binary search trees containing non-nested time intervals. However, space usage for storing these trees grows rapidly as more interactions are inserted. In this paper, we introduce a compact data structure that represents each tree as two dynamic bit-vectors. Furthermore, we present two variants of this data structure: one representing bits in dynamic bit-vectors explicitly and the other representing only the active bits by encoding their consecutive distances. In our experiments, we observed that our data structure improves space usage while having similar time performance for incremental updates when comparing with the previous strategy. The first variant of our data structure gives the best space improvement when constructing Time Transitive Closures for temporally dense temporal graphs, and the second variant uses less space for temporally sparse temporal graphs.

Scalable de novo classification of antibiotic resistance of Mycobacterium tuberculosis.

World Health Organization estimates that there were over 10 million cases of tuberculosis (TB) worldwide in 2019, resulting in over 1.4 million deaths, with a worrisome increasing trend yearly. The disease is caused by Mycobacterium tuberculosis (MTB) through airborne transmission. Treatment of TB is estimated to be 85% successful, however, this drops to 57% if MTB exhibits multiple antimicrobial resistance (AMR), for which fewer treatment options are available. We develop a robust machine-learning classifier using both linear and nonlinear models (i.e. LASSO logistic regression (LR) and random forests (RF)) to predict the phenotypic resistance of Mycobacterium tuberculosis (MTB) for a broad range of antibiotic drugs. We use data from the CRyPTIC consortium to train our classifier, which consists of whole genome sequencing and antibiotic susceptibility testing (AST) phenotypic data for 13 different antibiotics. To train our model, we assemble the sequence data into genomic contigs, identify all unique 31-mers in the set of contigs, and build a feature matrix M, where M[i, j] is equal to the number of times the ith 31-mer occurs in the jth genome. Due to the size of this feature matrix (over 350 million unique 31-mers), we build and use a sparse matrix representation. Our method, which we refer to as MTB++, leverages compact data structures and iterative methods to allow for the screening of all the 31-mers in the development of both LASSO LR and RF. MTB++ is able to achieve high discrimination (F-1 >80%) for the first-line antibiotics. Moreover, MTB++ had the highest F-1 score in all but three classes and was the most comprehensive since it had an F-1 score >75% in all but four (rare) antibiotic drugs. We use our feature selection to contextualize the 31-mers that are used for the prediction of phenotypic resistance, leading to some insights about sequence similarity to genes in MEGARes. Lastly, we give an estimate of the amount of data that is needed in order to provide accurate predictions. The models and source code are publicly available on Github at https://github.com/M-Serajian/MTB-Pipeline.

A lightweight packet forwarding verification in SDN using sketch

By decoupling the control plane and the data plane, Software Defined Networking (SDN) has reshaped the ossified network architecture and improved the programmability of the network. However, SDN is also susceptible to malicious injection, tampering, dropping and hijacking attacks against forwarding packets, and the SDN architecture cannot perceive the real behavior of switches in the data plane. Packet forwarding verification and exception localization are recognized as effective and promising methods, enabling reliable packet delivery in the data plane and allowing the controller in SDN to identify abnormal links. While existing mechanisms embed the linear-scale cryptographic tags into the packet header space as the transmission path lengthens to realize packet verification and exception localization, which cannot achieve a feasible tradeoff between efficiency and security. Leveraging the central controllability of SDN, we propose a lightweight packet forwarding verification mechanism. This mechanism splits the runtime of a flow into consecutive epochs by address hopping. The switches in the data plane forward packets based on hopping address, collecting the information of packets forwarded in a compact data structure, namely traffic sketch. The controller verifies traffic sketches and localizes exception in the epoch. We further prototype the proposed mechanism based on simulation network. The analysis and experiments demonstrate that the cost of proposed scheme is lower than the similar mechanisms, introducing no more than 9 % additional forwarding delay and less than 8 % throughput degradation.

Parallel Topology-aware Mesh Simplification on Terrain Trees

We address the problem of performing a topology-aware simplification algorithm on a compact and distributed data structure for triangle meshes, the Terrain trees. Topology-aware operators have been defined to coarsen a Triangulated Irregular Network (TIN) without affecting the topology of its underlying terrain, i.e., without modifying critical features of the terrain, such as pits, saddles, peaks, and their connectivity. However, their scalability is limited for large-scale meshes. Our proposed algorithm uses a batched processing strategy to reduce both the memory and time requirements of the simplification process, and thanks to the spatial decomposition on the basis of Terrain trees, it can be easily parallelized. Also, since a Terrain tree after the simplification process becomes less compact and efficient, we propose an efficient post-processing step for updating hierarchical spatial decomposition. Our experiments on real-world TINs, derived from topographic and bathymetric LiDAR data, demonstrate the scalability and efficiency of our approach. Specifically, topology-aware simplification on Terrain trees uses 40% less memory and half the time compared to the most compact and efficient connectivity-based data structure for TINs. Furthermore, the parallel simplification algorithm on the Terrain trees exhibits a 12× speedup with an OpenMP implementation. The quality of the output mesh is not significantly affected by the distributed and parallel simplification strategy of Terrain trees, and we obtain similar quality levels compared to the global baseline method.

Classic distance join queries using compact data structures

Distance-based Join Queries (DJQs) have multiple applications in spatial databases, Geographic Information Systems, and other areas. The K Closest Pairs Query (KCPQ) and the ε Distance Join Query (εDJQ) are well-known DJQs that have been widely studied and can be solved using plane-sweep techniques, which are efficient but must keep the whole datasets in main memory. In this work, we propose DJQ algorithms that work with data represented using a k2-tree, a compact data structure for binary grids. Our algorithms solve KCPQ and εDJQ queries, as well as several window-constrained variants, taking advantage of the indexing capabilities of k2-trees to efficiently answer queries without the need to decompress the data. Our experimental evaluation with large datasets shows that k2-tree algorithms are up to 5 times faster than plane-sweep algorithms in KCPQ, and 5–30 times faster in εDJQ. In variants that are window-constrained, our algorithms are competitive in most scenarios and faster for large windows. Additionally, our algorithms are not very affected by the distribution of the data and yield much more predictable query times, showing up to 30 times smaller variance in query times than plane sweep, depending on the location of the query window.

From CountMin to Super kJoin Sketches for Flow Spread Estimation

Real-time data stream processing is key to many Internet applications ranging from e-commerce, social networks, to network monitoring. Sketches (compact data structures) are often used to track large, high-rate streams and estimate their statistics. However, the traditional model for measuring flow size, e.g., number of packets in each flow, is unsuitable for more sophisticated applications that require flow spreads, i.e., number of distinct elements in each flow. When there are numerous flows, most existing work modifies CountMin sketches — which were designed to measure flow size — for measuring flow spread. They inherit a similar design and a commonly-used min operation from the CountMin sketches. This paper casts doubt on such a solution path, arguing that the inherited sketch design and the min operation were both intended for a different purpose and are not the best for flow spread measurement. Replacing the min operation with new position-aware operations, we propose the kJoin sketches that achieve much better average accuracy. We mathematically derive the estimation error bound. We also apply our sketches for network-wide measurement. Trace-driven experiments on software/GPU/FPGA platforms demonstrate our work significantly improves estimation accuracy, streaming rate (recording throughput), and query throughput, compared to the state-of-the-art.

Efficient non-isomorphic graph enumeration algorithms for several intersection graph classes

Intersection graphs are well-studied in the area of graph algorithms. Some intersection graph classes are known to have algorithms enumerating all unlabeled graphs by reverse search. Since these algorithms output graphs one by one and the numbers of graphs in these classes are vast, they work only for a small number of vertices. Binary decision diagrams (BDDs) are compact data structures for various types of data and useful for solving optimization and enumeration problems. This study proposes enumeration algorithms for five intersection graph classes, which admit O(n)-bit string representations for their member graphs. Our algorithm for each class enumerates all unlabeled graphs with n vertices over BDDs representing the binary strings in time polynomial in n. Moreover, our algorithms are extended so that it enumerates those with constraints on the maximum (bi)clique size and/or the number of edges.

The Ring: Worst-case Optimal Joins in Graph Databases using (Almost) No Extra Space

We present an indexing scheme for triple-based graphs that supports join queries in worst-case optimal (wco) time within compact space. This scheme, called a ring , regards each triple as a cyclic string of length 3. Each rotation of the triples is lexicographically sorted and the values of the last attribute are stored as a column, so we obtain the order of the next column by stably re-sorting the triples by its attribute. We show that, by representing the columns with a compact data structure called a wavelet tree, this ordering enables forward and backward navigation between columns without needing pointers. These wavelet trees further support wco join algorithms and cardinality estimations for query planning. While traditional data structures such as B-Trees, tries, and so on, require 6 index orders to support all possible wco joins over triples, we can use one ring to index them all. This ring replaces the graph and uses only sublinear extra space, thus supporting wco joins in almost no space beyond storing the graph itself. Experiments querying a large graph (Wikidata) in memory show that the ring offers nearly the best overall query times while using only a small fraction of the space required by several state-of-the-art approaches. We then turn our attention to some theoretical results for indexing tables of arity d higher than 3 in such a way that supports wco joins. While a single ring of length d no longer suffices to cover all d ! orders, we need much fewer rings to index them all: O (2 d ) rings with a small constant. For example, we need 5 rings instead of 120 orders for d =5. We show that our rings become a particular case of what we dub order graphs , whose nodes are attribute orders and where stably sorting by some attribute leads us from an order to another, thereby inducing an edge labeled by the attribute. The index is then the set of columns associated with the edges, and a set of rings is just one possible graph shape. We show that other shapes, like for example a single ring instead of several ones of length d , can lead us to even smaller indexes, and that other more general shapes are also possible. For example, we handle d =5 attributes within space equivalent to 4 rings.

Worst-Case-Optimal Similarity Joins on Graph Databases

We extend the concept of worst-case optimal equijoins in graph databases to the case where some nodes are required to be within the k-nearest neighbors (kNN) of others under some similarity function. We model the problem by superimposing the database graph with the kNN graph and show that a variant of Leapfrog TrieJoin (LTJ) implemented over a compact data structure called the Ring can be seamlessly extended to integrate similarity clauses with the equijoins in the LTJ query process, retaining worst-case optimality in many relevant cases. Our experiments on a benchmark that combines Wikidata and IMGpedia show that our enhanced LTJ algorithm outperforms by a considerable margin a baseline that first applies classic LTJ and then completes the query by applying the similarity predicates. The difference is more pronounced on queries where the similarity clauses are more densely connected to the query, becoming of an order of magnitude in some cases.

Dichotomy Graph Sketch: Summarizing Graph Streams with High Accuracy Based on Deep Learning

In many applications, data streams are indispensable to describe the relationships between nodes in networks, such as social networks, computer networks, and hyperlink networks. Fundamentally, a graph stream is a dynamic representation of a graph, which is usually composed of a sequence of edges, where each edge is represented by two endpoints and a weight. As a result of its large volume and highly dynamic nature, several graph sketches were proposed for the purposes of summarizing large-scale graph streams and enabling fast query processing. By using a compact data structure with hash functions, the graph sketches sequentially store the edges. Nevertheless, the existing graph sketches suffer from low performance on graph query tasks as a result of unpredictable collisions between heavy edges and light edges. To store heavy edges and light edges, this paper introduces a novel learning-based Dichotomy Graph Sketch (DGS) mechanism that uses two separate graph sketches, a heavy sketch and a light sketch. During a graph stream session, DGS obtains heavy edges and light edges, and uses these edges as training samples for a deep neural network (DNN) based binary classifier. The DNN-based classifier is then used to determine whether the upcoming edges are heavy or not. We will store the edges that are classified as heavy edges in the heavy sketch, and those that are classified as light edges in the light sketch. By combining the learnable classifier and Dichotomy Graph Sketches, the proposed mechanism resolves the hashing collision problem in conventional graph sketches and significantly improves graph query accuracy. The DGS algorithm outperforms the state-of-the-art graph sketches in a variety of graph query tasks based on extensive experiments that were conducted on four real-world graph stream datasets.

Efficient Temporal Butterfly Counting and Enumeration on Temporal Bipartite Graphs

Bipartite graphs characterize relationships between two different sets of entities, like actor-movie, user-item, and author-paper. The butterfly, a 4-vertices 4-edges (2,2)-biclique, is the simplest cohesive motif in a bipartite graph and is the fundamental component of higher-order substructures. Counting and enumerating the butterflies offer significant benefits across various applications, including fraud detection, graph embedding, and community search. While the corresponding motif, the triangle, in the unipartite graphs has been widely studied in both static and temporal settings, the extension of butterfly to temporal bipartite graphs remains unexplored. In this paper, we investigate the temporal butterfly counting and enumeration problem: count and enumerate the butterflies whose edges establish following a certain order within a given duration. Towards efficient computation, we devise a non-trivial baseline rooted in the state-of-the-art butterfly counting algorithm on static graphs, further, explore the intrinsic property of the temporal butterfly, and develop a new optimization framework with a compact data structure and effective priority strategy. The time complexity is proved to be significantly reduced without compromising on space efficiency. In addition, we generalize our algorithms to practical streaming settings and multi-core computing architectures. Our extensive experiments on 11 large-scale real-world datasets demonstrate the efficiency and scalability of our solutions.

A GPU-Accelerated Framework for Path-Based Timing Analysis

As a key routine in Static Timing Analysis (STA), Path-based Analysis (PBA) plays a very important role in refining the critical path report by reducing excessive slack pessimism. PBA is also well known for its long execution time, which makes it a hot topic for parallel computing in the STA community. However, nearly all of the parallel PBA algorithms are restricted to CPU architectures, which greatly limits their scalability. To achieve a new performance milestone on PBA, we must leverage the high throughput computing in the graphics processing unit (GPU). Therefore, in this work, we propose a new GPU-accelerated PBA framework which contains compact data structures and highly efficient kernels. By integrating with GPU-accelerated preprocessing steps, our framework can also effectively handle extensive critical path constraints. Besides, we highlight many optimization techniques that can overcome the execution bottleneck and further boost the performance. In experiments, we demonstrate 543× speed-up compared to the state-of-the-art PBA algorithm on the design with 1.6 million gates, which outperforms 25–45× over the state-of-the-art parallel PBA algorithm on 40 CPU cores. A fully optimized framework can achieve 3–5× speed-up on top of that.

Embedded Parallel Implementation of LDPC Decoder for Ultra-Reliable Low-Latency Communications

Ultra-reliable low-latency communications, URLLC, are designed for applications such as self-driving cars and telesurgery requiring a response in milliseconds and are very sensitive to transmission errors. To match the computational complexity of LDPC decoding algorithms to URLLC applications on IoT devices having very limited computational resources, this paper presents a new parallel and low-latency software implementation of the LDPC decoder. First, a decoding algorithm optimization and a compact data structure are proposed. Next, a parallel software implementation is performed on ARM multicore platforms in order to evaluate the latency of the proposed optimization. The synthesis results highlight a reduction in the memory size requirement by 50% and a three-time speedup in terms of processing time when compared to previous software decoder implementations. The reached decoding latency on the parallel processing platform is 150 μs for 288 bits with a bit error ratio of 3.410–9.

Targeted mining of contiguous sequential patterns

In recent years, sequential pattern mining (SPM) has been applied in many domains, including recommender systems, fraud detection, and other related industries. Target-oriented SPM (TSPM) has been proposed to improve the interpretability of SPM by addressing the issue that too many irrelevant or meaningless sequential patterns are often discovered, which facilitates data analysis. Nevertheless, current TSPM research does not consider contiguity to identify sequential patterns, despite the fact that it is an important feature of sequence data. Contiguity has significant implications in applications such as bioinformatics and network intrusion detection, as it can aid the detection of periodic patterns and anomalous behaviors. Therefore, this paper proposes a novel problem of target-oriented contiguous sequential pattern mining (TCSPM) and introduces two algorithms, called TCSPM and TCSPM+. TCSPM utilizes a compact data structure called sequence chain and relies on a sequence chain pruning strategy to reduce the search space. In addition, TCSPM applies a reverse matching technique to quickly search for targeted contiguous sequences. TCSPM+ is an optimized version of TCSPM that utilizes sequence segmentation operations and a query sequence pruning strategy to further improve performance. Extensive experiments on both real and synthetic datasets demonstrate that TCSPM is an efficient algorithm for discovering targeted contiguous sequential patterns (TCSPs). TCSPM+ substantially enhances the performance of TCSPM in terms of runtime, memory usage, and scalability, outperforming TCSPM by an order of magnitude on some datasets.

Efficient approach for mining high-utility patterns on incremental databases with dynamic profits

High-utility itemset mining (HUIM) is one of the heavily studied fields of data mining, which is due to its high compatibility with real-world applications. HUIM is a process of extracting a complete set of interesting patterns by considering the importance and the quantity of each item. Incremental high-utility itemset mining (IHUIM) further increases the compatibility of HUIM by mining from a data stream instead of a static database. It does this by mining interesting patterns in a single database scan and storing prior knowledge in a compact data structure. However, there is a critical limitation, because the importance of each item has to be guaranteed to remain unchanged throughout the data stream. This limitation hinders the compatibility of IHUIM to the real-world applications, because the importance of an item changes from time to time in the real world. Conventional IHUIM approaches consequently have to use outdated information when mining interesting patterns. This paper proposes a novel problem of mining high-utility itemsets in an incremental and dynamic profit environment in order to account for the fluctuation of the item's importance. Furthermore, a novel approach to mining patterns in this type of environment is introduced using an efficient list structure and tight upper bounds. Experiments on real and synthetic datasets show that the proposed approach performs well compared to the state-of-the-art approaches in terms of the runtime and memory usage, and it scales better than the other approaches.

String Indexing with Compressed Patterns

Given a string S of length n , the classic string indexing problem is to preprocess S into a compact data structure that supports efficient subsequent pattern queries. In this article, we consider the basic variant where the pattern is given in compressed form and the goal is to achieve query time that is fast in terms of the compressed size of the pattern. This captures the common client-server scenario, where a client submits a query and communicates it in compressed form to a server. Instead of the server decompressing the query before processing it, we consider how to efficiently process the compressed query directly. Our main result is a novel linear space data structure that achieves near-optimal query time for patterns compressed with the classic Lempel-Ziv 1977 (LZ77) compression scheme. Along the way, we develop several data structural techniques of independent interest, including a novel data structure that compactly encodes all LZ77 compressed suffixes of a string in linear space and a general decomposition of tries that reduces the search time from logarithmic in the size of the trie to logarithmic in the length of the pattern.