Index Data Structure Research Articles

Skip index: Supporting efficient inter-block queries and query authentication on the blockchain

Decentralized applications, the driving force behind the new Web3 paradigm, require continuous access to blockchain data. Their adoption, however, is hindered by the constantly increasing size of blockchains and the sequential scan nature of their read operations, which introduce a clear inefficiency bottleneck. Also, the growing amount of data recorded on the blockchain makes resource-constrained light nodes dependent on untrusted full nodes for fetching information, with a consequent need for query authentication protocols ensuring result integrity. Motivated by these reasons, in this paper we propose the skip index, an indexing data structure that allows users to quickly retrieve information simultaneously from multiple blocks of a blockchain. Our solution is also designed to be used as an authenticated data structure to guarantee the integrity of query results for light nodes. We discuss the theoretical properties of skip indices, propose efficient algorithms for their construction and querying, and detail their computational complexity. Finally, we assess the effectiveness of our proposal through an experimental evaluation on the Ethereum blockchain. As a reference use case, we focus on the popular CryptoKitties application and simulate a scenario where users seek to retrieve the events generated by the service. Our experimental results suggest that the use of skip indices offers a constant multiplicative speedup, thanks to search times that are at most logarithmic within a chosen search window. This allows to reduce the number of visited blocks by up to two orders of magnitude if compared to the naive sequential approach currently in use.

Designing a Multi-Level Caching Forward Index for Recommendation System Services

This paper introduces a forward index based on microservices, which effectively addresses the issue of forward read amplification in recommendation systems through a multi-layer caching mechanism. This allows the forward service to handle a large number of concurrent requests with limited resources while ensuring service stability and final value rate. The paper provides a detailed description of the forward index's data structure design, traffic penetration calculation, and overall service architecture design, dividing the system into three parts: SDK, proxy service, and storage, corresponding to data query, data processing, and data storage respectively. In the SDK design, three components are employed: Cache Slot, synchronous queue, and timer, supporting synchronous, asynchronous, and flexible request forms. The proxy service design handles heterogeneous data and online computation, supporting batch and asynchronous designs, and provides a dynamic data protocol to adapt to the characteristic needs of different microservices. The storage layer design considers performance, cost, and maintainability, utilizing a Cache & SSD model to cope with high-concurrency scenarios.

Fast and space-efficient taxonomic classification of long reads with hierarchical interleaved XOR filters.

Metagenomic long-read sequencing is gaining popularity for various applications, including pathogen detection and microbiome studies. To analyze the large data created in those studies, software tools need to taxonomically classify the sequenced molecules and estimate the relative abundances of organisms in the sequenced sample. Because of the exponential growth of reference genome databases, the current taxonomic classification methods have large computational requirements. This issue motivated us to develop a new data structure for fast and memory-efficient querying of long reads. Here, we present Taxor as a new tool for long-read metagenomic classification using a hierarchical interleaved XOR filter data structure for indexing and querying large reference genome sets. Taxor implements several k-mer-based approaches, such as syncmers, for pseudoalignment to classify reads and an expectation-maximization algorithm for metagenomic profiling. Our results show that Taxor outperforms state-of-the-art tools regarding precision while having a similar recall for long-read taxonomic classification. Most notably, Taxor reduces the memory requirements and index size by >50% and is among the fastest tools regarding query times. This enables real-time metagenomics analysis with large reference databases on a small laptop in the field.

DSTree: A Spatio-Temporal Indexing Data Structure for Distributed Networks

DSTree: A Spatio-Temporal Indexing Data Structure for Distributed Networks

Compression algorithm for colored de Bruijn graphs

A colored de Bruijn graph (also called a set of k-mer sets), is a set of k-mers with every k-mer assigned a set of colors. Colored de Bruijn graphs are used in a variety of applications, including variant calling, genome assembly, and database search. However, their size has posed a scalability challenge to algorithm developers and users. There have been numerous indexing data structures proposed that allow to store the graph compactly while supporting fast query operations. However, disk compression algorithms, which do not need to support queries on the compressed data and can thus be more space-efficient, have received little attention. The dearth of specialized compression tools has been a detriment to tool developers, tool users, and reproducibility efforts. In this paper, we develop a new tool that compresses colored de Bruijn graphs to disk, building on previous ideas for compression of k-mer sets and indexing colored de Bruijn graphs. We test our tool, called ESS-color, on various datasets, including both sequencing data and whole genomes. ESS-color achieves better compression than all evaluated tools and all datasets, with no other tool able to consistently achieve less than 44% space overhead. The software is available at http://github.com/medvedevgroup/ESSColor.

A fast DBSCAN algorithm using a bi-directional HNSW index structure for big data

A fast DBSCAN algorithm using a bi-directional HNSW index structure for big data

WARM-tree: Making Quadtrees Write-efficient and Space-economic on Persistent Memories

Recently, the value of data has been widely recognized, which highlights the significance of data-centric computing in diversified application scenarios. In many cases, the data are multidimensional, and the management of multidimensional data often confronts greater challenges in supporting efficient data access operations and guaranteeing the space utilization. On the other hand, while many existing index data structures have been proposed for multidimensional data management, however, their designs are not fully optimized for modern nonvolatile memories, in particular the byte-addressable persistent memories. As a result, they might undergo serious access performance degradation or fail to guarantee space utilization. This observation motivates the redesigning of index data structures for multidimensional point data on modern persistent memories, such as the phase-change memory. In this work, we present the WARM-tree , a m ultidimensional t ree for r educing the w rite a mplification effect, for multidimensional point data. In our evaluation studies, as compared to the bucket PR quadtree and R*-tree, the WARM-tree can provide any worst-case space utilization guarantees in the form of \(\frac{m-1}{m}\) ( m ∈ ℤ^+) and effectively reduces the write traffic of key insertions by up to 48.10% and 85.86%, respectively, at the price of degraded average space utilization and prolonged latency of query operations. This suggests that the WARM-tree is a potential multidimensional index structure for insert-intensive workloads.

Compression Algorithm for Colored de Bruijn Graphs.

A colored de Bruijn graph (also called a set of k-mer sets), is a set of k-mers with every k-mer assigned a set of colors. Colored de Bruijn graphs are used in a variety of applications, including variant calling, genome assembly, and database search. However, their size has posed a scalability challenge to algorithm developers and users. There have been numerous indexing data structures proposed that allow to store the graph compactly while supporting fast query operations. However, disk compression algorithms, which do not need to support queries on the compressed data and can thus be more space-efficient, have received little attention. The dearth of specialized compression tools has been a detriment to tool developers, tool users, and reproducibility efforts. In this paper, we develop a new tool that compresses colored de Bruijn graphs to disk, building on previous ideas for compression of k-mer sets and indexing colored de Bruijn graphs. We test our tool, called ESS-color, on various datasets, including both sequencing data and whole genomes. ESS-color achieves better compression than all evaluated tools and all datasets, with no other tool able to consistently achieve less than 44% space overhead.

Efficient taxa identification using a pangenome index.

Tools that classify sequencing reads against a database of reference sequences require efficient index data-structures. The r-index is a compressed full-text index that answers substring presence/absence, count, and locate queries in space proportional to the amount of distinct sequence in the database: [Formula: see text] space, where r is the number of Burrows-Wheeler runs. To date, the r-index has lacked the ability to quickly classify matches according to which reference sequences (or sequence groupings, i.e., taxa) a match overlaps. We present new algorithms and methods for solving this problem. Specifically, given a collection D of d documents, [Formula: see text] over an alphabet of size σ, we extend the r-index with [Formula: see text] additional words to support document listing queries for a pattern [Formula: see text] that occurs in [Formula: see text] documents in D in [Formula: see text] time and [Formula: see text] space, where w is the machine word size. Applied in a bacterial mock community experiment, our method is up to three times faster than a comparable method that uses the standard r-index locate queries. We show that our method classifies both simulated and real nanopore reads at the strain level with higher accuracy compared with other approaches. Finally, we present strategies for compacting this structure in applications in which read lengths or match lengths can be bounded.

A Bin-Based Indexing for Scalable Range Join on Genomic Data.

Range-join is an operation for finding overlaps in interval-form genomic data. Range-join is widely used in various genome analysis processes such as annotation, filtering and comparison of variants in whole-genome and exome analysis pipelines. The quadratic complexity of current algorithms with sheer data volume has surged the design challenges. Existing tools have limitations on algorithm efficiency, parallelism, scalability and memory consumption. This paper proposes BIndex, a novel bin-based indexing algorithm and its distributed implementation to attain high throughput range-join processing. BIndex features near-constant search complexity while the inherently parallel data structure facilitates exploitation of parallel computing architectures. Balanced partitioning of dataset further enables scalability on distributed frameworks. The implementation on Message Passing Interface shows upto 933.5x speedup in comparison to state-of-the-art tools. Parallel nature of BIndex further enables GPU-based acceleration with 3.72x speedup than CPU implementations. The add-in modules for Apache Spark provides upto 4.65x speedup than the previously best available tool. BIndex supports wide variety of input and output formats prevalent in bioinformatics community and the algorithm is easily extendable to streaming data in recent Big Data solutions. Furthermore, the index data structure is memory-efficient and consumes upto two orders-of-magnitude lesser RAM, while having no adverse effect on speedup.

A survey on the efficiency, reliability, and security of data query in blockchain systems

With the emergence of digital currencies, blockchain systems have been committed to the efficient and trusted storage of data, building decentralized, tamper-proof, persistent, and anonymous digital ledger. Researchers have devoted considerable attention to query methods on blockchain platforms, leading to numerous developments and advancements. There is an imperative need for a comprehensive investigation of all these efforts, as well as their most recent progress and results. This study strives to provide a comprehensive and thorough survey of notable works and recent progress related to query technologies and theories. In general, blockchain query methods rely on utilizing distributed databases, data indexing structures, and cryptographic algorithms to achieve efficient, verifiable, and secure queries. Beyond the above, we examine existing issues of blockchain query technologies and theories on query efficiency, reliability, and security. In the end, this work concludes with a summarization on typical scenarios of blockchain query schemes, as well as a discussion of future challenges to be addressed in future research.

Learned Sorted Table Search and Static Indexes in Small-Space Data Models

Machine-learning techniques, properly combined with data structures, have resulted in Learned Static Indexes, innovative and powerful tools that speed up Binary Searches with the use of additional space with respect to the table being searched into. Such space is devoted to the machine-learning models. Although in their infancy, these are methodologically and practically important, due to the pervasiveness of Sorted Table Search procedures. In modern applications, model space is a key factor, and a major open question concerning this area is to assess to what extent one can enjoy the speeding up of Binary Searches achieved by Learned Indexes while using constant or nearly constant-space models. In this paper, we investigate the mentioned question by (a) introducing two new models, i.e., the Learned k-ary Search Model and the Synoptic Recursive Model Index; and (b) systematically exploring the time–space trade-offs of a hierarchy of existing models, i.e., the ones in the reference software platform Searching on Sorted Data, together with the new ones proposed here. We document a novel and rather complex time–space trade-off picture, which is informative for users as well as designers of Learned Indexing data structures. By adhering to and extending the current benchmarking methodology, we experimentally show that the Learned k-ary Search Model is competitive in time with respect to Binary Search in constant additional space. Our second model, together with the bi-criteria Piece-wise Geometric Model Index, can achieve speeding up of Binary Search with a model space of 0.05% more than the one taken by the table, thereby, being competitive in terms of the time–space trade-off with existing proposals. The Synoptic Recursive Model Index and the bi-criteria Piece-wise Geometric Model complement each other quite well across the various levels of the internal memory hierarchy. Finally, our findings stimulate research in this area since they highlight the need for further studies regarding the time–space relation in Learned Indexes.

A New Outlier Detection Algorithm Based on Fast Density Peak Clustering Outlier Factor

Outlier detection is an important field in data mining, which can be used in fraud detection, fault detection, and other fields. This article focuses on the problem where the density peak clustering algorithm needs a manual parameter setting and time complexity is high; the first is to use the k nearest neighbors clustering algorithm to replace the density peak of the density estimate, which adopts the KD-Tree index data structure calculation of data objects k close neighbors. Then it adopts the method of the product of density and distance automatic selection of clustering centers. In addition, the central relative distance and fast density peak clustering outliers were defined to characterize the degree of outliers of data objects. Then, based on fast density peak clustering outliers, an outlier detection algorithm was devised. Experiments on artificial and real data sets are performed to validate the algorithm, and the validity and time efficiency of the proposed algorithm are validated when compared to several conventional and innovative algorithms.

Subgraph Query Matching in Multi-Graphs Based on Node Embedding

This paper presents an efficient algorithm for matching subgraph queries in a multi-graph based on features-based indexing techniques. The KD-tree data structure represents these nodes’ features, while the set-trie index data structure represents the multi-edges to make queries effectively. The vertex core number, triangle number, and vertex degree are the eight features’ main features. The densest vertex in the query graph is extracted based on these main features. The proposed model consists of two phases. The first phase’s main idea is that, for the densest extracted vertex in the query graph, find the density similar neighborhood structure in the data graph. Then find the k-nearest neighborhood query to obtain the densest subgraph. The second phase for each layer graph, mapping the vertex to feature vector (Vertex Embedding), improves the proposed model. To reduce the node-embedding size to be efficient with the KD-tree, indexing a dimension reduction, the principal component analysis (PCA) method is used. Furthermore, symmetry-breaking conditions will remove the redundancy in the generated pattern matching with the query graph. In both phases, the filtering process is applied to minimize the number of candidate data nodes of the initiate query vertex. The filtering process is applied to minimize the number of candidate data nodes of the initiate query vertex. Finally, testing the effect of the concatenation of the structural features (orbits features) with the meta-features (summary of general, statistical, information-theoretic, etc.) for signatures of nodes on the model performance. The proposed model is tested over three real benchmarks, multi-graph datasets, and two randomly generated multi-graph datasets. The results agree with the theoretical study in both random cliques and Erdos random graph. The experiments showed that the time efficiency and the scalability results of the proposed model are acceptable.

Nested spatial data structures for optimal indexing of LiDAR data

In this paper we present a flexible framework for creating spatial data structures to manage LiDAR point clouds in the context of spatial big data. For this purpose, standard approaches typically include the use of a single data structure to index point clouds. Some of them use a hybrid two-tier solution to optimize specific application purposes such as storage or rendering. In this article we introduce a meta-structure that can have unlimited depth and a custom, user-defined combination of nested structures, such as grids, quadtrees, octrees, or kd-trees. With our approach, the out-of-core indexing of point clouds can be adapted to different types of datasets, taking into account the spatial distribution of the data. Therefore, the most suitable spatial indexing can be achieved for any type of dataset, from small TLS-based scenes to planetary-scale ALS-based scenes. This approach allows us to work with overlapping datasets of different resolutions from different acquisition technologies in the same structure.

On the Efficiency of the Advancing-Front Surface Mesh Generation Algorithm

An efficient advancing front algorithm is presented to create high-quality surface elements on CAD models. Various novel schemes are developed to improve the efficiency of the algorithm while not sacrificing its robustness and the resulting mesh quality. New features include a fast projection scheme that can perform robustly in case of badly defined surfaces, a procedure that computes the ideal point by an approach that can preserve the advantageous features of both the mapping and direct methods, and an octree data structure that can speed up the local search of all existing front nodes about the ideal point. To our knowledge, the proposed algorithm is one of the fastest algorithms for surface mesh generation. On an AMD Ryzen 5 3600 machine, the sequential version of the proposed algorithm can create a quality mesh at a speed of up to 100,000 elements per second. This speed is more than 3 times higher than a state-of-the-art code can achieve. Another advantage of the proposed algorithm is its ability to maintain its high efficiency when millions or more elements are created on one single surface. In this circumstance, the efficiency of some state-of-the-arts codes slows down by one or two orders. After parallelized based on the taskflow approach, the proposed algorithm can create more than 1.6 million elements on real-world configurations in 7 s by using 6 computer cores. This speed is higher than a state-of-the-art commercial code can achieve by one order. • An efficient surface mesh generator for large-scale triangular mesh generation of industrial CAD models, which is far faster than existing schemes. • The nonlinear calculation such as orthogonal projection and ideal point calculation is comprehensively improved. • Traditional surface mesh generation methods such as direct method and mapping method are combined. • The design of spatial index data structure (octree and AD tree) makes the algorithm have strong scalability.

Practical Implementation of Encoding Range Top-2 Queries

Abstract We design a practical variant of an encoding for range Top-2 query (RT2Q) and evaluate its performance. Given an array $A[1,n]$ of $n$ elements from a total order, the range Top-2 encoding problem is to construct a data structure that answers ${\textsf{RT2Q}}{}$, which returns the positions of the first and second largest elements within a given range of $A$, without accessing the array $A$ at query time. We design the following two implementations: (i) an implementation based on an alternative representation of Davoodi et al.’s [Phil. Trans. Royal Soc. A, 2016] data structure, which supports queries efficiently. Experimental results show that our implementation is efficient in practice and gives improved time-space trade-offs compared with the indexing data structures (which keep the original array $A$ as part of the data structure) for range maximum queries. (ii) Another implementation based on Jo et al.’s ${\textsf{RT2Q}}{}$ encoding on $2 \times n$ array [CPM, 2016], which can be constructed in $O(n)$ time. We compare our encoding with Gawrychowski and Nicholson’s optimal encoding [ICALP, 2015] and show that in most cases, our encoding shows faster construction time while using a competitive space in practice.

Data Drops for Tunnel Information Modelling

AbstractIn the Architecture, Engineering and Construction (AEC) industry, as well as in the tunnelling domain, inter‐company processes between partners in different roles in large‐scale construction projects still exhibit great potential towards digitalisation. Thereby, information should be seamlessly shared between partners according to the Building Information Modelling (BIM) paradigm. Today, different types of artefacts (e.g., models, plans, documents, etc.) are shared at different points in time, which differ in terms of requirements, information content, as well as data formats. In this article, we extend and prototypically implement the concept of Data Drops to provide those artefacts in a digitalised form via a shared Data Drop management platform. For this purpose, we have developed a formal, well‐defined indexed data structure on a metadata level. This not only facilitates traceability, but also enables searching for specific meta‐information and provides a common view on Data Drops. In addition, a networked view between different drops can be provided. The approach is being evaluated on the use case of a real tunnel construction project.

KmerKeys: a web resource for searching indexed genome assemblies and variants.

K-mers are short DNA sequences that are used for genome sequence analysis. Applications that use k-mers include genome assembly and alignment. However, the wider bioinformatic use of these short sequences has challenges related to the massive scale of genomic sequence data. A single human genome assembly has billions of k-mers. As a result, the computational requirements for analyzing k-mer information is enormous, particularly when involving complete genome assemblies. To address these issues, we developed a new indexing data structure based on a hash table tuned for the lookup of short sequence keys. This web application, referred to as KmerKeys, provides performant, rapid query speeds for cloud computation on genome assemblies. We enable fuzzy as well as exact sequence searches of assemblies. To enable robust and speedy performance, the website implements cache-friendly hash tables, memory mapping and massive parallel processing. Our method employs a scalable and efficient data structure that can be used to jointly index and search a large collection of human genome assembly information. One can include variant databases and their associated metadata such as the gnomAD population variant catalogue. This feature enables the incorporation of future genomic information into sequencing analysis. KmerKeys is freely accessible at https://kmerkeys.dgi-stanford.org.

Accelerating minimap2 for long-read sequencing applications on modern CPUs.

Long-read sequencing is now routinely used at scale for genomics and transcriptomics applications. Mapping long reads or a draft genome assembly to a reference sequence is often one of the most time-consuming steps in these applications. Here we present techniques to accelerate minimap2, a widely used software for this task. We present multiple optimizations using single-instruction multiple-data parallelization, efficient cache utilization and a learned index data structure to accelerate the three main computational modules of minimap2: seeding, chaining and pairwise sequence alignment. These optimizations result in an up to 1.8-fold reduction of end-to-end mapping time of minimap2 while maintaining identical output.