Efficient Index Structure Research Articles

An index structure supporting efficient querying of historical transactions on the blockchain

Blockchain technology is increasingly being utilized in the field of digital assets, especially in asset tracking and traceability. In order to ensure the efficiency of asset tracking and traceability, it is necessary to improve the efficiency of querying historical transactions. However, the traditional database search method in the blockchain system is no longer applicable. Searching for historical transactions requires traversing the entire chain, which is very inefficient. Designing an efficient and safe index structure for querying historical transactions is an urgent problem that needs to be solved. While the Account Transaction Trace Chain (ATTC) improves the efficiency of querying historical transactions between two accounts, the long transaction chains of active accounts still pose challenges in terms of traversal time. To further enhance query efficiency, we propose a new index structure Account Transaction Subchain (ATSC), which connects transactions with the same account in the transaction chain through hash pointers to form a subchain, converting the query method from sequential traversal to based on subchain query. At the same time, the construction algorithm and query algorithm of ATSC are designed, and the simulation experiment results show that the index structure effectively improves query efficiency.

Backbone Index and GNN Models for Skyline Path Query Evaluation over Multi-cost Road Networks

Skyline path queries (SPQs) extend skyline queries to multi-dimensional networks, such as multi-cost road networks (MCRNs). Such queries return a set of non-dominated paths between two given network nodes. Despite the existence of extensive works on evaluating different SPQ variants, SPQ evaluation is still very inefficient due to the nonexistence of efficient index structures to support such queries. Existing index building approaches for supporting shortest-path query execution, when directly extended to support SPQs, use an unreasonable amount of space and time to build, making them impractical for processing large graphs. In this article, we propose a novel index structure, backbone index , and a corresponding index construction method that condenses an initial MCRN to multiple smaller summarized graphs with different granularity. We present efficient approaches to find approximate solutions to SPQs by utilizing the backbone index structure. Furthermore, considering making good use of historical query and query results, we propose two models, S kyline P ath G raph N eural N etwork (SP-GNN) and T ransfer SP-GNN (TSP-GNN), to support effective SPQ processing. Our extensive experiments on real-world large road networks show that the backbone index can support finding meaningful approximate SPQ solutions efficiently. The backbone index can be constructed in a reasonable time, which dramatically outperforms the construction of other types of indexes for road networks. As far as we know, this is the first compact index structure that can support efficient approximate SPQ evaluation on large MCRNs. The results on the SP-GNN and TSP-GNN models also show that both models can help get approximate SPQ answers efficiently.

SWIX: A Memory-efficient Sliding Window Learned Index

Data stream processing systems enable querying over sliding windows of streams of data. Efficient index structures for the streaming window are a crucial building block to enable querying the sliding window for operations such as aggregation and joins. This paper proposes SWIX, a novel memory-efficient learned index for sliding windows. Unlike conventional learned indexes that rely on tree structures to achieve logarithmic query cost, SWIX has a flat structure that uses substantially less memory and enables efficient query execution while having a low cost for index maintenance when inserting (and retraining). SWIX dynamically adapts itself to the real-time distribution shifts of data streams. SWIX outperforms existing indexes in terms of query execution time and memory footprint for workloads characterized by very frequent updates. Our results show that SWIX has a significantly smaller memory footprint than conventional, streaming, and learned indexes, using only 22% to 42% of the size compared to state-of-the-art approaches, yet outperforming them by up 1.2× to 1.6× on average (and up to 52×) in terms of query time, making it a space- and time-efficient method for indexing data streams. For concurrent learned indexes, Parallel SWIX can achieve up to 3.45× throughput with only 34% of memory consumption.

An efficient flattened index structure with lazy restructuring and hotness awareness

An efficient flattened index structure with lazy restructuring and hotness awareness

When Tree Meets Hash: Reducing Random Reads for Index Structures on Persistent Memories

Indexing structures are widely used in modern data-processing applications to support high-performance queries, and there are a variety of recent designs specifically optimized for the newly available persistent memory (PM). The primary focus of previous PM indexes is on reducing the expensive PM writes for persisting data. However, we find that in tree-based PM indexes, because of the smaller performance gap between writes and random reads on real PM devices, the read-intensive tree traversal phase dominates the overall latency. This observation calls for further optimizations on existing indexing structures for PM. In this paper, we propose Extendible Radix Tree (ERT), an efficient indexing structure for PM that significantly reduces tree heights to minimize random reads, while still maintaining fast in-node search speed. The key idea is to use extendible hashing for each node in a radix tree. This design allows us to have a relatively large fanout of the radix tree to keep the tree height small, and also to realize constant-time lookups within a node. Using extendible hashing also allows for incremental node modification without excessive writes during inserts and updates. Range queries are efficiently and robustly handled by enforcing partial ordering among the keys in the hash table of each node without introducing more hash collisions. Our experiments on both synthetic and real-world data sets demonstrate that ERT achieves up to 2.65×, 4.41×, and 2.43× speedups for search, insert, and range queries over the respectively state-of-the-art PM index.

HCFTL: A Locality-Aware Flash Translation Layer for Efficient Address Translation

In modern solid-state drives (SSDs), a flash translation layer (FTL) is the core engine performing the logical-to-physical address translations. However, the increasing capacity of SSDs requires a large DRAM space to hold the mapping table in a page-level FTL. Due to the limited size of built-in DRAM, existing FTL schemes selectively cache some active mapping entries in DRAM, while store the entire mapping table on flash. However, the low cache hit ratio introduced by the traditional mapping entry eviction policy degrades the access performance of SSDs. In this article, we propose a novel FTL, hot-clusterity FTL (HCFTL), to improve the hit ratio of cached mapping table (CMT) of SSDs with limited cache space. HCFTL clusters mapping entries, which are newly evicted from the cache, into dynamic translation pages (DTPs). Reading entries from DTPs will increase the CMT hit ratio due to spatial and temporal localities. Because the logical page numbers (LPNs) in a DTP are not consecutive, we introduce two different efficient index structures to speedup the lookup of mapping entries in DTPs. Furthermore, to make the DTP index highly accurate, we additionally allocate an auxiliary cache to buffer the mapping entries newly evicted from CMT, and produce dynamic pages with minimum difference between the maximal and the minimal LPNs of entries in those pages. Our experiments show that HCFTL can improve the CMT hit ratio by up to 41.1% and decrease the system response time by up to 33.3%, compared to baseline FTL schemes.

Explainable Multimedia Feature Fusion for Medical Applications.

Due to the exponential growth of medical information in the form of, e.g., text, images, Electrocardiograms (ECGs), X-rays, and multimedia, the management of a patient’s data has become a huge challenge. In particular, the extraction of features from various different formats and their representation in a homogeneous way are areas of interest in medical applications. Multimedia Information Retrieval (MMIR) frameworks, like the Generic Multimedia Analysis Framework (GMAF), can contribute to solving this problem, when adapted to special requirements and modalities of medical applications. In this paper, we demonstrate how typical multimedia processing techniques can be extended and adapted to medical applications and how these applications benefit from employing a Multimedia Feature Graph (MMFG) and specialized, efficient indexing structures in the form of Graph Codes. These Graph Codes are transformed to feature relevant Graph Codes by employing a modified Term Frequency Inverse Document Frequency (TFIDF) algorithm, which further supports value ranges and Boolean operations required in the medical context. On this basis, various metrics for the calculation of similarity, recommendations, and automated inferencing and reasoning can be applied supporting the field of diagnostics. Finally, the presentation of these new facilities in the form of explainability is introduced and demonstrated. Thus, in this paper, we show how Graph Codes contribute new querying options for diagnosis and how Explainable Graph Codes can help to readily understand medical multimedia formats.

A Hybrid Spatial Indexing Structure of Massive Point Cloud Based on Octree and 3D R*-Tree

The spatial index structure is one of the most important research topics for organizing and managing massive 3D Point Cloud. As a point in Point Cloud consists of Cartesian coordinates (x,y,z), the common method to explore geometric information and features is nearest neighbor searching. An efficient spatial indexing structure directly affects the speed of the nearest neighbor search. Octree and kd-tree are the most used for Point Cloud data. However, octree or KD-tree do not perform best in nearest neighbor searching. A highly balanced tree, 3D R*-tree is considered the most effective method so far. So, a hybrid spatial indexing structure is proposed based on octree and 3D R*-tree. In this paper, we discussed how thresholds influence the performance of nearest neighbor searching and constructing the tree. Finally, an adaptive way method adopted to set thresholds. Furthermore, we obtained a better performance in tree construction and nearest neighbor searching than octree and 3D R*-tree.

Searching Geometric Patterns in Protein Binding Sites and Their Application to Data Mining in Protein Kinase Structures.

The ever-growing number of protein-ligand complex structures can give fundamental insights into protein functions and protein-ligand interactions, especially in the field of protein kinase research. The number of tools to mine this data for individually defined structural motifs is restricted due to the challenging task of developing efficient index structures for 3D data in relational databases. Herein we present GeoMine, a database system with web front-end mining of more than 900 000 binding sites. It enables database searches for geometric (interaction) patterns in protein-ligand interfaces by, for example, textual, numerical, substructure, similarity, and 3D searches. GeoMine processes reasonably selective user-defined queries within minutes. We demonstrate its usability for advancing protein kinase research with a special emphasis on unusual interactions, their use in designing selective kinase inhibitors, and the analysis of reactive cysteine residues that are amenable to covalent kinase inhibitors. GeoMine is freely available as part of our modeling support server at https://proteins.plus.

An efficient indexing structure for multi-dimensional range query

An efficient indexing structure for multi-dimensional range query

Challenges and Recommendations in Big Data Indexing Strategies

Index structures are one of the main strategies for effective data access applied for indexing data. According to expansion of data, traditional indexing strategies on big data meets several challenges that lead to weak performance; they haven't abilities to handle the rapid increase of data in terms of accurate retrieval results and processing time. So, it is necessary to substitute traditional index with another efficient index structure called learned index. Learned index goes to use machine learning models to tackle such issues and achieve more enhancements of processing time and accurate results. In this research, the authors discuss different indexing strategies on big data both traditional and learned indexes, demonstrate the main features of them, perform comparison in terms of its performance, and present big data indexing challenges and solutions. Consequently, the research suggests replacing traditional indexes by dynamic index models, which lead to less processing time and more accurate results taking into consideration specification of hardware used.

GLAD: A Grid and Labeling Framework with Scheduling for Conflict-Aware NN Queries

The intelligent transportation systems, e.g., DiDi and Uber, have served as essential travel tools for customers, which foster plenty of studies for the location-based queries on road network. In particular, given a set O of objects and a query point q on a road network, the k Nearest Neighbor (kNN) query returns the k nearest objects in O with the shortest road network distance to q. In literature, most existing solutions for kNN queries tend to reduce the query time, indexing storage, or throughput of the kNN queries while overlooking the correctness of the queries caused by query-query and update-query conflicts. In our work, we propose a grid-based framework on conflict-aware kNN queries on moving objects which aims to optimize system throughput while guaranteeing query correctness. In particular, we first propose efficient index structures and new query algorithms that significantly improve the throughput. We further present novel scheduling algorithms that aim to avoid conflicts and improve the system throughput. Moreover, we devise approximate solutions that provide a controllable trade-off between the conflict of kNN queries and system throughput. Finally, we propose a cost-based dispatching strategy to assign the kNN results to the corresponding queries. Extensive experiments on real-world data demonstrate the effectiveness and efficiency of our proposed solutions over alternatives.

Template Matching for 3D Objects in Large Point Clouds Using DBMS

LIDAR and depth cameras have gone through a profound technological evolution, making large-scale recording of 3D point cloud data possible which raises new challenges for data processing. Most of the existing 3D point cloud processing methods were developed to work properly when the entire data set fits into the memory of a single server. When point clouds are significantly larger than the main memory and data are only available on slow storage, new approaches are necessary. In this paper, we propose a DBMS-based point cloud processing pipeline that solves the template matching problem, i.e., finding the – potentially multiple – occurrences of a small query point cloud in an extensive scene data set that is preprocessed and stored in a database. The storage layer uses a compact and novel data representation to exploit the benefits of efficient indexing structures whereas the query algorithm consists of a novel combination of existing point cloud processing and matching methods. To the best of our knowledge, this is the first template matching proposal in the literature that exploits the benefits of databases.

An Efficient Insertion Operator in Dynamic Ridesharing Services

Dynamic ridesharing refers to services that arrange one-time shared rides on short notice. It underpins various real-world intelligent transportation applications such as car-pooling, food delivery and last-mile logistics. A core operation in dynamic ridesharing is the “ <i>insertion operator</i> ”. Given a worker and a feasible route which contains a sequence of origin-destination pairs from previous requests, the insertion operator inserts a new origin-destination pair from a newly arrived request into the current route such that certain objective is optimized. Common optimization objectives include minimizing the maximum/sum flow time of all requests and minimizing the total travel time of the worker. Despite its frequent usage, the insertion operator has a time complexity of <inline-formula><tex-math notation="LaTeX">$O(n^3)$</tex-math></inline-formula> , where <inline-formula><tex-math notation="LaTeX">$n$</tex-math></inline-formula> is the number of all requests assigned to the worker. The cubic running time of insertion fundamentally limits the efficiency of urban-scale dynamic ridesharing based applications. In this paper, we propose a novel partition framework and a dynamic programming based insertion with a time complexity of <inline-formula><tex-math notation="LaTeX">$O(n^2)$</tex-math></inline-formula> . We further improve the time efficiency of the insertion operator to <inline-formula><tex-math notation="LaTeX">$O(n)$</tex-math></inline-formula> harnessing efficient index structures, such as fenwick tree. Evaluations on two real-world large-scale datasets show that our methods can accelerate insertion by 1.5 to 998.1 times.

TDHPPIR: An Efficient Deep Hashing Based Privacy-Preserving Image Retrieval Method

With the rapid development of multimedia techniques, mobile Internet and cloud computing, large-scale image retrieval service has become a necessity in our daily life. To overcome the challenges of privacy-preserving and data security during the image retrieval, we propose a novel deep hashing based privacy-preserving image retrieval method named TDHPPIR that can generate high quality hash codes of image and provide an efficient index structure for fast image retrieval in a security manner in cloud. A triplet Deep CNN structure model is introduced to learn deep visual representations and hash codes of images simultaneously, which can generate higher quality hash codes than the traditional way that mainly utilizes hand-crafted features. Besides, A novel hierachical bit-scalable hash codes based S-Tree, named H2S-Tree, is developed to increase the search efficiency. Comprehensive experiments on four benchmarks show that our method can combat the state-of-the-arts in both aspects of accuracy and efficiency.

Indexing Object Database using HC Tree

One of the recent applications of object technology is in the area of databases. One of the stumbling blocks in the commercial development and deployment of object databases is the lack of an efficient indexing technique. The properties of object databases make the task of development of an indexing technique all the more difficult. This paper discusses the development of an indexing technique for object databases. A new indexing technique based on a new structure, HC-Tree has been proposed. Performance analysis has been conducted, and experimental and analytical results indicate that the HC-Tree is an efficient indexing structure for object databases. The performance of the HC-Tree has also been compared with that of the other popular existing techniques - CH-Tree, H-Tree and hcC-Tree.

An efficient indexing for Internet of Things massive data based on cloud‐fog computing

AbstractIn recent years, the number of sensor and actuator nodes in the Internet of Things (IoT) networks has increased, generating a large amount of data. Most research techniques are based on dividing target data into subsets. On a large scale, this volume increases exponentially, which will affect search algorithms. This problem is caused by the inherent deficiencies of space partitioning. This paper introduces a new and efficient indexing structure to index massive IoT data called BCCF‐tree (Binary tree based on containers at the cloud‐fog computing level). This structure is based on recursive partitioning of space using the k‐means clustering algorithm to effectively separate space into nonoverlapping subspace to improve the quality of search and discovery algorithm results. A good topology should avoid a biased allocation of objects for separable sets and should not influence the structure of the index. BCCF‐tree structure benefits to the emerging cloud‐fog computing system, which represents the most powerful real‐time processing capacity provided by fog computing due to its proximity to sensors and the largest storage capacity provided by cloud computing. The paper also discusses the effectiveness of construction and search algorithms, as well as the quality of the index compared to other recent indexing data structures. The experimental results showed good performance.

Analyses of Indexing Techniques on Uncertain Data With High Dimensionality

Deploying a solution for handling critical decision-based problem efficiently requires the processing of high-dimensional data. Over the years, due to modern technological advancement, unprecedented volume of uncertain data is been captured and this has necessitated the need to organize such data for better data access performance. To this effect, the use of indexing technique for supporting, organizing, and storing of uncertain data with high dimensionality has become pertinent. However, the choice of an indexing technique to improve search performance is highly influenced by the properties of the underlying data set, data construction methods employed by the indexing structure, and the query types it supports. This paper is motivated to conduct an extensive performance analysis among existing indexing techniques, namely: R-tree, R*-tree and X-tree, in order to realize the most efficient indexing structure for organizing, storing and ultimately improving search performance over uncertain data with high dimensionality. The results of the analyses with regard to CPU processing time and number of nodes visited clearly show the superiority of X-tree over R-tree and R*-tree, as its superiority holds for different data set sizes, data distributions, number of dimensions and even with varying selectivity ratio.

Secure string pattern query for open data initiative

The open data initiative allows the organizations from different countries to share and integrate their data to produce value added information products and services. However, the trust relationship between any two organizations can hardly be established due to the data sensitivity and their specific responsibilities. In this paper, we propose a secure string pattern query processing framework for multiple data owners. With this framework, we design an efficient and secure indexing structure called S ecure S tring PA ttern S earch tree ( S2PAStree ). S2PAStree allows the execution of the secure string pattern query with logarithmic time complexity. We further propose a set of secure index construction protocols under the scenario that multiple data owners share and integrate their sensitive data. Finally, we conduct comprehensive experiments on three public datasets to evaluate the efficiency and the scalability of our solutions.

Toward efficient indexing structure for scalable content-based music retrieval

With advancement of various information processing and storage techniques, the scale of digital music collections has been growing at very fast speed during recent decades. To support high-quality content-based retrieval over such a large volume of music data, how to develop indexing structure with good effectiveness, efficiency and scalability becomes an important research issue. However, existing techniques mainly focus on improving query efficiency. Very few approaches have been proposed to address issues related to scalability and accuracy. In this study, we address the problem via introducing a novel indexing technique called effective music indexing framework (EMIF) to facilitate scalable and accurate music retrieval. It is designed based on a “classification-and-indexing” principle and consists of two main functionality modules: (1) music classification—a novel semantic-sensitive classification to identify an input song’s category and (2) indexing module—multiple local indexing structures, one for each semantic category to reduce query response time significantly. In particular, the classification model combining linear discriminative mixture model (LDMM) and advanced score fusion scheme has been applied to estimate category of music accurately. Layered architecture enables EMIF to enjoy superior scalability and efficiency. To evaluate the approach, a set of experimental studies has been carried out using two large music test collections and the results demonstrate various advantages of EMIF over state-of-the-art approaches including efficiency, scalability and effectiveness.