Multidimensional Index Structure Research Articles

WARM-tree: Making Quadtrees Write-efficient and Space-economic on Persistent Memories

Recently, the value of data has been widely recognized, which highlights the significance of data-centric computing in diversified application scenarios. In many cases, the data are multidimensional, and the management of multidimensional data often confronts greater challenges in supporting efficient data access operations and guaranteeing the space utilization. On the other hand, while many existing index data structures have been proposed for multidimensional data management, however, their designs are not fully optimized for modern nonvolatile memories, in particular the byte-addressable persistent memories. As a result, they might undergo serious access performance degradation or fail to guarantee space utilization. This observation motivates the redesigning of index data structures for multidimensional point data on modern persistent memories, such as the phase-change memory. In this work, we present the WARM-tree , a m ultidimensional t ree for r educing the w rite a mplification effect, for multidimensional point data. In our evaluation studies, as compared to the bucket PR quadtree and R*-tree, the WARM-tree can provide any worst-case space utilization guarantees in the form of \(\frac{m-1}{m}\) ( m ∈ ℤ^+) and effectively reduces the write traffic of key insertions by up to 48.10% and 85.86%, respectively, at the price of degraded average space utilization and prolonged latency of query operations. This suggests that the WARM-tree is a potential multidimensional index structure for insert-intensive workloads.

Fast Search-by-Classification for Large-Scale Databases Using Index-Aware Decision Trees and Random Forests

The vast amounts of data collected in various domains pose great challenges to modern data exploration and analysis. To find "interesting" objects in large databases, users typically define a query using positive and negative example objects and train a classification model to identify the objects of interest in the entire data catalog. However, this approach requires a scan of all the data to apply the classification model to each instance in the data catalog, making this method prohibitively expensive to be employed in large-scale databases serving many users and queries interactively. In this work, we propose a novel framework for such search-by-classification scenarios that allows users to interactively search for target objects by specifying queries through a small set of positive and negative examples. Unlike previous approaches, our framework can rapidly answer such queries at low cost without scanning the entire database. Our framework is based on an index-aware construction scheme for decision trees and random forests that transforms the inference phase of these classification models into a set of range queries, which in turn can be efficiently executed by leveraging multidimensional indexing structures. Our experiments show that queries over large data catalogs with hundreds of millions of objects can be processed in a few seconds using a single server, compared to hours needed by classical scanning-based approaches.

Text-based multi-dimensional medical images retrieval according to the features-usage correlation.

Emerging medical imaging applications in healthcare, the number and volume of medical images is growing dramatically. Information needs of users in such circumstances, either for clinical or research activities, make the role of powerful medical image search engines more significant. In this paper, a text-based multi-dimensional medical image indexing technique is proposed in which correlation of the features-usages (according to the user’s queries) is considered to provide an off-the content indexing while taking users’ interestingness into account. Assuming that each medical image has some extracted features (e.g., based on the DICOM standard), correlations of the features are discovered by performing data mining techniques (i.e., quantitative association pattern discovery), on the history of users’ queries as a data set. Then, based on the pairwise correlation of the features of medical images (a.k.a. Affinity), set of the all features is fragmented into subsets (using method like the vertical fragmentation of the tables in distribution of relational DBs). After that, each of these subsets of the features turn into a hierarchy of the features (by applying a hierarchical clustering algorithm on that subset), subsequently all of these distinct hierarchies together make a multi-dimensional structure of the features of medical images, which is in fact the proposed text-based (feature-based) multi-dimensional index structure. Constructing and using such text-based multi-dimensional index structure via its specific required operations, medical image retrieval process would be improved in the underlying medical image search engine. Generally, an indexing technique is to provide a logical representation of documents in order to optimize the retrieval process. The proposed indexing technique is designed such that can improve retrieval of medical images in a medical image search engine in terms of its effectiveness and efficiency. Considering correlation of the features of the image would semantically improve precision (effectiveness) of the retrieval process, while traversing them through the hierarchy in one dimension would try to optimize (i.e., minimize) the resources to have a better efficiency. The proposed text-based multi-dimensional indexing technique is implemented using the open source search engine Lucene, and compared with the built-in indexing technique available in the Lucene search engine, and also with the Terrier platform (available for the benchmarking of information retrieval systems) and other the most related indexing techniques. Evaluation results of memory usage and time complexity analysis, beside the experimental evaluations efficiency and effectiveness measures show that the proposed multi-dimensional indexing technique significantly improves both efficiency and effectiveness for a medical image search engine.Graphical abstract

Concurrent linearizable nearest neighbour search in LockFree-kD-tree

The Nearest neighbour search (NNS) is a fundamental problem in many application domains dealing with multidimensional data. In a concurrent setting, where dynamic modifications are allowed, a linearizable implementation of the NNS is highly desirable.This paper introduces the LockFree-kD-tree (LFkD-tree ): a lock-free concurrent kD-tree, which implements an abstract data type (ADT) that provides the operations Add, Remove, Contains, and NNS. Our implementation is linearizable. The operations in the LFkD-tree use single-word read and compare-and-swap (▪ ) atomic primitives, which are readily supported on available multi-core processors.We experimentally evaluate the LFkD-tree using several benchmarks comprising real-world and synthetic datasets. The experiments show that the presented design is scalable and achieves significant speed-up compared to the implementations of an existing sequential kD-tree and a recently proposed multidimensional indexing structure, PH-tree.

Selective caching: a persistent memory approach for multi-dimensional index structures

After the introduction of Persistent Memory in the form of Intel’s Optane DC Persistent Memory on the market in 2019, it has found its way into manifold applications and systems. As Google and other cloud infrastructure providers are starting to incorporate Persistent Memory into their portfolio, it is only logical that cloud applications have to exploit its inherent properties. Persistent Memory can serve as a DRAM substitute, but guarantees persistence at the cost of compromised read/write performance compared to standard DRAM. These properties particularly affect the performance of index structures, since they are subject to frequent updates and queries. However, adapting each and every index structure to exploit the properties of Persistent Memory is tedious. Hence, we require a general technique that hides this access gap, e.g., by using DRAM caching strategies. To exploit Persistent Memory properties for analytical index structures, we propose selective caching. It is based on a mixture of dynamic and static caching of tree nodes in DRAM to reach near-DRAM access speeds for index structures. In this paper, we evaluate selective caching on the OLAP-optimized main-memory index structure Elf, because its memory layout allows for an easy caching. Our experiments show that if configured well, selective caching with a suitable replacement strategy can keep pace with pure DRAM storage of Elf while guaranteeing persistence. These results are also reflected when selective caching is used for parallel workloads.

DimensionSlice: A main-memory data layout for fast scans of multidimensional data

Multidimensional data are exploited in many application areas such as scientific data analysis, business intelligence, and geographic information systems. One of the most frequent operations applied to such multidimensional data is the selection of a subspace of the given multidimensional space, which involves predicate evaluation on multiple dimensions. Existing main-memory data layouts optimized for evaluating predicates on the columnar data can be used to accelerate the subspace extraction by sequentially performing filter scans on each dimension one at a time. However, optimization opportunities emerge if we can consider all predicates together. In this paper, we propose DimensionSlice, a new main-memory data layout optimized for evaluating predicates on multiple dimensions. More specifically, the dimension values are sliced into portions and the portions with the same order of each dimension are arranged together. Multiple predicates are simultaneously evaluated with the sliced dimension values during the scan. In addition, by storing the different portions separately, unnecessary loads and computations of lower portions can be eliminated if the evaluation results are assured after examining the upper portions. For further acceleration of scans, the DimensionSlice layout is designed to easily leverage the SIMD capabilities that most mainstream processors are equipped with. Through experiments, we demonstrate the performance gains of the proposed method over the columnar main-memory layout that evaluates the partial predicates one dimension at a time. We also show that the proposed method outperforms the state-of-the-art multidimensional index structure when the selectivity is over a very low threshold.

Grid-R-tree: a data structure for efficient neighborhood and nearest neighbor queries in data mining

The use of multi-dimensional indexing structures has gained a lot of attention in data mining. The most commonly used data structures for indexing data are R-tree and its variants, quad-tree, k-d-tree, etc. These data structures support region queries (point, window and neighborhood queries) and nearest neighbor queries. These queries are extensively used in data mining algorithms. Although these data structures facilitate execution of the above queries in logarithmic time, the constraints associated with them become bottleneck in query execution, when used for large and high-dimensional datasets. Moreover, these indexing structures do not cater to specific data access patterns of data mining algorithms. In this paper, we propose a new data structure Grid-R-tree, a grid based R-tree which is specifically designed to address the querying requirements of multiple data mining algorithms. Grid-R-tree is a simple, yet effective adaptation of R-tree using the concept of Grid. We also introduce a new query over Grid-R-tree, called cell-wise epsilon neighborhood query (CellWiseNBH), which captures the locality in query execution pattern of density-based clustering algorithms, and enables us to redesign them for improving their efficiency. Our theoretical and experimental analysis shows that the proposed data structure outperforms the conventional R-tree in terms of neighborhood and nearest neighbor queries. The experiments were conducted on datasets of size up to 100 million and dimensionality up to 74. The results also suggest that Grid-R-tree improves the efficiency of data mining algorithms such as k-nearest neighbor classifier and DBSCAN clustering (including the redesigned version that uses CellWiseNBH). Additionally, an adaptive grid optimization has been applied on dense cells that have number of indexed data points greater than a threshold $$\tau $$ to keep equal load distribution in the cells, which resulted in more efficient query performance for datasets that have skewed distribution of data points.

On approximate k-nearest neighbor searches based on the earth mover’s distance for efficient content-based multimedia information retrieval

The Earth Mover's Distance (EMD) is one of the most-widely used distance functions to measure the similarity between two multimedia objects. While providing good search results, the EMD is too much time consuming to be used in large multimedia databases. To solve the problem, we propose an approximate k-nearest neighbor (k-NN) search method based on the EMD. In the proposed method, the overhead for both disk accesses and EMD computations is reduced significantly, thanks to the approximation. First, the proposed method builds an index using the M-tree, a distance-based multi-dimensional index structure, to reduce the disk access overhead. When building the index, we reduce the number of features in the multimedia objects through dimensionalityreduction. When performing the k-NN search on the M-tree, we find a small set of candidates from the disk using the index and then perform the post-processing on them. Second, the proposed method uses the approximate EMD for index retrieval and post-processing to reduce the computational overhead of the EMD. To compensate the errors due to the approximation, the method provides a way of accuracy improvement of the approximate EMD. We performed extensive experiments to show the efficiency of the proposed method. As a result, the method achieves significant improvement in performance with only small errors: the proposed method outperforms the previous method by up to 67.3% with only 3.5% error.

BLBTree: an Efficient Index Structure for Fast Search

Due of its interest in different areas, similarity search in high-dimensional spaces is one of the principal research axes today for CBIR Systems. The properties for high-dimensional data demand some adequate search methods to have a research in optimal time. Due to the curse of dimensionality, search time in the index structure exponentially increases, according the dimension of the descriptor. However, the performances of classical index structures become less than the sequential scan of data in search time when answering exact nearest neighbor queries. To overcome this problematic, two major approaches for high-dimensional data in CBIR systems have been proposed: The first approach permits to speed up the search by using multiple levels of lower bound distances, the second approach exploits the BTree index structure to speed up the search. We propose to combine both techniques to search for large nearest neighbors in a high-dimensional space. We develop a new multidimensional index structure, called BLBTree. In BLBTree, instead of computing the distances in the high-dimensional original space to find the nearest neighbor, lot of candidates are to be rejected based on the lower bound distances: the research in BLBTree does not calculate the distances in the high-dimensional original space to find the nearest neighbor if the lower bounding distance exceeds the threshold, because the distances in the high-dimensional cannot be lower than the distances in the low-dimensional. We note that each object in BLBTree is described by a pyramid structure of histograms, with each histogram representing the lower resolution of the other histogram; due to this property, our index structure is well-suited for high-dimensional and large-scale problems. We have shown that our approach provides interesting and powerful experimental results.

Towards trajectory anonymisation using multi-dimensional index structures

Trajectory datasets are increasingly available due to the technological advances in location-sensing devices, wireless technologies, and hand-held devices. However, the datasets also causes consumer privacy concerns. This paper addresses the privacy issues by using the internal structure of R-tree, a multi-dimensional index structure. The benefit of using R-tree is that it clusters trajectories in a way that their bounding spatiotemporal extension is minimised, thus achieving better quality in the anonymised database. This is a desirable property of the resulting anonymised database. In order to improve the quality of service requirements, a novel algorithm has been proposed.

Towards trajectory anonymisation using multi-dimensional index structures

Trajectory datasets are increasingly available due to the technological advances in location-sensing devices, wireless technologies, and hand-held devices. However, the datasets also causes consumer privacy concerns. This paper addresses the privacy issues by using the internal structure of R-tree, a multi-dimensional index structure. The benefit of using R-tree is that it clusters trajectories in a way that their bounding spatiotemporal extension is minimised, thus achieving better quality in the anonymised database. This is a desirable property of the resulting anonymised database. In order to improve the quality of service requirements, a novel algorithm has been proposed.

Exploiting Massive Parallelism for IndexingMulti-Dimensional Datasets on the GPU

Inherently multi-dimensional n-ary indexing structures such as R-trees are not well suited for the GPU because of their irregular memory access patterns and recursive back-tracking function calls. It has been known that traversing hierarchical tree structures in an irregular manner makes it difficult to exploit parallelism and to maximize the utilization of GPU processing units. Moreover, the recursive tree search algorithms often fail with large indexes because of the GPU's tiny runtime stack size. In this paper, we propose a novel parallel tree traversal algorithm-massively parallel restart scanning (MPRS) for multi-dimensional range queries that avoids recursion and irregular memory access. The proposed MPRS algorithm traverses hierarchical tree structures with mostly contiguous memory access patterns without recursion, which offers more chances to optimize the parallel SIMD algorithm. We implemented the proposed MPRS range query processing algorithm on n-ary bounding volume hierarchies including R-trees and evaluated its performance using real scientific datasets on an NVIDIA Tesla M2090 GPU. Our experiments show braided parallel SIMD friendly MPRS range query algorithm achieves at least 80 percent warp execution efficiency while task parallel tree traversal algorithm shows only 9-15 percent efficiency. Moreover, braided parallel MPRS algorithm accesses 7-20 times less amount of global memory than task parallel parent link algorithm by virtue of minimal warp divergence.

A Divisive Hierarchical Clustering Based Method for Indexing Image Information

In most practical applications of image retrieval, high-dimensional feature vectors are required, but current multi-dimensional indexing structures lose their efficiency with growth of dimensions. Our goal is to propose a divisive hierarchical clustering-based multi-dimensional indexing structure which is efficient in high-dimensional feature spaces. A projection pursuit method has been used for finding a component of the data, which data's projections onto it maximizes the approximation of negentropy for preparing essential information in order to partitioning of the data space. Various tests and experimental results on high-dimensional datasets indicate the performance of proposed method in comparison with others.

Discovery of time series $$k$$ k -motifs based on multidimensional index

Time series motifs are frequently occurring but previously unknown subsequences of a longer time series. Discovering time series motifs is a crucial task in time series data mining. In time series motif discovery algorithm, finding nearest neighbors of a subsequence is the basic operation. To make this basic operation efficient, we can make use of some advanced multidimensional index structure for time series data. In this paper, we propose two novel algorithms for discovering motifs in time series data: The first algorithm is based on $$\hbox {R}^{*}$$R?-tree and early abandoning technique and the second algorithm makes use of a dimensionality reduction method and state-of-the-art Skyline index. We demonstrate that the effectiveness of our proposed algorithms by experimenting on real datasets from different areas. The experimental results reveal that our two proposed algorithms outperform the most popular method, random projection, in time efficiency while bring out the same accuracy.

Tensor index for large scale image retrieval

Recently, the bag-of-words representation is widely applied in the image retrieval applications. In this model, visual word is a core component. However, compared with text retrieval, one major problem associated with image retrieval consists in the visual word ambiguity, i.e., a trade-off between precision and recall of visual matching. To address this problem, this paper proposes a tensor index structure to improve precision and recall simultaneously. Essentially, the tensor index is a multi-dimensional index structure. It combines the strengths of two state-of-the-art indexing strategies, i.e., the inverted multi-index [Babenko and Lempitsky (Computer vision and pattern recognition (CVPR), 2012 IEEE Conference, 3069---3076, 2012)] as well as the joint inverted index [Xia et al. (ICCV, 2013)] which are initially designed for approximate nearest neighbor search problems. This paper, instead, exploits their usage in the scenario of image retrieval and provides insights into how to combine them effectively. We show that on the one hand, the multi-index enhances the discriminative power of visual words, thus improving precision; on the other hand, the introduction of multiple codebooks corrects quantization artifacts, thus improving recall. Extensive experiments on two benchmark datasets demonstrate that tensor index significantly improves the baseline approach. Moreover, when incorporating methods such as Hamming embedding, we achieve competitive performances compared to the state-of-the-art ones.

K-Nearest neighbor searching in hybrid spaces

Little work has been reported in the literature to support k-nearest neighbor (k-NN) searches/queries in hybrid data spaces (HDS). An HDS is composed of a combination of continuous and non-ordered discrete dimensions. This combination presents new challenges in data organization and search ordering. In this paper, we present an algorithm for k-NN searches using a multidimensional index structure in hybrid data spaces. We examine the concept of search stages and use the properties of an HDS to derive a new search heuristic that greatly reduces the number of disk accesses in the initial stage of searching. Further, we present a performance model for our algorithm that estimates the cost of performing such searches. Our experimental results demonstrate the effectiveness of our algorithm and the accuracy of our performance estimation model.

A Comparative Study on the Performance of HZR+ Trees on Query Processing

Data is being produced in new forms and unimaginable quantities. Researches and other scientific and commercial applications are engrossing the scientific community for their size and need of faster accessibility. The conventional access methods previously available in multidimensional databases are no longer suitable for the new form of data produced. In traditional databases, multicolumn index is created using B-tree [5]. This indexing cannot slide over columns, so the primary index column must be in the WHERE clause filters of the query. The R-tree [3], an extension of the B-tree, is a hierarchical, height balanced multidimensional indexing structure that guarantees space utilization above a certain threshold. But the data produced in most of the cases are not spatial in nature. Therefore, the data should be restructured in order to map the non-spatial data to geometric space. Thus, the multidimensional accessibility of spatial access methods, experimented on non spatial data for the first time and the analysis of which has produced interesting results forms the major contributions of this paper. The sequence of procedures followed to arrive at the analytical study is as follows: 1. The packing of non spatial data converts the data into a form that paves the way for multidimensional access, similar to using spatial access methods for spatial data. 2. The proposal of reduction of overlap of data using Hilbert curves for ordering the data before insertion into the proposed indexing structure 3. The proposal of a new index structure, Hilbert ZR+ Tree [HZR+ Tree]. 4. A collection of experiments and analysis which validates and proves the efficiency of the proposed data model.

A Multi-dimensional Index Structure Based on Improved VA-file and CAN in the Cloud

Currently, cloud computing systems use simple key-value data processing, which cannot support similarity search effectively due to lack of efficient index structures, and with increase of dimensionality, existing tree-like index structures could lead to problem of the curse of dimensionality. In this paper, a novel VF-CAN indexing scheme is proposed. VF-CAN integrates content addressable network (CAN) based routing protocol and improved vector approximation file (VA-file) index. There are two index levels in this scheme: global index and local index. The local index VAK-file is built for data in each storage node. VAK-file is k-means clustering result of VA-file approximation vectors according to their degree of proximity. Each cluster forms a separate local index file and each file stores approximate vectors that are contained in cluster. The vector of each cluster center is stored in cluster center information file of corresponding storage node. In global index, storage nodes are organized into an overlay network CAN, and in order to reduce cost of calculation, only clustering information of local index is issued to entire overlay network through CAN interface. The experimental results show that VF-CAN reduces index storage space and improves query performance effectively.

조상-자손 관계 탐색을 지원하기 위한 XML 타입상속 색인구조의 계층적 구성기법

본 논문에서는 XML 데이터베이스에서 XML 질의처리를 효율적으로 지원하기 위한 다차원 타입상속 색인구조(MD-TIX)들의 계층적 구성기법을 제시한다. MD-TIX는 중첩요소와 여러 타입상속 계층으로 이루어진 중첩술어의 조상-자손 관계 탐색을 효율적으로 지원하기 위하여 다차원 색인구조를 이용하는 색인기법이다. 그러나 이러한 MD-TIX는 질의에 주어진 Xpath의 길이가 긴 경우에 색인 엔트리의 구성문제 때문에 색인구조의 구축과 유지관리에 어려움이 있다. 이를 극복하기 위해서, 본 논문에서는 먼저 주어진 Xpath에서 인접한 두 타입 사이의 직접 관계 탐색을 지원하는 기본 색인구조들을 구축하고, 이들을 바탕으로 Xpath상의 임의의 두 타입 사이의 간접 관계 탐색을 지원하는 유도 색인구조들을 구축한다. 이러한 과정을 전체 길이의 Xpath를 지원하는 하나의 목표 색인구조를 구축하기까지 계층적으로 구성하는 방법을 제시한다. 또한 Xpath 상에 주어진 몇 개의 부경로 탐색만을 지원하기 위한 부분적인 색인계층을 효율적으로 구축하기 위한 알고리즘을 제안한다. This paper presents a hierarchical structuring method for the multidimensional type inheritance indices (MD-TIXs) that support the processing of XML queries in XML databases. MD-TIX uses a multidimensional index structure for efficiently supporting ancestor-descendant explorations that involve both nested element and type inheritance hierarchies. However, In the case of a long Xpath, the building and maintenance of MD-TIX are very difficult because of index entry construction problem. So, we propose a type inheritance index hierarchy method for solving this difficulty. We first construct base indices that support direct relationship explorations between adjacent two types on a given Xpath, and then, based on these base indices, we construct hierarchically the derived indices that support indirect relationship explorations between any two types of Xpath until constructing one target index for supporting the full Xpath. And we also present an algorithm that efficiently constructs a partial index hierarchy for supporting given a set of sub-pathes explorations.

$\mathcal{MD}$ -HBase: design and implementation of an elastic data infrastructure for cloud-scale location services

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open-source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, key-value stores can effectively support large scale operation, but do not natively provide multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present the design and implementation of $\mathcal {MD}$ -HBase, a scalable data management infrastructure for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a key-value store. The underlying key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. Our optimized query processing technique accesses only the index and storage level entries that intersect with the query region, thus ensuring efficient query processing. We present the design of $\mathcal {MD}$ -HBase that demonstrates how two standard index structures--the K-d tree and the Quad tree--can be layered over a range partitioned key-value store to provide scalable multi-dimensional data infrastructure. Our prototype implementation using HBase, a standard open-source key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multi-dimensional range queries and nearest neighbor queries in real-time with response times as low as few hundreds of milliseconds.