Compact Index Structure Research Articles

CIVET: Exploring Compact Index for Variable-Length Subsequence Matching on Time Series

Nowadays the demands for managing and analyzing substantially increasing collections of time series are becoming more challenging. Subsequence matching, as a core subroutine in time series analysis, has drawn significant research attention. Most of the previous works only focus on matching the subsequences with equal length to the query. However, many scenarios require support for efficient variable-length subsequence matching. In this paper, we propose a new representation, Uniform Piecewise Aggregate Approximation (UPAA) with the capability of aligning features for variable-length time series while remaining the lower bounding property. Based on UPAA, we present a compact index structure by grouping adjacent subsequences and similar subsequences respectively. Moreover, we propose an index pruning algorithm and a data filtering strategy to efficiently support variable-length subsequence matching without false dismissals. The experiments conducted on both real and synthetic datasets demonstrate that our approach achieves considerably better efficiency, scalability, and effectiveness than existing approaches.

Backbone Index and GNN Models for Skyline Path Query Evaluation over Multi-cost Road Networks

Skyline path queries (SPQs) extend skyline queries to multi-dimensional networks, such as multi-cost road networks (MCRNs). Such queries return a set of non-dominated paths between two given network nodes. Despite the existence of extensive works on evaluating different SPQ variants, SPQ evaluation is still very inefficient due to the nonexistence of efficient index structures to support such queries. Existing index building approaches for supporting shortest-path query execution, when directly extended to support SPQs, use an unreasonable amount of space and time to build, making them impractical for processing large graphs. In this paper, we propose a novel index structure, backbone index , and a corresponding index construction method that condenses an initial MCRN to multiple smaller summarized graphs with different granularity. We present efficient approaches to find approximate solutions to SPQs by utilizing the backbone index structure. Furthermore, considering making good use of historical query and query results, we propose two models, S kyline P ath G raph N eural N etwork (SP-GNN) and T ransfer SP-GNN (TSP-GNN), to support effective SPQ processing. Our extensive experiments on real-world large road networks show that the backbone index can support finding meaningful approximate SPQ solutions efficiently. The backbone index can be constructed in a reasonable time, which dramatically outperforms the construction of other types of indexes for road networks. As far as we know, this is the first compact index structure that can support efficient approximate SPQ evaluation on large MCRNs. The results on the SP-GNN and TSP-GNN models also show that both models can help get approximate SPQ answers efficiently.

A Distributed Hybrid Indexing for Continuous KNN Query Processing over Moving Objects

The magnitude of highly dynamic spatial data is expanding rapidly due to the instantaneous evolution of mobile technology, resulting in challenges for continuous queries. We propose a novel indexing approach model, namely, the Velocity SpatioTemporal indexing approach (VeST), for continuous queries, mainly Continuous K-nearest Neighbor (CKNN) and continuous range queries using Apache Spark. The proposed structure is based on a selective velocity partitioning method, i.e., since different objects have varying speeds, we divide the objects into two sets according to the actual mean speed we calculate before building the index and accessing data. Then the adopted indexing structure base unit comprises a nonoverlapping R-tree and a two dimension grid. The tree divides the space into nonoverlapping minimum bounding regions that point to the grids. Then, the uniform grid stores the object data of leaf nodes. This access method reduces the update cost and improves response time and query precision. In order to enhance performances for large-scale processing, we design a compact multilayer index structure on a distributed setting and propose a CKNN search algorithm for accurate results using a candidate cell identification process. We provide a comprehensive vision of our indexing model and the adopted query technique. The simulation results show that for query intervals of 100, the proposed approach is 13.59 times faster than the traditional approach, and the average time of the VeST approach is less than 0.005 for all query intervals. This proposed method improves response time and query precision. The precision of the VeST algorithm is almost equal to 100% regardless of the length of the query interval.

Angular Quantization Online Hashing for Image Retrieval

Online hash method with fast search mechanism and compact index structure plays a pivotal role. The inner product between label data has become one of the important means to measure the similarity between existing data and new data streams in online hashing methods. However, due to its discrete attributes and semantic gap, it often leads to a large amount of information loss. In this article, we propose a novel method called Angular Quantization Online Hashing (AQOH) to focus on learning compact binary codes with the help of cosine distance. Specifically, we propose an online hashing method for angular quantization, by minimizing the quantization error between the cosine similarity calculated from the original data and the generated binary code between the existing data and the new data stream. Further, within this framework, two effective algorithms to complete the optimization of the objective function to be designed, including continuous and discrete methods, respectively. Extensive experiments on various benchmark databases for online retrieval verify that our method outperforms many state-of-the art learning to hash methods.

I/O efficient k-truss community search in massive graphs

Community detection that discovers all densely connected communities in a network has been studied a lot. In this paper, we study online community search for query-dependent communities, which is a different but practically useful task. Given a query vertex in a graph, the problem is to find meaningful communities that the vertex belongs to in an online manner. We propose a community model based on the k-truss concept, which brings nice structural and computational properties. We design a compact and elegant index structure which supports the efficient search of k-truss communities with a linear cost with respect to the community size. We also investigate the k-truss community search problem in a dynamic graph setting with frequent insertions and deletions of graph vertices and edges. In addition, to support k-truss community search over massive graphs which cannot entirely fit in main memory, we propose I/O-efficient algorithms for query processing under the semi-external model. Extensive experiments on massive real-world networks demonstrate the effectiveness of our k-truss community model, the efficiency, and the scalability of our in-memory and semi-external community search algorithms.

Online supervised hashing

Fast nearest neighbor search is becoming more and more crucial given the advent of large-scale data in many computer vision applications. Hashing approaches provide both fast search mechanisms and compact index structures to address this critical need. In image retrieval problems where labeled training data is available, supervised hashing methods prevail over unsupervised methods. Most state-of-the-art supervised hashing approaches employ batch-learners. Unfortunately, batch-learning strategies may be inefficient when confronted with large datasets. Moreover, with batch-learners, it is unclear how to adapt the hash functions as the dataset continues to grow and new variations appear over time. To handle these issues, we propose OSH: an Online Supervised Hashing technique that is based on Error Correcting Output Codes. We consider a stochastic setting where the data arrives sequentially and our method learns and adapts its hashing functions in a discriminative manner. Our method makes no assumption about the number of possible class labels, and accommodates new classes as they are presented in the incoming data stream. In experiments with three image retrieval benchmarks, our method yields state-of-the-art retrieval performance as measured in Mean Average Precision, while also being orders-of-magnitude faster than competing batch methods for supervised hashing. Also, our method significantly outperforms recently introduced online hashing solutions.

MrsFAST-Ultra: a compact, SNP-aware mapper for high performance sequencing applications

High throughput sequencing (HTS) platforms generate unprecedented amounts of data that introduce challenges for processing and downstream analysis. While tools that report the ‘best’ mapping location of each read provide a fast way to process HTS data, they are not suitable for many types of downstream analysis such as structural variation detection, where it is important to report multiple mapping loci for each read. For this purpose we introduce mrsFAST-Ultra, a fast, cache oblivious, SNP-aware aligner that can handle the multi-mapping of HTS reads very efficiently. mrsFAST-Ultra improves mrsFAST, our first cache oblivious read aligner capable of handling multi-mapping reads, through new and compact index structures that reduce not only the overall memory usage but also the number of CPU operations per alignment. In fact the size of the index generated by mrsFAST-Ultra is 10 times smaller than that of mrsFAST. As importantly, mrsFAST-Ultra introduces new features such as being able to (i) obtain the best mapping loci for each read, and (ii) return all reads that have at most n mapping loci (within an error threshold), together with these loci, for any user specified n. Furthermore, mrsFAST-Ultra is SNP-aware, i.e. it can map reads to reference genome while discounting the mismatches that occur at common SNP locations provided by db-SNP; this significantly increases the number of reads that can be mapped to the reference genome. Notice that all of the above features are implemented within the index structure and are not simple post-processing steps and thus are performed highly efficiently. Finally, mrsFAST-Ultra utilizes multiple available cores and processors and can be tuned for various memory settings. Our results show that mrsFAST-Ultra is roughly five times faster than its predecessor mrsFAST. In comparison to newly enhanced popular tools such as Bowtie2, it is more sensitive (it can report 10 times or more mappings per read) and much faster (six times or more) in the multi-mapping mode. Furthermore, mrsFAST-Ultra has an index size of 2GB for the entire human reference genome, which is roughly half of that of Bowtie2. mrsFAST-Ultra is open source and it can be accessed at http://mrsfast.sourceforge.net.

Using compressed index structures for processing moving objects in large spatio-temporal databases

This paper develops a novel, compressed B +-tree based indexing scheme that supports the processing of moving objects in one-, two-, and multi- dimensional spaces. The past, current, and anticipated future trajectories of movements are fully indexed and well organized. No parameterized functions and geometric representations are introduced in our data model so that update operations are not required and the maintenance of index structures can be accomplished by basic insertion and deletion operations. The proposed method has two contributions. First, the spatial and temporal attributes of trajectories are accurately preserved and well organized into compact index structures with very efficient memory space utilization and storage requirement. Second, index maintenance overheads are more economical and query performance is more responsive than those of conventional methods. Both analytical and empirical studies show that our proposed indexing scheme outperforms the TPR-tree.

A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets

We present an adaptive out-of-core technique for rendering massive scalar volumes employing single-pass GPU ray casting. The method is based on the decomposition of a volumetric dataset into small cubical bricks, which are then organized into an octree structure maintained out-of-core. The octree contains the original data at the leaves, and a filtered representation of children at inner nodes. At runtime an adaptive loader, executing on the CPU, updates a view and transfer function-dependent working set of bricks maintained on GPU memory by asynchronously fetching data from the out-of-core octree representation. At each frame, a compact indexing structure, which spatially organizes the current working set into an octree hierarchy, is encoded in a small texture. This data structure is then exploited by an efficient stackless ray casting algorithm, which computes the volume rendering integral by visiting non-empty bricks in front-to-back order and adapting sampling density to brick resolution. Block visibility information is fed back to the loader to avoid refinement and data loading of occluded zones. The resulting method is able to interactively explore multi-gigavoxel datasets on a desktop PC.

Interactive High-Resolution Isosurface Ray Casting on Multicore Processors

We present a new method for the interactive rendering of isosurfaces using ray casting on multi-core processors. This method consists of a combination of an object-order traversal that coarsely identifies possible candidate 3D data blocks for each small set of contiguous pixels, and an isosurface ray casting strategy tailored for the resulting limited-size lists of candidate 3D data blocks. While static screen partitioning is widely used in the literature, our scheme performs dynamic allocation of groups of ray casting tasks to ensure almost equal loads among the different threads running on multi-cores while maintaining spatial locality. We also make careful use of memory management environment commonly present in multi-core processors. We test our system on a two-processor Clovertown platform, each consisting of a Quad-Core 1.86-GHz Intel Xeon Processor, for a number of widely different benchmarks. The detailed experimental results show that our system is efficient and scalable, and achieves high cache performance and excellent load balancing, resulting in an overall performance that is superior to any of the previous algorithms. In fact, we achieve an interactive isosurface rendering on a 1024(2) screen for all the datasets tested up to the maximum size of the main memory of our platform.

Efficient and compact indexing structure for processing of spatial queries in line-based databases

Points, lines and regions are the three basic entities for constituting vector-based objects in spatial databases. Many indexing schemes have been widely discussed for handling point or region data. These traditional schemes can efficiently organize point or region objects in a space into a hashing or hierarchical directory, and they provide efficient access methods for accurate retrievals. However, two difficulties arise when applying such methods to line segments: (1) the spatial information of line segments may not be precisely expressed in terms of that of points and/or regions, and (2) traditional methods for handling line segments can generate a large amount of dead space and overlapping areas in internal and external nodes in the hierarchical directory. The first problem impedes high-quality spatial conservation of line segments in a line-based database, while the second degrades the system performance over time. This study develops a novel indexing structure of line segments based on compressed B + trees. The proposed method significantly improves the time and space efficiencies over that of the R-tree indexing scheme.

Converting suffix trees into factor/suffix oracles

Several methods to compress suffix trees were defined, most of them with the aim of obtaining compact (that is, space economical) index structures. Besides this practical aspect, a compression method can reveal structural properties of the resulting data structure, allowing a better understanding of it and a better estimation of its performances. In this paper, we propose a simple method to compress suffix trees by merging couples of nodes. This idea was already used in the literature in a context different from ours. The originality of our approach is that the nodes we merge are not chosen with respect to their subtrees (which is difficult to test algorithmically), nor with respect to the words spelled along branches (which usually requires testing several branches before finding the good one) but with respect to their position in the tree (which is easy to compute). Another particularity of our method is it needs to read no edge label: it is exclusively based on the topology of the suffix tree. The compact structure resulting after compression is the factor/suffix oracle introduced by Allauzen, Crochemore and Raffinot whose accepted language includes the accepted language of the corresponding suffix tree. The interest of our paper is therefore threefold: 1. A topology-based compression method is defined for (compact) suffix trees. 2. A new property of a factor/suffix oracle is established, that is, like a DAG, it results from the corresponding suffix tree after a linear number of appropriate node mergings; unlike a DAG, the merged nodes do not necessarily have isomorphical subtrees. 3. A new algorithm to transform a suffix tree into a factor/suffix oracle is given, which has linear running time and thus improves the quadratic complexity previously known for the same task.

An efficient and scalable parallel algorithm for out-of-core isosurface extraction and rendering

We consider the problem of isosurface extraction and rendering for large scale time-varying data. Such data sets have been appearing at an increasing rate especially from physics-based simulations, and can range in size from hundreds of gigabytes to tens of terabytes. Isosurface extraction and rendering is one of the most widely used visualization techniques to explore and analyze such data sets. A common strategy for isosurface extraction involves the determination of the so-called active cells followed by a triangulation of these cells based on linear interpolation, and ending with a rendering of the triangular mesh. We develop a new simple indexing scheme for out-of-core processing of large scale data sets, which enables the identification of the active cells extremely quickly, using more compact indexing structure and more effective bulk data movement than previous schemes. Moreover, our scheme leads to an efficient and scalable implementation on multiprocessor environments in which each processor has access to its own local disk. In particular, our parallel algorithm provably achieves load balancing across the processors independent of the isovalue, with almost no overhead in the total amount of work relative to the sequential algorithm. We conduct a large number of experimental tests on the University of Maryland Visualization Cluster using the Richtmyer–Meshkov instability data set, and obtain results that consistently validate the efficiency and the scalability of our algorithm.