Hierarchical Indexing Research Articles

Building socially-enabled event-enriched maps

With the advancement of social sensing technologies, digital maps have recently witnessed a tremendous evolution with the aim of integrating enriched semantic layers from heterogeneous and diverse data sources. Current generations of digital maps are often crowd-sourced, allow interactive route planning, and may contain live updates, such as traffic congestion states. Within this context, we believe that the next generation of maps will introduce the concept of extracting Events of Interest (EoI) from crowdsourced data, and displaying them at different spatial scales based on their significance. This paper introduces Hadath1, a scalable and efficient system that extracts social events from unstructured data streams, e.g. Twitter. Hadath applies natural language processing and multi-dimensional clustering techniques to extract relevant events of interest at different map scales, and to infer the spatio-temporal scope of detected events. Hadath also implements a hierarchical in-memory spatio-temporal indexing scheme to allow efficient and scalable access to raw data, as well as to extracted clusters of events. Initially, data packets are processed to discover events at a local scale, then, the proper spatio-temporal scope and the significance of detected events at a global scale is determined. As a result, live events can be displayed at different spatio-temporal resolutions, thus allowing a smooth and unique browsing experience. Finally, to validate our proposed system, we conducted experiments on real-time and historical social media streams.

Lattice Quad-Tree Indexing Algorithm for a Hexagonal Discrete Global Grid System

Hexagonal discrete global grid systems are the preferred data models supporting multisource geospatial information fusion. Related research has aroused widespread concern in the academic community, and hierarchical indexing algorithms are one of the main research focuses. In this paper, we propose an algorithm for indexing the cell of a ringed spatial area based on a hexagonal lattice quad-tree (HLQT) structure and the indexing characteristics. First, we design a single-resolution indexing algorithm in which indexing starts from the initial quad tree and expands ring by ring using coding operations, and a quad-tree structure is applied to accelerate this process. Second, the hierarchical indexing algorithm is implemented based on single-resolution indexing, and a pyramid hierarchical model is established. Finally, we perform comparison experiments with existing algorithms. The results of the experiments indicate that the single-level indexing efficiency of the proposed algorithm is approximately twice that of the traditional method and that the hierarchical indexing efficiency is approximately 67 times that of the traditional method. These findings verify the feasibility and superiority of the algorithm proposed in this paper.

Stereoscopic Image Feature Indexing Based on Hybrid Grid Multiple Suffix Tree and Hierarchical Clustering

In order to achieve content-based binocular stereoscopic image or video retrieval efficiently, a feature indexing algorithm based on hybrid grid multiple suffix tree and hierarchical clustering is proposed. With the RGB-D image model, the shape features of depth map obtained from the matching of binocular stereoscopic left image and right image and the color features of left image are extracted respectively. The features are quantified and hashed, and the optimized underlying features are sorted as leaf nodes of hierarchical indexing. Then the shape and color feature values of leaf nodes are mapped to the two-dimensional coordinates, and the 2D feature points are put into different grid hash areas respectively by clustering and labeled with multiple suffix tree. Furthermore, to construct the global index, a pointer to an array of clustering grid center point is defined according to the computation of grid area feature values. The experimental results show that compared to the double grid suffix tree and typical stereoscopic image feature indexing structure, the proposed algorithm can effectively reduce the indexing construction complexity While maintaining high recall, it can also greatly improve the query efficiency, which can better realize the feature indexing of binocular stereoscopic images or videos.

Skyline-based dissimilarity of images

Large image collections are being used in many modern applications. In this paper, we aim at capturing the intrinsic dissimilarities of image descriptors in large image collections, i.e., to detect dissimilar (or else diverse) images without defining an explicit similarity or distance measure. Towards this goal, we adopt skyline processing techniques for large image databases, based on their high-dimensional descriptor vectors. The novelty of the proposed methodology lies in the use of skyline techniques empowered by state-of-the-art hashing schemes to enable effective data partitioning and indexing in secondary memory, towards supporting large image databases. The proposed approach is evaluated experimentally by using three real-world image datasets. Performance evaluation results demonstrate that images lying on the skyline have significantly different characteristics, which depend on the type of the descriptor. Thus, these skyline items may be used as seeds to apply clustering in large image databases. In addition, we observe that skyline processing using hash-based indexing structures is significantly faster than index-free skyline computation and also more efficient than skyline computation with hierarchical indexing structures. Based on our results, the proposed approach is both efficient (regarding runtime) and effective (with respect to image diversity) and therefore can be used as a base for more complex data mining tasks such as clustering.

Graph-based genome alignment and genotyping with HISAT2 and HISAT-genotype

The human reference genome represents only a small number of individuals, which limits its usefulness for genotyping. We present a method named HISAT2 (hierarchical indexing for spliced alignment of transcripts 2) that can align both DNA and RNA sequences using a graph Ferragina Manzini index. We use HISAT2 to represent and search an expanded model of the human reference genome in which over 14.5 million genomic variants in combination with haplotypes are incorporated into the data structure used for searching and alignment. We benchmark HISAT2 using simulated and real datasets to demonstrate that our strategy of representing a population of genomes, together with a fast, memory-efficient search algorithm, provides more detailed and accurate variant analyses than other methods. We apply HISAT2 for HLA typing and DNA fingerprinting; both applications form part of the HISAT-genotype software that enables analysis of haplotype-resolved genes or genomic regions. HISAT-genotype outperforms other computational methods and matches or exceeds the performance of laboratory-based assays.

Verb inflection in Muklom Tangsa

Muklom Tangsa is a Tibeto-Burman (TB) language variety with rich verbal inflection that exhibits hierarchical indexing and a non-canonical inverse system. Indexes will align with S, A, P, or R arguments, depending on the configuration, but not with the T argument. Inverse marking is triggered by high-ranked P arguments, i.e. the speech act participant (SAP) P, but also by SAP R and even SAP possessors. We can conclude that verb marking and NP marking are relatively disintegratedː the system of expressing semantic roles by case markers or postpositions does not nicely align with the system of indexing and inverse marking on the verb. This structure, commonly found among TB languages, is known as ‘associative agreement’, as opposed to ‘integrative agreement’, which nicely aligns NP and verb domains (see Bickel 2000). This chapter provides an overview of verb inflection in Muklom based on primary data.

S2R-tree: a pivot-based indexing structure for semantic-aware spatial keyword search

Semantic-aware spatial keyword search is an important technique for digital map services. However, existing indexing and search methods have limited pruning effect due to the high dimensionality in semantic space, causing query efficiency to be a serious issue. To handle this problem, this paper proposes a novel pivot-based hierarchical indexing structure S2R-tree to integrate spatial and semantic information in a seamless way. Instead of indexing objects in the original semantic space, we carefully design a space mechanism to transform the high dimensional semantic vectors to a low dimensional space, so that more effective pruning effect can be achieved. On top of the S2R-tree, an efficient query processing algorithm is further designed, which not only ensures efficient query processing by a set of theoretical bounds, but also returns accurate results despite of the indexing in the low dimensional space. Furthermore, we conduct extensive experiments to evaluate and compare our proposed and baseline methods.

Hierarchical Hexagonal Clustering and Indexing

Space-filling curves (SFCs) represent an efficient and straightforward method for sparse-space indexing to transform an n-dimensional space into a one-dimensional representation. This is often applied for multidimensional point indexing which brings a better perspective for data analysis, visualization and queries. SFCs are involved in many areas such as big data analysis and visualization, image decomposition, computer graphics and geographic information systems (GISs). The indexing methods subdivide the space into logic clusters of close points and they differ in various parameters including the cluster order, the distance metrics, and the pattern shape. Beside the simple and highly preferred triangular and square uniform grids, the hexagonal uniform grids have gained high interest especially in areas such as GISs, image processing and data visualization for the uniform distance between cells and high effectiveness of circle coverage. While the linearization of hexagons is an obvious approach for memory representation, it seems there is no hexagonal SFC indexing method generally used in practice. The main limitation of hexagons lies in lacking infinite decomposition into sub-hexagons and similarity of tiles on different levels of hierarchy. Our research aims at defining a fast and robust hexagonal SFC method. The Gosper fractal is utilized to preserve the benefits of hexagonal grids and to efficiently and hierarchically linearize points in a hexagonal grid while solving the non-convex shape and recursive transformation issues of the fractal. A comparison to other SFCs and grids is conducted to verify the robustness and effectiveness of our hexagonal method.

GLAD: Global–Local-Alignment Descriptor for Scalable Person Re-Identification

The huge variance of human pose and the misalign-ment of detected human images significantly increase the difficulty of pedestrian image matching in person Re-Identification (Re-ID). Moreover, the massive visual data being produced by surveillance video cameras requires highly efficient person Re-ID systems. Targeting to solve the first problem, this work proposes a robust and discriminative pedestrian image descriptor, namely, the Global–Local-Alignment Descriptor (GLAD). For the second problem, this work treats person Re-ID as image retrieval and proposes an efficient indexing and retrieval framework. GLAD explicitly leverages the local and global cues in the human body to generate a discriminative and robust representation. It consists of part extraction and descriptor learning modules, where several part regions are first detected and then deep neural networks are designed for representation learning on both the local and global regions. A hierarchical indexing and retrieval framework is designed to perform offline relevance mining to eliminate the huge person ID redundancy in the gallery set, and accelerate the online Re-ID procedure. Extensive experimental results on widely used public benchmark datasets show GLAD achieves competitive accuracy compared to the state-of-the-art methods. On a large-scale person, with the Re-ID dataset containing more than 520 K images, our retrieval framework significantly accelerates the online Re-ID procedure while also improving Re-ID accuracy. Therefore, this work has the potential to work better on person Re-ID tasks in real scenarios.

Central Place Indexing: Hierarchical Linear Indexing Systems for Mixed-Aperture Hexagonal Discrete Global Grid Systems

Hexagonal discrete global grid systems (DGGSs) with integer spatial indexes are a promising new approach to designing geospatial data structures and location reference systems. Central place indexing (CPI) is a class of multi-precision hierarchical linear spatial indexing systems for pure and mixed-aperture hexagonal DGGSs. Definitions for CPI systems are given both on the plane and on the polyhedral surfaces of geodesic DGGSs, and examples of real-world DGGSs indexed using CPI are described. The semantic advantages of CPI systems are discussed, including their ability to exactly represent their own geometries.

Divergent gene expression through PI3K/akt signalling pathway cause different models of hypertrophy growth in chicken

RNA-Seq technology was used to investigate differences in the gene expression of pectoralis muscle tissue between two chicken breeds [Ross as commercial (rapidly growing) and Isfahani as Iranian local breed (slow-growing)]. Total RNA was isolated from breast muscle samples at end of 4 weeks of age and sequenced by an Illumina Hiseq 2000 sequencer. Hierarchical Indexing for Spliced Alignment of Transcripts (HISAT2) was applied to align clean reads to chicken reference genome. Then, Cufflinks package was used to assemble transcripts and identify significantly differentially expressed genes (DEGs). Between two groups, the 606 significantly DEGs were identified (p-adjusted ≤.05). The confirmation of RNA-seq data was performed by Quantitative real-time PCR (qRT-PCR), consistent expression results were found for 10 selected genes. This study identified DEGs that regulate peroxisome proliferator-activated receptors (PPAR) signalling and Phosphatidylinositol-3 kinase Akt (PI3K/Akt) pathways in the Commercial breed, which might contribute to its higher metabolism of energy and growth characteristics compared to the Native breed. Our results suggested that the different expression patterns of some interesting genes including the PIK3IP1, SGK1, FOXO3, FBXO32, FBXO30, CUL3 and ASB1 in Native chicken might represent a cause for the poor growth performance for this breed than the Commercial breed.Research highlightsExpression patterns of PIK3IP1, FoxO3 and FBXO32 and some other genes might represent a cause for the poor growth performance in Native chickens.DEGs that regulate PPAR signalling and PI3K/Akt pathways, which might contribute to the higher metabolism of energy and growth characteristics in Commercial chickens.Some important candidate genes related to skeletal muscle growth is obtained.

A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data

ABSTRACTEarth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.

A GPU-Aware Parallel Index for Processing High-Dimensional Big Data

The problem of the curse of dimensionality for processing large high-dimensional datasets has been an open challenge. Numerous research efforts have been proposed for improving query performance in high-dimensional space through hierarchical indexing using the R-tree or its variants and exploring parallel processing of the R-tree on GPUs. Despite these existing efforts, the curse of dimensionality remains to be a grand challenge since the existing methods deteriorate drastically as the dimensionality of datasets increases. To cope with this problem, we present a novel GPU-aware parallel indexing method called G-tree, which offers consistent and stable performance in high-dimensional space. The rationale of the G-tree is to combine the efficiency of the R-tree in low-dimensional space with the massive parallel processing potential of GPUs by introducing a new data structure and three new optimization techniques to better utilize the GPU memory structure for accelerating both index search and index node access on GPUs. The first two optimizations promote effective parallelism utilization in GPU memory access. We dedicate the third optimization to further speed up the G-tree index by conducting progressive filtering using our dimension filters. We evaluate the validity of the G-tree approach by extensive experiments on high-dimensional datasets, showing that the G-tree outperforms the existing state-of-the-art techniques.

Multi-dimensional data indexing and range query processing via Voronoi diagram for internet of things

In a typical Internet of Things (IoT) deployment such as smart cities and Industry 4.0, the amount of sensory data collected from physical world is significant and wide-ranging. Processing large amount of real-time data from the diverse IoT devices is challenging. For example, in IoT environment, wireless sensor networks (WSN) are typically used for the monitoring and collecting of data in some geographic area. Spatial range queries with location constraints to facilitate data indexing are traditionally employed in such applications, which allows the querying and managing the data based on SQL structure. One particular challenge is to minimize communication cost and storage requirements in multi-dimensional data indexing approaches. In this paper, we present an energy- and time-efficient multidimensional data indexing scheme, which is designed to answer range query. Specifically, we propose data indexing methods which utilize hierarchical indexing structures, using binary space partitioning (BSP), such as kd-tree, quad-tree, k-means clustering, and Voronoi-based methods to provide more efficient routing with less latency. Simulation results demonstrate that the Voronoi Diagram-based algorithm minimizes the average energy consumption and query response time.

Quadrant-Based Minimum Bounding Rectangle-Tree Indexing Method for Similarity Queries over Big Spatial Data in HBase.

With the rapid development of mobile devices and sensors, effective searching methods for big spatial data have recently received a significant amount of attention. Owing to their large size, many applications typically store recently generated spatial data in NoSQL databases such as HBase. As the index of HBase only supports a one-dimensional row keys, the spatial data is commonly enumerated using linearization techniques. However, the linearization techniques cannot completely guarantee the spatial proximity of data. Therefore, several studies have attempted to reduce false positives in spatial query processing by implementing a multi-dimensional indexing layer. In this paper, we propose a hierarchical indexing structure called a quadrant-based minimum bounding rectangle (QbMBR) tree for effective spatial query processing in HBase. In our method, spatial objects are grouped more precisely by using QbMBR and are indexed based on QbMBR. The QbMBR tree not only provides more selective query processing, but also reduces the storage space required for indexing. Based on the QbMBR tree index, two query-processing algorithms for range query and kNN query are also proposed in this paper. The algorithms significantly reduce query execution times by prefetching the necessary index nodes into memory while traversing the QbMBR tree. Experimental analysis demonstrates that our method significantly outperforms existing methods.

Unsupervised image segmentation via Stacked Denoising Auto-encoder and hierarchical patch indexing

For decades, image segmentation is a hot research direction in computer vision because of its extensive and practical applications. In this work, we propose a method of image segmentation based on auto-encoders and hierarchical clustering algorithm, aiming at dealing with the segmentation problem in an unsupervised way. More specifically, this proposed method consists of two stages: training and segmenting. In training stage, we divide sample images into non-overlapped patches and extract deep-level feature representations from the patches using Stacked Denoising Auto-encoder (SDA), then we perform unsupervised and hierarchical K-means clustering on these feature representations and build an indexing tree structure. In segmenting stage, we achieve segmentation of an arbitrary image based on the indexing tree structure. This unsupervised methodology is demonstrated to be an improvement over traditional unsupervised segmentation methods owing to the introduction of sample images. Experimental evaluations on several benchmark datasets indicate that our algorithm outperforms several other methods in both time efficiency and accuracy.

Framework for Managing Uncertain Distributed Categorical Data

In recent years, data has become uncertain due to the flourishing advanced technologies that participate continuously and increasingly in producing large amounts of incomplete data. Often, many modern applications where uncertainty occurs are distributed in nature, e.g., distributed sensor networks, information extraction, data integration, social network, etc. Consequently, even though the data uncertainty has been studied in the past for centralized behavior, it is still a challenging issue to manage uncertainty over the data in situ. In this paper, we propose a framework to managing uncertain categorical data over distributed environments that is built upon a hierarchical indexing technique based on inverted index, and a distributed algorithm to efficiently process queries on uncertain data in distributed environment. Leveraging this indexing technique, we address two kinds of queries on the distributed uncertain databases 1) a distributed probabilistic thresholds query, where its answers satisfy the probabilistic threshold requirement; and 2) a distributed top-k-queries, optimizing, the transfer of the tuples from the distributed sources to the coordinator site and the time treatment. Extensive experiments are conducted to verify the effectiveness and efficiency of the proposed method in terms of communication costs and response time.

Semantic indexing with deep learning: a case study

Deep learning techniques, particularly convolutional neural networks (CNNs), are poised for widespread application in the research fields of information retrieval and natural language processing. However, there are very few publications addressing semantic indexing with deep learning. In particular, there are few studies of semantic indexing in biomedical literature because of several specific challenges including a vast amount of semantic labels from automatically annotating MeSH terms for MEDLINE citations and a massive collection with only the title and abstract information. In this paper, we introduce a novel CNN-based semantic indexing method for biomedical abstract document collections. First, we adaptively group word2vec categories into (coarse) subsets by clustering. Next, we construct a high-dimensional space representation with Wikipedia category extension, which contains more semantic information than bag-of-words. Thereafter, we design a hierarchical CNN indexing architecture for learning documents from a coarse- to fine-grained level with several multi-label training techniques. We believe that the low-dimensional representation of the output layer in CNNs should be more compact and effective. Finally, we perform comparative experiments for semantic indexing of biomedical abstract documents. Experimental results on the MEDLINE dataset show that our model achieves superior performance than conventional models.

Transcript-level expression analysis of RNA-seq experiments with HISAT, StringTie and Ballgown.

High-throughput sequencing of mRNA (RNA-seq) has become the standard method for measuring and comparing the levels of gene expression in a wide variety of species and conditions. RNA-seq experiments generate very large, complex data sets that demand fast, accurate and flexible software to reduce the raw read data to comprehensible results. HISAT (hierarchical indexing for spliced alignment of transcripts), StringTie and Ballgown are free, open-source software tools for comprehensive analysis of RNA-seq experiments. Together, they allow scientists to align reads to a genome, assemble transcripts including novel splice variants, compute the abundance of these transcripts in each sample and compare experiments to identify differentially expressed genes and transcripts. This protocol describes all the steps necessary to process a large set of raw sequencing reads and create lists of gene transcripts, expression levels, and differentially expressed genes and transcripts. The protocol's execution time depends on the computing resources, but it typically takes under 45 min of computer time. HISAT, StringTie and Ballgown are available from http://ccb.jhu.edu/software.shtml.

Tessellations in GIS: Part II–making changes

We attempt to describe the role of tessellated models of space within the discipline of Geographic Information Systems (GIS) – a speciality coming largely out of Geography and Land Surveying, where there was a strong need to represent information about the land’s surface within a computer system rather than on the original paper maps. We look at some of the basic operations in GIS, including dynamic and kinetic applications. We examine issues of topology and data structures, and produced a tessellation model that may be widely applied both to traditional “object” and “field” data types. The Part I of this study examined object and field spatial models, the Voronoi extension of objects, and the graphs that express the resulting adjacencies. The required data structures were also briefly described, along with 2D and 3D structures and hierarchical indexing. The importance of graph duality was emphasized. Here, this second paper builds on the structures described in the first, and examines how these may be modified: change may often be associated with either viewpoint or time. Incremental algorithms permit additional point insertion, and applications involving the addition of skeleton points, for map scanning, contour enrichment or watershed delineation and simulation. Dynamic algorithms permit skeleton smoothing, and higher order Voronoi diagram applications, including Sibson interpolation. Kinetic algorithms allow collision detection applications, free-Lagrange flow modeling, and pen movement simulation for map drawing. If desired these methods may be extended to 3D. Based on this framework, it can be argued that tessellation models are fundamental to our understanding and processing of geographical space, and provide a coherent framework for understanding the “space” in which we exist.