Tree-based Data Structure Research Articles

CyCadas: accelerating interactive annotation and analysis of clustered cytometry data.

Single cell profiling by cytometry has emerged as a key technology in biology, immunology and clinical-translational medicine. The correct annotation, which refers to the identification of clusters as specific cell populations based on their marker expression, of clustered high-dimensional cytometry data, is a critical step of the analysis. Its accuracy determines the correct interpretation of the biological data. Despite the progress in various clustering algorithms, the annotation of clustered data still remains a manual, time consuming and error-prone task. We developed a user-friendly cluster annotation and differential abundance detection tool that can be applied on data generated with Self Organizing Map clustering algorithms, thus simplifying the annotation process of datasets that consist of hundreds or thousands of clusters. We present Cytometry Cluster Annotation and Differential Abundance Suite (CyCadas), a semi-automated software tool that facilitates cluster annotation in cytometry data by offering both visual and computational guidance. CyCadas addresses the critical need for efficient and accurate annotation of high-resolution clustered cytometry data, significantly reducing the time needed to perform the analysis compared to both manual gating approaches and manual annotation of clustered data. The tool features a user-friendly interface, visual tools enabling data exploration and automated threshold estimation to separate negative and positive marker expression. It facilitates the definition and annotation of cell phenotypes among multiple clusters in a tree-based data structure. Finally, it calculates the abundance of various cell populations across the conditions with statistical interpretation. It is an ideal resource for researchers aiming to streamline their cytometry workflow. CyCadas is available as open source at: https://github.com/DII-LIH-Luxembourg/cycadas.

Efficient Parallel Processing of R-Tree on GPUs

R-tree is an important multi-dimensional data structure widely employed in many applications for storing and querying spatial data. As GPUs emerge as powerful computing hardware platforms, a GPU-based parallel R-tree becomes the key to efficiently port R-tree-related applications to GPUs. However, traditional tree-based data structures can hardly be directly ported to GPUs, and it is also a great challenge to develop highly efficient parallel tree-based data structures on GPUs. The difficulty mostly lies in the design of tree-based data structures and related operations in the context of many-core architecture that can facilitate parallel processing. We summarize our contributions as follows: (i) design a GPU-friendly data structure to store spatial data; (ii) present two parallel R-tree construction algorithms and one parallel R-tree query algorithm that can take the hardware characteristics of GPUs into consideration; and (iii) port the vector map overlay system from CPU to GPU to demonstrate the feasibility of parallel R-tree. Experimental results show that our parallel R-tree on GPU is efficient and practical. Compared with the traditional CPU-based sequential vector map overlay system, our vector map overlay system based on parallel R-tree can achieve nearly 10-fold speedup.

VADIA-Verkle Tree-based Approach for Dealing Data Integrity Attacks in Opportunistic Mobile Social Networks

Opportunistic Mobile Social Networks (OMSN) are prone to data integrity attacks that jeopardize the integrity of the routing data inside the network. Among the several techniques that cope with these attacks in OMSN, tree-based approaches have proven to be the most effective due to its ease of data verification and ensurance in data integrity. This paper evaluates two tree-based data structures, Merkle tree and Verkle tree in terms of their effectiveness in detecting and preventing such attacks. The evaluation considers tree-generation time and proof-checking time, and the results demonstrate that the Verkle tree is a bandwidth-efficient solution and have lower proof-checking time, with a reduction of 98.33% than Merkle tree. This makes Verkle tree a good choice for handling data integrity attacks in OMSN. A Verkle tree-based approach, named VADIA, is proposed to handle data integrity attacks such as packet dropping, packet modification and pollution attack in OMSN. The proposed approach is implemented in the Opportunistic Network Environment (ONE) simulator and is shown to be effective in detecting malicious nodes and paths, reducing false negative rates, and improving accuracy in detecting malicious activities. The results demonstrate a 47%, 84% and 69% improvement in malicious node, malicious path and malicious activity detection over a period of time. Furthermore, the approach achieves an 80% reduction in false negative rates.

Efficient lattice-based revocable attribute-based encryption against decryption key exposure for cloud file sharing

Cloud file sharing (CFS) has become one of the important tools for enterprises to reduce technology operating costs and improve their competitiveness. Due to the untrustworthy cloud service provider, access control and security issues for sensitive data have been key problems to be addressed. Current solutions to these issues are largely related to the traditional public key cryptography, access control encryption or attribute-based encryption based on the bilinear mapping. The rapid technological advances in quantum algorithms and quantum computers make us consider the transition from the tradtional cryptographic primitives to the post-quantum counterparts. In response to these problems, we propose a lattice-based Ciphertext-Policy Attribute-Based Encryption(CP-ABE) scheme, which is designed based on the ring learing with error problem, so it is more efficient than that designed based on the learing with error problem. In our scheme, the indirect revocation and binary tree-based data structure are introduced to achieve efficient user revocation and dynamic management of user groups. At the same time, in order to further improve the efficiency of the scheme and realize file sharing across enterprises, the scheme also allows multiple authorities to jointly set up system parameters and manage distribute keys. Furthermore, by re-randomizing the user’s private key and update key, we achieve decryption key exposure resistance(DKER) in the scheme. We provide a formal security model and a series of security experiments, which show that our scheme is secure under chosen-plaintext attacks. Experimental simulations and evaluation analyses demonstrate the high efficiency and practicality of our scheme.

Multi-Resolution Grids in Earthquake Forecasting: The Quadtree Approach

ABSTRACTThe Collaboratory for the Study of Earthquake Predictability (CSEP) is an international effort to evaluate probabilistic earthquake forecasting models. CSEP provides the cyberinfrastructure and testing methods needed to evaluate earthquake forecasts. The most common way to represent a probabilistic earthquake forecast involves specifying the average rate of earthquakes within discrete spatial cells, subdivided into magnitude bins. Typically, the spatial component uses a single-resolution Cartesian grid with spatial cell dimensions of 0.1° × 0.1° in latitude and longitude, leading to 6.48 million spatial cells for the global testing region. However, the quantity of data (e.g., number of earthquakes) available to generate and test a forecast model is usually several orders of magnitude less than the millions of spatial cells, leading to a huge disparity in the number of earthquakes and the number of cells in the grid. In this study, we propose the Quadtree to create multi-resolution grid, locally adjusted mirroring the available data for forecast generation and testing, thus providing a data-driven resolution of forecasts. The Quadtree is a hierarchical tree-based data structure used in combination with the Mercator projection to generate spatial grids. It is easy to implement and has numerous scientific and technological applications. To facilitate its application to end users, we integrated codes handling Quadtrees into pyCSEP, an open-source Python package containing tools for evaluating earthquake forecasts. Using a sample model, we demonstrate how forecast model generation can be improved significantly in terms of information gain if constrained on a multi-resolution grid instead of a high-resolution uniform grid. In addition, we demonstrate that multi-resolution Quadtree grids lead to reduced computational costs. Thus, we anitcipate that Quadtree grids will be useful for developing and evaluating earthquake forecasts.

Branch-and-Price Approaches for Real-Time Vehicle Routing with Picking, Loading, and Soft Time Windows

We propose and evaluate branch-and-price approaches for vehicle routing problems with picking, loading, and soft time windows. This general type of vehicle routing problem is of particular relevance in the same-day delivery context, in which fast routing algorithms are required because of the commitment to real-time delivery in the presence of high customer order frequencies. To boost the performance of the branch-and-price algorithms, we introduce the new method of tree-compatible labeling with nondominance trees. This method represents cost functions by a fixed number of breakpoints and uses a specialized tree-based data structure to store Pareto-optimal labels. We prove the theoretical soundness of the new method and evaluate its performance numerically with respect to pricing, column generation, and branch-and-price. Our numerical results show that the method yields substantial performance gains. In particular, we show that, with the new method, branch-and-price is able to reliably generate within a few minutes close to optimal solutions for problem instances with 50 customers. By additional experiments with classic vehicle routing problems with hard time windows, we show that the performance gains of our method result from its ability to handle cost functions in the pricing step. Our approach is the first branch-and-price approach for vehicle routing with picking, loading, and soft time windows. As such, it represents an exact routing algorithm that is able to reliably satisfy the runtime requirements of real-time delivery services. Summary of Contribution: In this paper, we propose the first branch-and-price approaches for a general class of vehicle routing problems with picking, loading, and soft time windows. This problem class is of particular importance for real-time delivery services, for which customers expect to be served within only a few hours after their order has been placed. We provide a problem formulation that takes into account that short runtimes of routing algorithms are required, that the upper bounds of the customers’ time windows may be violated at a certain cost, and that a significant part of the delivery time window is consumed by picking orders from a warehouse and loading orders into vehicles. Solving the problem with branch-and-price requires the use of cost functions as elements of the labels in the pricing step. We propose a method that approximates these cost functions and leverages the computational performance of nondominance tree data structures for solving the pricing problem. We prove the theoretical soundness of this new method for exact branch-and-price, and we show numerically that the method leads to a significant performance increase of branch-and-price. The paper lies at the intersection of computing and operations research, in particular because our algorithms are designed to leverage the computational performance of advanced data structures for solving a combinatorial optimization problem.

A Distributed Framework for Large-scale Protein-protein Interaction Data Analysis and Prediction Using MapReduce

Protein-protein interactions are of great significance for human to understand the functional mechanisms of proteins. With the rapid development of high-throughput genomic technologies, massive protein-protein interaction (PPI) data have been generated, making it very difficult to analyze them efficiently. To address this problem, this paper presents a distributed framework by reimplementing one of state-of-the-art algorithms, i.e., CoFex, using MapReduce. To do so, an in-depth analysis of its limitations is conducted from the perspectives of efficiency and memory consumption when applying it for large-scale PPI data analysis and prediction. Respective solutions are then devised to overcome these limitations. In particular, we adopt a novel tree-based data structure to reduce the heavy memory consumption caused by the huge sequence information of proteins. After that, its procedure is modified by following the MapReduce framework to take the prediction task distributively. A series of extensive experiments have been conducted to evaluate the performance of our framework in terms of both efficiency and accuracy. Experimental results well demonstrate that the proposed framework can considerably improve its computational efficiency by more than two orders of magnitude while retaining the same high accuracy.

Storing data once in M-trees and PM-trees: Revisiting the building principles of metric access methods

Since the introduction of the M-tree, a fundamental tree-based data structure for indexing multi-dimensional information, several structural enhancements have been proposed. One of the most effective ones is the use of additional global pivots that resulted in the PM-tree. These two indexing structures, however, can store the same data element in multiple nodes. In this article, we revisit both the M-tree and the PM-tree to propose a new construction algorithm that stores data elements only once in the tree hierarchies. The main challenge to accomplish this, is to properly select data elements when an inner node split is needed. To address it, we propose an approach based on the use of aggregate nearest neighbor queries. The new algorithms enable building the search result set as data elements are evaluated for pruning during traversal, allowing faster retrieval of k-nearest neighbors and range searches. We conducted an extensive set of experiments with different real datasets. The results show that our proposed algorithms have considerably superior performance when compared with the standard M-tree and PM-tree.

Ultrafast Sample placement on Existing tRees (UShER) enables real-time phylogenetics for the SARS-CoV-2 pandemic.

As the SARS-CoV-2 virus spreads through human populations, the unprecedented accumulation of viral genome sequences is ushering in a new era of 'genomic contact tracing'-that is, using viral genomes to trace local transmission dynamics. However, because the viral phylogeny is already so large-and will undoubtedly grow many fold-placing new sequences onto the tree has emerged as a barrier to real-time genomic contact tracing. Here, we resolve this challenge by building an efficient tree-based data structure encoding the inferred evolutionary history of the virus. We demonstrate that our approach greatly improves the speed of phylogenetic placement of new samples and data visualization, making it possible to complete the placements under the constraints of real-time contact tracing. Thus, our method addresses an important need for maintaining a fully updated reference phylogeny. We make these tools available to the research community through the University of California Santa Cruz SARS-CoV-2 Genome Browser to enable rapid cross-referencing of information in new virus sequences with an ever-expanding array of molecular and structural biology data. The methods described here will empower research and genomic contact tracing for SARS-CoV-2 specifically for laboratories worldwide.

Toward a better understanding and evaluation of tree structures on flash SSDs

Solid-state drives (SSDs) are extensively used to deploy persistent data stores, as they provide low latency random access, high write throughput, high data density, and low cost. Tree-based data structures are widely used to build persistent data stores, and indeed they lie at the backbone of many of the data management systems used in production and research today. We show that benchmarking a persistent tree-based data structure on an SSD is a complex process, which may easily incur subtle pitfalls that can lead to an inaccurate performance assessment. At a high-level, these pitfalls stem from the interaction of complex software running on complex hardware. On the one hand, tree structures implement internal operations that have non-trivial effects on performance. On the other hand, SSDs employ firmware logic to deal with the idiosyncrasies of the underlying flash memory, which are well known to also lead to complex performance dynamics. We identify seven benchmarking pitfalls using RocksDB and WiredTiger, two widespread implementations of an LSM-Tree and a B+Tree, respectively. We show that such pitfalls can lead to incorrect measurements of key performance indicators, hinder the reproducibility and the representativeness of the results, and lead to suboptimal deployments in production environments. We also provide guidelines on how to avoid these pitfalls to obtain more reliable performance measurements, and to perform more thorough and fair comparisons among different design points.

Physically Inspired Data Compression and Management for Industrial Data Analytics

With the huge and ever-growing volume of industrial data, an enormous challenge of how this data should be handled, stored, and analyzed emerges. In this paper, we describe a novel method that facilitates automated signal parsing into a set of exhaustive and mutually exclusive segments, which is coupled with extraction of physically interpretable signatures that characterize those segments. The resulting numerical signatures can be used to approximate a wide range of signals within some arbitrary accuracy, thus effectively turning the aforementioned signal parsing and signature extraction procedure into a signal compression process. This compression converts raw signals into physically plausible and interpretable features that can then be directly mined in order to extract useful information via anomaly detection and characterization, quality prediction, or process control. In addition, distance-based unsupervised clustering is utilized to organize the compressed data into a tree-structured database enabling rapid searches through the data and consequently facilitating efficient data mining. Application of the aforementioned methods to multiple large datasets of sensor readings collected from several advanced manufacturing plants showed the feasibility of physics-inspired compression of industrial data, as well as tremendous gains in terms of search speeds when compressed data were organized into a distance-based, tree-structured database.

VRKSHA: a novel tree structure for time-profiled temporal association mining

Mining association patterns from a time-stamped temporal database distributed over finite time slots is implicitly associated with task of scanning the input database. Finding supports of itemsets requires scanning the input database. Database scan can be performed in either snapshot or lattice-based approach. Sequential and SPAMINE methods for similarity-profiled association pattern mining originally proposed by Jin Soung Yoo and Sashi Sekhar are based on the snapshot database scan and lattice scan, respectively. Snapshot database scan involves scanning multi-time slot database time slot by time slot. The major limitation of Sequential method is the requirement to retain original temporal database in the disk for finding itemset support computations. In this paper, a novel multi-tree structure called VRKSHA is proposed that eliminates the need to store the original temporal database in the memory and also eliminates the need to retain database in memory. The basic idea is to generate a compressed time-stamped temporal tree and use this multi-tree structure to obtain true supports of temporal itemsets for a given time slot. Discovery of similar temporal itemsets is based on finding distance between temporal itemset and reference w.r.t each time slot and validating whether the computed distance satisfies specified user dissimilarity threshold. A pattern is pruned if the dissimilarity condition fails at any given time slot well before computing true support of itemset w.r.t all time slots. The advantage of proposed Sequential approach is from the fact that it is a single database scan approach excluding the initial database scan performed for computing true supports of singleton items. VRKSHA overcomes the major limitation of retaining database in memory that is required by SPAMINE, G-SPAMINE, MASTER algorithms. Experiment results prove that computational time and memory consumed by VRKSHA are significantly very much better than by approaches such as Naive, Sequential, SPAMINE, and G-SPAMINE. To the best of our survey and knowledge, VRKSHA is the pioneering work to introduce and propose a compressed tree-based data structure for mining similarity-profiled temporal association patterns in the area of time-profiled temporal association mining.

An approach for high utility pattern mining

Mining high utility pattern has become prominent as it provides semantic significance (utility/weighted patterns) associated with items in a transaction. Data analysis and respective strategies for mining high utility patterns is important in real world scenarios. Recent researches focused on high utility pattern mining using tree-based data structure which suffers greater computation time, since they generate multiple tree branches. To cope up with these problems, this work proposes a novel binary tree-based data structure with average maximum utility (AvgMU) and mining algorithm to mine high utility patterns from incremental data which reduces tree constructions and computation time. The proposed algorithms are implemented using synthetic, real datasets and compared with state-of-the-art tree-based algorithms. Experimental results show that the proposed work has better performance in terms of running time, scalability and memory consumption than the other algorithms compared in this research work.

An approach for high utility pattern mining

Mining high utility pattern has become prominent as it provides semantic significance (utility/weighted patterns) associated with items in a transaction. Data analysis and respective strategies for mining high utility patterns is important in real world scenarios. Recent researches focused on high utility pattern mining using tree-based data structure which suffers greater computation time, since they generate multiple tree branches. To cope up with these problems, this work proposes a novel binary tree-based data structure with average maximum utility (AvgMU) and mining algorithm to mine high utility patterns from incremental data which reduces tree constructions and computation time. The proposed algorithms are implemented using synthetic, real datasets and compared with state-of-the-art tree-based algorithms. Experimental results show that the proposed work has better performance in terms of running time, scalability and memory consumption than the other algorithms compared in this research work.

Ode on a random urn (functional pearl)

We present the urn, a simple tree-based data structure that supports sampling from and updating discrete probability distributions in logarithmic time. We avoid the usual complexity of traditional self-balancing binary search trees by not keeping values in a specific order. Instead, we keep the tree maximally balanced at all times using a single machine word of overhead: its size. Urns provide an alternative interface for the frequency combinator from the QuickCheck library that allows for asymptotically more efficient sampling from dynamically-updated distributions. They also facilitate backtracking in property-based random testing, and can be applied to such complex examples from the literature as generating well-typed lambda terms or information flow machine states, demonstrating significant speedups.

Efficient Move Evaluations for Time-Dependent Vehicle Routing Problems

In this paper, we introduce several new methods for efficiently evaluating moves in neighborhood search heuristics for routing problems with time-dependent travel times. We consider both the case that route duration is constrained and the case that route duration appears in the objective. We observe that the composition of piecewise linear functions can be evaluated in various orders when computing the route duration. We use this to develop a new tree-based data structure to improve the complexity of computations and memory usage. This approach also allows us to present methods that have the best known computational complexity but that do not require a lexicographic order of search. Our numerical experiments illustrate the trade-off between computation time and memory usage among the various methods. For 1,000 customer instances, our methods are able to speed up a construction heuristic by up to 8.89 times and an exchange neighborhood improvement heuristic by up to 3.94 times, without requiring excessive amounts of memory.

RECOGNITION OF MULTI-VIEW HUMAN FACES BASED ON MACHINE INTELLIGENCE USING KLT ALGORITHM

Nowadays Image Processing has become a proficient domain due to the prolific techniques like face detection and face recognition. They play an important role in our society due to their use in wide range of applications such as surveillance, security, banking, and multimedia. One of major challenges faced in this technique of face recognition is difficulty in handling arbitrary pose variations in three dimensional representations. In video retrieval system, many approaches have been developed for recognition across pose variations and to assume the face poses to be known. These constraints made it semi-automatic. In this paper we propose a fully automatic method for multi-view face recognition of improving the accuracy or efficiency using local binary patterns. It uses tree-based data structure to create sub-grids. In this system we use KLT algorithm to detect and extract features automatically by using Eigen vectors and estimation of hessian value.

Conservative interpolation between general spherical meshes

Abstract. An efficient, local, explicit, second-order, conservative interpolation algorithm between spherical meshes is presented. The cells composing the source and target meshes may be either spherical polygons or latitude–longitude quadrilaterals. Second-order accuracy is obtained by piece-wise linear finite-volume reconstruction over the source mesh. Global conservation is achieved through the introduction of a supermesh, whose cells are all possible intersections of source and target cells. Areas and intersections are computed exactly to yield a geometrically exact method. The main efficiency bottleneck caused by the construction of the supermesh is overcome by adopting tree-based data structures and algorithms, from which the mesh connectivity can also be deduced efficiently.The theoretical second-order accuracy is verified using a smooth test function and pairs of meshes commonly used for atmospheric modelling. Experiments confirm that the most expensive operations, especially the supermesh construction, have O(NlogN) computational cost. The method presented is meant to be incorporated in pre- or post-processing atmospheric modelling pipelines, or directly into models for flexible input/output. It could also serve as a basis for conservative coupling between model components, e.g., atmosphere and ocean.

High utility pattern mining over data streams with sliding window technique

Processing changeable data streams in real time is one of the most important issues in the data mining field due to its broad applications such as retail market analysis, wireless sensor networks, and stock market prediction. In addition, it is an interesting and challenging problem to deal with the stream data since not only the data have unbounded, continuous, and high speed characteristics but also their environments have limited resources. High utility pattern mining, meanwhile, is one of the essential research topics in pattern mining to overcome major drawbacks of the traditional framework for frequent pattern mining that takes only binary databases and identical item importance into consideration. This approach conducts mining processes by reflecting characteristics of real world databases, non-binary quantities and relative importance of items. Although relevant algorithms were proposed for finding high utility patterns in stream environments, they suffer from a level-wise candidate generation-and-test and a large number of candidates by their overestimation techniques. As a result, they consume a huge amount of execution time, which is a significant performance issue since a rapid process is necessary in stream data analysis. In this paper, we propose an algorithm for mining high utility patterns from resource-limited environments through efficient processing of data streams in order to solve the problems of the overestimation-based methods. To improve mining performance with fewer candidates and search space than the previous ones, we develop two techniques for reducing overestimated utilities. Moreover, we suggest a tree-based data structure to maintain information of stream data and high utility patterns. The proposed tree is restructured by our updating method with decreased overestimation utilities to keep up-to-date stream information whenever the current window slides. Our approach also has an important effect on expert and intelligent systems in that it can provide users with more meaningful information than traditional analysis methods by reflecting the characteristics of real world non-binary databases in stream environments and emphasizing on recent data. Comprehensive experimental results show that our algorithm outperforms the existing sliding window-based one in terms of runtime efficiency and scalability.

A novel approach for mining frequent patterns from incremental data

Incremental data can be defined as dynamic data that changes as time advances. Mining frequent patterns from such data is costly as most of the approaches need repetitive scanning and generates a large number of candidate keys. It is important to develop an efficient approach to enhance the performance of mining. This paper proposes a novel tree-based data structure for mining frequent pattern of incremental data called Tree for Incremental Mining of Frequent Pattern (TIMFP) which is compact as well as almost balanced. TIMFP is also suitable for interactive mining (build once and mine many). We have compared TIMFP with canonical-order tree (CanTree), Compressed and Arranged Transaction Sequences (CATS) Tree and Incremental Mining Binary Tree (IMBT). The experimental results show that the proposed work has better performance than other data structures compared in the paper in terms of time required for constructing the tree as well as mining frequent patterns from the tree.