Efficient Data Structure Research Articles

Fully Dynamic Submodular Maximization over Matroids

Maximizing monotone submodular functions under a matroid constraint is a classic algorithmic problem with multiple applications in data mining and machine learning. We study this significant problem in the fully dynamic setting, where elements can be both inserted and deleted in real-time. Our main result is a randomized algorithm that maintains an efficient data structure with an \({\tilde{O}({k^{2}}{\varepsilon})}\) amortized update time (in the number of insertions and deletions) and yields a \({(4+O(\varepsilon))}\) -approximate solution with respect to the dynamic optimum, where \(k\) is the rank of the matroid.

An Improved Variable Neighborhood Search for the Reconfigurable Assembly Line Reconfiguring Problem

Throughout the past several decades, the manufacturing industry has been confronted with rapidly evolving market demands. The Reconfigurable Assembly Line (RAL) changes the quantity and variety of products produced through reconfiguration at different stages. The reconfiguration stages of RAL include the reallocation of operations and resources. However, existing research has neglected the resources. In the paper, a mixed integer linear program for the Reconfigurable Assembly Line Reconfiguring Problem is first formulated. Subsequently, an efficient data structure is developed to model the process precedence constraint and alternative resources. And an improved Variable Neighborhood Search which includes new problem-specific neighborhood structures and local search for resource selection is proposed to solve the problem. Computational results demonstrate the contribution of the proposed data structure and the improved procedure. Furthermore, the experiments verify the superiority and stability of the proposed method compared with other heuristic algorithms.

Concurrent Data Structures Made Easy

Design of an efficient thread-safe concurrent data structure is a balancing act between its implementation complexity and performance. Lock-based concurrent data structures, which are relatively easy to derive from their sequential counterparts and to prove thread-safe, suffer from poor throughput under even light multi-threaded workload. At the same time, lock-free concurrent structures allow for high throughput, but are notoriously difficult to get right and require careful reasoning to formally establish their correctness. In this work, we explore a solution to this conundrum based on a relatively old idea of batch parallelism---an approach for designing high-throughput concurrent data structures via a simple insight: efficiently processing a batch of a priori known operations in parallel is easier than optimising performance for a stream of arbitrary asynchronous requests. Alas, batch-parallel structures have not seen wide practical adoption due to (i) the inconvenience of having to structure multi-threaded programs to explicitly group operations and (ii) the lack of a systematic methodology to implement batch-parallel structures as simply as lock-based ones. We present OBatcher---a Multicore OCaml library that streamlines the design, implementation, and usage of batch-parallel structures. OBatcher solves the first challenge (how to use) by suggesting a new lightweight implicit batching design pattern that is built on top of generic asynchronous programming mechanisms. The second challenge (how to implement) is addressed by identifying a family of strategies for converting common sequential structures into the corresponding efficient batch-parallel versions, and by providing a library of functors that embody those strategies. We showcase OBatcher with a diverse set of benchmarks ranging from Red-Black and AVL trees to van Emde Boas trees, skip lists, and a thread-safe implementation of a Datalog solver. Our evaluation of all the implementations on large asynchronous workloads shows that (a) they consistently outperform the corresponding coarse-grained lock-based implementations---the only ones available in OCaml to date, and that (b) their throughput scales reasonably with the number of processors.

Parallel Implementation of the CCSDS Turbo Decoder on GPU

This paper presents a software turbo decoder on graphics processing units (GPU). Unlike previous works, the proposed decoding architecture for turbo codes mainly focuses on the Consultative Committee for Space Data Systems (CCSDS) standard. However, the information frame lengths of the CCSDS turbo codes are not suitable for flexible sub-frame parallelism design. To mitigate this issue, we propose a padding method that inserts several bits before the information frame header. To obtain low-latency performance and high resource utilization, two-level intra-frame parallelisms and an efficient data structure are considered. The presented Max-Log-Map decoder can be adopted to decode the Long Term Evolution (LTE) turbo codes with only small modifications. The proposed CCSDS turbo decoder at 10 iterations on NVIDIA RTX3070 achieves about 150 Mbps and 50 Mbps throughputs for the code rates 1/6 and 1/2, respectively.

GPU parallel implementation of a finite volume lattice Boltzmann method for incompressible flows

This work presents a graphics processing units (GPU) parallel algorithm of a cell-centered finite volume lattice Boltzmann method (FVLBM) on unstructured meshes. In the present GPU parallel algorithm, the parallelization is performed in the physical space. To reduce the frequency of GPU memory accesses, this algorithm develops coalesced access to GPU memory. In addition, to avoid the race for resources leading to data anomalies, such as dirty read or phantom read etc., and the double counting for flux calculation, the efficient face-based data structure often used for flux calculation in cells in the central processing unit (CPU) version of FVLBM is modified into a face-based data structure used for the fluxes on all faces, followed by a cell-based loop for the final residuals in all cells. Therefore, the proposed GPU parallel algorithm does not need to use the resource lock and retains the high efficiency of the face-based data structure in the fluxes computation to enhance its’ parallel efficiency. Additionally, to demonstrate the computational efficiency of the proposed GPU parallel algorithm, various benchmark studies are performed in this work by the proposed parallel scheme on a double precision NVIDIA GeForce RTX 3090Ti GPU card, including (a) the lid-driven flow in a two-dimensional (2D) square cavity, (b) a 2D flow past a cylinder, and (c) the lid-driven flow in a three-dimensional (3D) cubic cavity. The numerical results show that the proposed GPU parallel algorithm can be as accurate as the original CPU serial scheme with 1 to 2 orders of speedup.

Simona: a comprehensive R package for semantic similarity analysis on bio-ontologies

BackgroundBio-ontologies are keys in structuring complex biological information for effective data integration and knowledge representation. Semantic similarity analysis on bio-ontologies quantitatively assesses the degree of similarity between biological concepts based on the semantics encoded in ontologies. It plays an important role in structured and meaningful interpretations and integration of complex data from multiple biological domains.ResultsWe present simona, a novel R package for semantic similarity analysis on general bio-ontologies. Simona implements infrastructures for ontology analysis by offering efficient data structures, fast ontology traversal methods, and elegant visualizations. Moreover, it provides a robust toolbox supporting over 70 methods for semantic similarity analysis. With simona, we conducted a benchmark against current semantic similarity methods. The results demonstrate methods are clustered based on their mathematical methodologies, thus guiding researchers in the selection of appropriate methods. Additionally, we explored annotation-based versus topology-based methods, revealing that semantic similarities solely based on ontology topology can efficiently reveal semantic similarity structures, facilitating analysis on less-studied organisms and other ontologies.ConclusionsSimona offers a versatile interface and efficient implementation for processing, visualization, and semantic similarity analysis on bio-ontologies. We believe that simona will serve as a robust tool for uncovering relationships and enhancing the interoperability of biological knowledge systems.

HEMLAB Algorithm Applied to the High-Lift Japan Aerospace Exploration Agency Standard Model

The high-lift Japan Aerospace Exploration Agency (JAXA) standard model (HL-JSM) has been numerically analyzed in order to further validate the HEMLAB code for realistic aircraft configurations. The numerical algorithm is based on highly efficient edge-based data structure for a vertex-based finite volume algorithm on hybrid meshes. The data pattern is arranged to meet the requirements of a vertex-based finite volume algorithm by considering data access patterns and cache efficiency. A fully implicit version of the numerical algorithm has also been implemented based on the Portable, Extensible Toolkit for Scientific Computation (PETSc) library in order to improve the robustness of the algorithm. The resulting algebraic equations, including the one-equation Spalart–Allmaras, are solved in a monolithic manner using the restricted additive Schwarz preconditioner combined with the flexible generalized minimal residual method [FGMRES(m)] Krylov subspace algorithm. The numerical method is also combined with the metric-based anisotropic mesh refinement library Python Adaptive Mesh Geometry Suite–INRIA (pyAMG) from National Institute for Research in Digital Science and Technology in order to improve the numerical accuracy. The numerical algorithm is initially applied to the two-dimensional L1T2 (National High Lift Programme) high-lift system, and then the calculations around the HL-JSM are carried out at relatively high angles of attack. The numerical results with the anisotropic mesh refinement library pyAMG indicate significant improvements in numerical accuracy.

Mining alternative splicing patterns in scRNA-seq data using scASfind

Single-cell RNA-seq (scRNA-seq) is widely used for transcriptome profiling, but most analyses focus on gene-level events, with less attention devoted to alternative splicing. Here, we present scASfind, a novel computational method to allow for quantitative analysis of cell type-specific splicing events using full-length scRNA-seq data. ScASfind utilizes an efficient data structure to store the percent spliced-in value for each splicing event. This makes it possible to exhaustively search for patterns among all differential splicing events, allowing us to identify marker events, mutually exclusive events, and events involving large blocks of exons that are specific to one or more cell types.

Efficient Parallel Processing of R-Tree on GPUs

R-tree is an important multi-dimensional data structure widely employed in many applications for storing and querying spatial data. As GPUs emerge as powerful computing hardware platforms, a GPU-based parallel R-tree becomes the key to efficiently port R-tree-related applications to GPUs. However, traditional tree-based data structures can hardly be directly ported to GPUs, and it is also a great challenge to develop highly efficient parallel tree-based data structures on GPUs. The difficulty mostly lies in the design of tree-based data structures and related operations in the context of many-core architecture that can facilitate parallel processing. We summarize our contributions as follows: (i) design a GPU-friendly data structure to store spatial data; (ii) present two parallel R-tree construction algorithms and one parallel R-tree query algorithm that can take the hardware characteristics of GPUs into consideration; and (iii) port the vector map overlay system from CPU to GPU to demonstrate the feasibility of parallel R-tree. Experimental results show that our parallel R-tree on GPU is efficient and practical. Compared with the traditional CPU-based sequential vector map overlay system, our vector map overlay system based on parallel R-tree can achieve nearly 10-fold speedup.

Integrating OpenCV and Pandas for Enhanced Image Filtering and Color Detection

Image filtering and color detection in OpenCV is performed using functions like cv2.filter2D() for filtering, and cv2.cvtColor() for color space conversion and detection. These functions can be combined with other image processing techniques such as thresholding, erosion and dilation, and edge detection to perform more advanced tasks. The process typically starts with loading an image into Python using OpenCV's imread() function. From there, the image may be converted to a different color space using cvtColor() to make it easier to detect certain colors or features. Various image filters can then be applied to enhance the image and reduce noise or other unwanted features. This can include techniques such as blurring, sharpening, or edge detection, depending on the desired outcome. Once the image has been filtered, color detection can be performed using thresholding, which involves setting a threshold value and creating a binary image based on whether each pixel falls above or below the threshold. This can help to identify specific colors or features in the picture. Pandas are particularly useful for working with image data as it provides a flexible and efficient data structure called a DataFrame. This makes it easy to perform data manipulation tasks such as selecting, filtering, and grouping data, which can be useful for image processing tasks. Meanwhile, OpenCV provides a comprehensive set of tools for image filtering and color detection. These tools include a variety of filters, such as Gaussian, Median, and Bilateral filters, as well as thresholding techniques, such as binary and adaptive thresholding, which can be used to detect specific colors or features in an image. Keywords:- Image Processing, OpenCV, Color Detection, Image Filtering, cv2.filter2D, cv2.cvtColor, Thresholding, Erosion and Dilation, Edge Detection, Pandas DataFrame, Gaussian Filter, Median Filter, Bilateral Filter, Adaptive Thresholding, Feature Detection..

Advanced incremental erasable pattern mining from the time-sensitive data stream

Pattern mining has been actively advanced and studied in order to process data that is generated in real time, called incremental data. Erasable pattern mining is a concept that extracts erasable patterns as one of the fields belonging to pattern mining. Erasable patterns have useful information in terms of removal, unlike patterns found by other pattern mining methods. The former erasable pattern mining methods, which target incremental data, regard the importance of data identically. However, in real life, recent data are more important and useful than old data, so erasable pattern mining should also process the latest data more significantly in consideration of the feature of real-world data. For this reason, we propose an erasable pattern mining algorithm using an efficient data structure from the time-sensitive data stream. Since the suggested technique performs erasable pattern mining based on a damped window model, it is possible to relatively prioritize the importance of the recent data over the previous data. In addition, it is possible to quickly reflect and efficiently analyze the newly added incremental data due to a list based data structure of the proposed technique. The results of the various performance tests, which are conducted to compare our algorithm with state-of-the-art algorithms, are analyzed, and the analyses indicate that our approach is more efficient than the other latest approaches.

Parallel Topology-aware Mesh Simplification on Terrain Trees

We address the problem of performing a topology-aware simplification algorithm on a compact and distributed data structure for triangle meshes, the Terrain trees. Topology-aware operators have been defined to coarsen a Triangulated Irregular Network (TIN) without affecting the topology of its underlying terrain, i.e., without modifying critical features of the terrain, such as pits, saddles, peaks, and their connectivity. However, their scalability is limited for large-scale meshes. Our proposed algorithm uses a batched processing strategy to reduce both the memory and time requirements of the simplification process, and thanks to the spatial decomposition on the basis of Terrain trees, it can be easily parallelized. Also, since a Terrain tree after the simplification process becomes less compact and efficient, we propose an efficient post-processing step for updating hierarchical spatial decomposition. Our experiments on real-world TINs, derived from topographic and bathymetric LiDAR data, demonstrate the scalability and efficiency of our approach. Specifically, topology-aware simplification on Terrain trees uses 40% less memory and half the time compared to the most compact and efficient connectivity-based data structure for TINs. Furthermore, the parallel simplification algorithm on the Terrain trees exhibits a 12× speedup with an OpenMP implementation. The quality of the output mesh is not significantly affected by the distributed and parallel simplification strategy of Terrain trees, and we obtain similar quality levels compared to the global baseline method.

Simple & Optimal Quantile Sketch: Combining Greenwald-Khanna with Khanna-Greenwald

Estimating the ε-approximate quantiles or ranks of a stream is a fundamental task in data monitoring. Given a stream x_1,..., x_n from a universe \mathcalU with total order, an additive-error quantile sketch \mathcalM allows us to approximate the rank of any query y\in \mathcalU up to additive ε n error. In 2001, Greenwald and Khanna gave a deterministic algorithm (GK sketch) that solves the ε-approximate quantiles estimation problem using O(ε^-1 łog(ε n)) space \citegreenwald2001space ; recently, this algorithm was shown to be optimal by Cormode and Vesleý in 2020 \citecormode2020tight. However, due to the intricacy of the GK sketch and its analysis, over-simplified versions of the algorithm are implemented in practical applications, often without any known theoretical guarantees. In fact, it has remained an open question whether the GK sketch can be simplified while maintaining the optimal space bound. In this paper, we resolve this open question by giving a simplified deterministic algorithm that stores at most (2 + o(1))ε^-1 łog (ε n) elements and solves the additive-error quantile estimation problem; as a side benefit, our algorithm achieves a smaller constant factor than the \frac11 2 ε^-1 łog(ε n) space bound in the original GK sketch~\citegreenwald2001space. Our algorithm features an easier analysis and still achieves the same optimal asymptotic space complexity as the original GK sketch. Lastly, our simplification enables an efficient data structure implementation, with a worst-case runtime of O(łog(1/ε) + łog łog (ε n)) per-element for the ordinary ε-approximate quantile estimation problem. Also, for the related "weighted'' quantile estimation problem, we give efficient data structures for our simplified algorithm which guarantee a worst-case per-element runtime of O(łog(1/ε) + łog łog (ε W_n/w_\textrmmin )), achieving an improvement over the previous upper bound of \citeassadi2023generalizing.

PC-IDN: Fast 3D loop closure detection using projection context descriptor and incremental dynamic nodes

3D Loop Closure Detection (LCD) is a crucial technique for robotics, which can correct accumulated errors after long time exploration. Existing 3D LCD methods have been designed for mechanical LiDAR, limiting their performance for various solid-state LiDARs. In this paper, we propose a new LiDAR LCD method, PC-IDN, based on a novel descriptor called Projection Context (PC), and an efficient data structure named Incremental Dynamic Nodes (IDN). The PC considers side-view and describes scenes by leveraging distance and intensity values of the point clouds. Subsequently, the PC descriptors are collected in corresponding nodes for candidate scans selection. To ensure real-time performance, the IDN module dynamically adjusts the visited PCs in the database. The results demonstrate that our method is able to achieve better precision–recall and efficiency performance than existing 3D LCD methods.

Real-time Graph Visualization with JavaFX: Exploring Large-scale Network Structures

The ability to visualize and comprehend complex network structures is crucial in various domains, including social media analysis, computer networks, and transportation systems. However, visualizing large-scale graphs poses significant challenges due to computational limitations and rendering performance constraints. This research paper presents a novel approach to real-time graph visualization using JavaFX, a powerful Java-based framework for developing rich client applications. By leveraging efficient data structures, rendering optimizations, and multithreading techniques, our proposed system achieves real-time visualization of large-scale graphs, enabling users to explore and interact with dynamic network structures seamlessly. The system incorporates advanced layout algorithms and visual encodings to enhance the clarity and interpretability of the visualizations. Extensive experiments were conducted using real-world and synthetic datasets to evaluate the system's performance, scalability, and usability. The results demonstrate the effectiveness of our approach in rendering large graphs in real-time, outperforming existing techniques. Furthermore, a user study was conducted to assess the system's usability and gather feedback on interaction and exploration features, highlighting potential applications in various domains. Keywords— Graph Visualization, Real-time Rendering, JavaFX, Large-scale Networks, Layout Algorithms, Rendering Optimizations, User Interaction, Performance Evaluation.

On Breaking Truss-based and Core-based Communities

We introduce the general problem of identifying a smallest edge subset of a given graph whose deletion makes the graph community-free. We consider this problem under two community notions that have attracted significant attention: k -truss and k -core. We also introduce a problem variant where the identified subset contains edges incident to a given set of nodes and ensures that these nodes are not contained in any community: k -truss or k -core, in our case. These problems are directly applicable in social networks: The identified edges can be hidden by users or sanitized from the output graph; or in communication networks: the identified edges correspond to vital network connections. We present a series of theoretical and practical results. On the theoretical side, we show through non-trivial reductions that the problems we introduce are NP-hard and, in fact, hard to approximate. For the k -truss-based problems, we also show exact exponential-time algorithms, as well as a non-trivial lower bound on the size of an optimal solution. On the practical side, we develop a series of heuristics that are sped up by efficient data structures that we propose for updating the truss or core decomposition under edge deletions. In addition, we develop an algorithm to compute the lower bound. Extensive experiments on 11 real-world and synthetic graphs show that our heuristics are effective, outperforming natural baselines, and also efficient (up to two orders of magnitude faster than a natural baseline), thanks to our data structures. Furthermore, we present a case study on a co-authorship network and experiments showing that the removal of edges identified by our heuristics does not substantially affect the clustering structure of the input graph. This work extends a KDD 2021 paper, providing new theoretical results as well as introducing core-based problems and algorithms.

A Novel Skip Orthogonal List for Dynamic Optimal Transport Problem

Optimal transport is a fundamental topic that has attracted a great amount of attention from the optimization community in the past decades. In this paper, we consider an interesting discrete dynamic optimal transport problem: can we efficiently update the optimal transport plan when the weights or the locations of the data points change? This problem is naturally motivated by several applications in machine learning. For example, we often need to compute the optimal transport cost between two different data sets; if some changes happen to a few data points, should we re-compute the high complexity cost function or update the cost by some efficient dynamic data structure? We are aware that several dynamic maximum flow algorithms have been proposed before, however, the research on dynamic minimum cost flow problem is still quite limited, to the best of our knowledge. We propose a novel 2D Skip Orthogonal List together with some dynamic tree techniques. Although our algorithm is based on the conventional simplex method, it can efficiently find the variable to pivot within expected O(1) time, and complete each pivoting operation within expected O(|V|) time where V is the set of all supply and demand nodes. Since dynamic modifications typically do not introduce significant changes, our algorithm requires only a few simplex iterations in practice. So our algorithm is more efficient than re-computing the optimal transport cost that needs at least one traversal over all |E|=O(|V|^2) variables, where |E| denotes the number of edges in the network. Our experiments demonstrate that our algorithm significantly outperforms existing algorithms in the dynamic scenarios.

Hierarchical categories in colored searching

In colored range counting (CRC), the input is a set of points where each point is assigned a “color” (or a “category”) and the goal is to store them in a data structure such that the number of distinct categories inside a given query range can be counted efficiently. CRC has strong motivations as it allows data structure to deal with categorical data.However, colors (i.e., the categories) in the CRC problem do not have any internal structure, whereas this is not the case for many datasets in practice where hierarchical categories exist or where a single input belongs to multiple categories. Motivated by these, we consider variants of the problem where such structures can be represented. We define two variants of the problem called hierarchical range counting (HCC) and sub-category colored range counting (SCRC) and consider hierarchical structures that can either be a DAG or a tree. We show that the two problems on some special trees are in fact equivalent to other well-known problems in the literature. Based on these, we also give efficient data structures when the underlying hierarchy can be represented as a tree. We show a conditional lower bound for the general case when the existing hierarchy can be any DAG, through a reduction from the orthogonal vectors problem.

A parallel variable neighborhood search for [formula omitted]-neighbor facility location problems

In this paper, we employ the less is more approach to develop a Parallel Variable Neighborhood Search (VNS) algorithm for the α-neighbor p-center problem (αNpCP) and the α-neighbor p-median problem (αNpMP). The αNpCP and the αNpMP are generalizations of the p-center (pCP) and p-median (pMP) problems, respectively. In the α-neighbor problems, one seeks to open p facilities and assign each of the n customers to their closest α ones. The objective is to minimize the maximum distance of a customer to its αth facility, in the case of the αNpCP, and the sum of the distances from each customer to their α nearest facilities, in the case of the αNpMP. Our VNS adapts simple but efficient algorithms and data structures from the pCP and pMP literature to the αNpCP and αNpMP context. We also introduce an updated objective function for the αNpCP, which adds more information to the solution cost and helps the VNS to escape from local optima. Several experimental tests show that our VNS outperforms more complex state-of-the-art algorithms. Regarding the αNpCP, on 120 instances derived from the OR-library set, our algorithm improved best-known solutions for 22, with an average improvement of 34.26%; the overall gap on the 120 instances is 6.18% in favor of our algorithm. Moreover, on 231 instances derived from the TSPLIB set, we improved the solutions for 115, with an average improvement of 5.30%, and an overall improvement gap of 2.47% for all 231 instances. Considering the αNpMP results, our heuristic obtained better results than a heuristic from literature in all 80 instances tested, finding optimal solutions in all these instances.

Time and memory scalable algorithms for clustering tendency assessment of big data

Large-volume and high-dimensional big datasets are being generated quickly. They are expected to provide data-driven solutions for various pressing challenges such as cybersecurity, credit card fraud, network intrusion detection, etc. The visual assessment of clustering tendency (VAT) family of algorithms provides an excellent tool for assessing clustering tendency and subsequent clustering of these novel datasets to extract groups and anomalies to better understand the data generation phenomenon. However, VAT and improved VAT (iVAT) algorithms are plagued with O(n2) time and space complexity as they use Prim's algorithm for building Euclidean minimum spanning tree (EMST) of the data points, leaving it inapplicable to large datasets. This paper develops three novel time- and memory-scalable algorithms for fast computation of VAT EMST by reducing the search space of the following possible EMST edge and using an efficient data structure, the k-d tree. Next, we develop a novel approach to compute iVAT reordered dissimilarity image (RDI) in a novel memory-preserving manner using the EMST edge lengths calculated previously without having to compute memory intensive n×n VAT RDI. We experimented on several synthetic and real-life datasets to showcase that the proposed approaches can help quickly assess their clustering tendency in a memory-saving manner. We also demonstrate the applicability of the proposed methods for the anomaly detection task in large volumes of high-dimensional data.