Hybrid Data Structure Research Articles

Model for Automated Generation of DTM with Hybrid Data Structure Using Point Clouds

Model for Automated Generation of DTM with Hybrid Data Structure Using Point Clouds

Leveraging ontochains for distributed public transit ticketing: An investigation with the system for ticketing ubiquity with blockchains

AbstractTransport ticketing systems are crucial for enabling seamless, efficient, and sustainable mobility. However, traditional ticketing systems face limitations such as ticket fraud, lack of interoperability, and the inability to adapt to changes in the dynamic transport networks they issue tickets for. This paper presents new approaches to the system for ticketing ubiquity with blockchains (STUB), a novel smart transport ticketing solution that employs ontochains, a hybrid data structure combining blockchains and ontologies to form a type of distributed knowledge graph. STUB aims to address these limitations by providing a secure, transparent, and flexible platform for ticket issuance, validation, and management. We describe the key components and workflow of the STUB system, highlighting the use of transport network ontologies for modelling complex relationships within transportation systems and blockchain technologies for transport network ontology's state. Additionally, the implementation of Merkle proofs for efficient and secure validation between on‐chain and off‐chain ontological data is discussed. Finally, a simulated toy example is used as a lightweight proof‐of‐concept to demonstrate these capabilities. The proposed STUB system has the potential to significantly impact the future of transportation ticketing by offering a more seamless, interoperable, and user‐friendly experience whilst addressing the challenges associated with traditional ticketing systems.

NeuralVDB: High-resolution Sparse Volume Representation using Hierarchical Neural Networks

We introduce NeuralVDB, which improves on an existing industry standard for efficient storage of sparse volumetric data, denoted VDB [Museth 2013 ], by leveraging recent advancements in machine learning. Our novel hybrid data structure can reduce the memory footprints of VDB volumes by orders of magnitude, while maintaining its flexibility and only incurring small (user-controlled) compression errors. Specifically, NeuralVDB replaces the lower nodes of a shallow and wide VDB tree structure with multiple hierarchical neural networks that separately encode topology and value information by means of neural classifiers and regressors respectively. This approach is proven to maximize the compression ratio while maintaining the spatial adaptivity offered by the higher-level VDB data structure. For sparse signed distance fields and density volumes, we have observed compression ratios on the order of 10× to more than 100× from already compressed VDB inputs, with little to no visual artifacts. Furthermore, NeuralVDB is shown to offer more effective compression performance compared to other neural representations such as Neural Geometric Level of Detail [Takikawa et al. 2021 ], Variable Bitrate Neural Fields [Takikawa et al. 2022a ], and Instant Neural Graphics Primitives [Müller et al. 2022 ]. Finally, we demonstrate how warm-starting from previous frames can accelerate training, i.e., compression, of animated volumes as well as improve temporal coherency of model inference, i.e., decompression.

The Doubly Linked Tree of Singly Linked Rings: Providing Hard Real-Time Database Operations on an FPGA

We present a hardware data structure specifically designed for FPGAs that enables the execution of the hard real-time database CRUD operations using a hybrid data structure that combines trees and rings. While the number of rows and columns has to be limited for hard real-time execution, the actual content can be of any size. Our structure restricts full navigational freedom to every but the leaf layer, thus keeping the memory overhead for the data stored in the leaves low. Although its nodes differ in function, all have exactly the same size and structure, reducing the number of cascaded decisions required in the database operations. This enables fast and efficient hardware implementation on FPGAs. In addition to the usual comparison with known data structures, we also analyze the tradeoff between the memory consumption of our approach and a simplified version that is doubly linked in all layers.

UniCon: A unified star-operation to efficiently find connected components on a cluster of commodity hardware.

With a cluster of commodity hardware, how can we efficiently find all connected components of an enormous graph containing hundreds of billions of nodes and edges? The problem of finding connected components has been used in various applications such as pattern recognition, reachability indexing, graph compression, graph partitioning, and random walk. Several studies have been proposed to efficiently find connected components in various environments. Most existing single-machine and distributed-memory algorithms are limited in scalability as they have to load all data generated during the process into the main memory; they require expensive machines with vast memory capacities to handle large graphs. Several MapReduce algorithms try to handle large graphs by exploiting distributed storage but fail due to data explosion problems, which is a phenomenon that significantly increases the size of data as the computation proceeds. The latest MapReduce algorithms resolve the problem by proposing two distinguishing star-operations and executing them alternately, while the star-operations still cause massive network traffic as a star-operation is a distributed operation that connects each node to its smallest neighbor. In this paper, we unite the two star-operations into a single operation, namely UniStar, and propose UniCon, a new distributed algorithm for finding connected components in enormous graphs using UniStar. The partition-aware processing of UniStar effectively resolves the data explosion problems. We further optimize UniStar by filtering dispensable edges and exploiting a hybrid data structure. Experimental results with a cluster of 10 cheap machines each of which is equipped with Intel Xeon E3-1220 CPU (4-cores at 3.10GHz), 16GB RAM, and 2 SSDs of 1TB show that UniCon is up to 13 times faster than competitors on real-world graphs. UniCon succeeds in processing a tremendous graph with 129 billion edges, which is up to 4096 times larger than graphs competitors can process.

A hybrid adjacency and time-based data structure for analysis of temporal networks

Dynamic or temporal networks enable representation of time-varying edges between nodes. Conventional adjacency-based data structures used for storing networks such as adjacency lists were designed without incorporating time and can thus quickly retrieve all edges between two sets of nodes (a node-based slice) but cannot quickly retrieve all edges that occur within a given time interval (a time-based slice). We propose a hybrid data structure for storing temporal networks that stores edges in both an adjacency dictionary, enabling rapid node-based slices, and an interval tree, enabling rapid time-based slices. Our hybrid structure also enables compound slices, where one needs to slice both over nodes and time, either by slicing first over nodes or slicing first over time. We further propose an approach for predictive compound slicing, which attempts to predict whether a node-based or time-based compound slice is more efficient. We evaluate our hybrid data structure on many real temporal network data sets and find that they achieve much faster slice times than existing data structures with only a modest increase in creation time and memory usage.

Improved A-Star Algorithm for Long-Distance Off-Road Path Planning Using Terrain Data Map

To overcome the limitation of poor processing times for long-distance off-road path planning, an improved A-Star algorithm based on terrain data is proposed in this study. The improved A-Star algorithm for long-distance off-road path planning tasks was developed to identify a feasible path between the start and destination based on a terrain data map generated using a digital elevation model. This study optimised the algorithm in two aspects: data structure, retrieval strategy. First, a hybrid data structure of the minimum heap and 2D array greatly reduces the time complexity of the algorithm. Second, an optimised search strategy was designed that does not check whether the destination is reached in the initial stage of searching for the global optimal path, thus improving execution efficiency. To evaluate the efficiency of the proposed algorithm, three different off-road path planning tasks were examined for short-, medium-, and long-distance path planning tasks. Each group of tasks corresponded to three different off-road vehicles, and nine groups of experiments were conducted. The experimental results show that the processing efficiency of the proposed algorithm is significantly better than that of the conventional A-Star algorithm. Compared with the conventional A-Star algorithm, the path planning efficiency of the improved A-Star algorithm was accelerated by at least 4.6 times, and the maximum acceleration reached was 550 times for long-distance off-road path planning. The simulation results show that the efficiency of long-distance off-road path planning was greatly improved by using the improved algorithm.

On fast enumeration of maximal cliques in large graphs

Maximal Clique Enumeration (MCE) is a fundamental and challenging problem in graph theory and various network applications. Numerous algorithms have been proposed in the past decades, however, only a few of them focus on improving the practical efficiency in large graphs. To this end, we propose an efficient algorithm called FACEN based on the Bron–Kerbosch framework. To optimize the memory and time consumption, we apply a hybrid data structure with adjacency list and partial adjacency matrix, and introduce a dynamic pivot selection rule based on the degeneracy order. FACEN is evaluated on a total of 64 benchmark instances from various sources. Computational results indicate that the proposed algorithm is highly competitive with the current leading MCE methods. In particular, our algorithm is able to enumerate all maximal cliques on the tested real-world social networks with millions of vertices and edges. For very large graphs, we provide an additional experiment for solving the MCE variant with lower bound, and investigate the benefits of FACEN.

Scheduling algorithm based on preemptive priority and hybrid data structure for cognitive radio technology with vehicular ad hoc network

There are different types of messages containing different priorities in vehicular ad hoc networks. Hence, queue rescheduling is required according to priorities of arrived messages. In this study, a data structure with less computational complexity is proposed to minimise queuing delay. Further, to maintain quality of service, preemptive priority is applied to time-bound safety messages by transferring non-safety messages to other bands using the concept of cognitive radio technology. The time-bound messages are transmitted using the dedicated short-range communication spectrum without the need for spectrum sensing by vehicles. The other messages with no deadline constraint are switched to other bands near-dedicated short-range communication spectrum. The results show that 6.25% improvement in packet delivery ratio of cognitive radio-enabled preemptive priority is achieved in comparison to existing cognitive radio protocol. The delay shows a slight increment of 1.1%. The packet delivery ratio of cognitive radio-enabled non-preemptive priority is improved by 3.24% while the delay is improved by 3.17%. The data storage required for storing sensing data of 50 channels for 10 days is only 45 Mb.

WAVETRISK-1.0: an adaptive wavelet hydrostatic dynamical core

Abstract. This paper presents the new adaptive dynamical core wavetrisk. The fundamental features of the wavelet-based adaptivity were developed for the shallow water equation on the β plane and extended to the icosahedral grid on the sphere in previous work by the authors. The three-dimensional dynamical core solves the compressible hydrostatic multilayer rotating shallow water equations on a multiscale dynamically adapted grid. The equations are discretized using a Lagrangian vertical coordinate version of the dynamico model. The horizontal computational grid is adapted at each time step to ensure a user-specified relative error in either the tendencies or the solution. The Lagrangian vertical grid is remapped using an arbitrary Lagrangian–Eulerian (ALE) algorithm onto the initial hybrid σ-pressure-based coordinates as necessary. The resulting grid is adapted horizontally but uniform over all vertical layers. Thus, the three-dimensional grid is a set of columns of varying sizes. The code is parallelized by domain decomposition using mpi, and the variables are stored in a hybrid data structure of dyadic quad trees and patches. A low-storage explicit fourth-order Runge–Kutta scheme is used for time integration. Validation results are presented for three standard dynamical core test cases: mountain-induced Rossby wave train, baroclinic instability of a jet stream and the Held and Suarez simplified general circulation model. The results confirm good strong parallel scaling and demonstrate that wavetrisk can achieve grid compression ratios of several hundred times compared with an equivalent static grid model.

Chaintegrity: blockchain-enabled large-scale e-voting system with robustness and universal verifiability

Blockchain-enabled voting (BEV) systems have emerged as the next generation of modern electronic voting (e-voting) systems, because the immutable property of the blockchain has made itself a perfect distributed ballot box. Further, recent investigations have utilized the smart contract to build a decentralized autonomous voting application over blockchain. We identify nine critical desiderata, such as scalability, verifiability, and robustness, that a BEV system can and should achieve. However, we find that existing BEV systems violate at least one of the nine desiderata. In light of this deficiency, we propose a novel BEV system, named Chaintegrity, that fulfills all the specified desiderata. In addition, to make our system more cost-effective, we also propose a hybrid data structure which combines the counting Bloom filter and the Merkle hash tree for fast authentication. To enhance robustness, we as well introduce the code-voting technique as a component in our system. Our empirical results also show that our system achieves high efficiency and enjoys low computational and communication overhead.

LevelFilesSet: An efficient Data Structure for Scalable Web Tiled Map Management Systems

Abstract. Modern map visualizations are built using data structures for storing tile images, while their main concerns are to maximize efficiency and usability. The core functionality of a web tiled map management system is to provide tile images to the end user; several tiles combined construe the web map. To achieve this, several data structures are showcased and analyzed. Specifically, this paper focuses on the SimpleFormat, which stores the tiles directly on the file system; the ImageBlock, which divides each tile folder (a folder where the tile images are stored) into subfolders that contain multiple tiles prior to storing the tiles on the file system; the LevelFilesSet, a data structure that creates dedicated Random-Access files, wherein the tile dataset is first stored and then parsed in files to retrieve the tile images; and, finally, the LevelFilesBlock, a hybrid data structure which combines ImageBlock and LevelFilesSet data structures. This work signifies the first time this hybrid approach has been implemented and applied in a web tiled map context. The JDBC API was used for integrating with the PostgreSQL database. This database was then used to conduct cross-testing amongst the data structures. Subsequently, several benchmark tests on local and cloud environments are developed anew and assessed under different system configurations to compare the data structures and provide a thorough analysis of their efficiency. These benchmarks showcased the efficiency of LevelFilesSet, which retrieved tiles up to 3.3 times faster than the other data structures. Peripheral features and principles of implementing scalable web tiled map management systems among different software architectures and system configurations are analyzed and discussed.

DESCRIPTION AND IDENTIFICATION OF TEMPORAL REGULARITIES FOR FUZZY TIME SERIES WITH APPLICATION OF HYBRID OLS-PATTERNS

ObjectivesThe aim of the research is to develop the principle of storing data templates to take their temporal natureinto account, making it possible to reduce decision-making times.In order to describe and identify temporal patterns in fuzzy time series behaviour in real time, the task was set to develop a hybrid data structure that allows for a consideration of sequences of fuzzy values formed from clear observable data as well as a determination of the length of these sequences and possible uneven time intervals between the observations.MethodsThe article discussesan approach to formalising the description of temporal cause-effect relationships between events occurring at the object location as well as that of its environment, based on a set of singly-connected lists of triplets. Each triplet contains a fuzzy linguistic variable, the duration of its observation and the permitted interval of observation of insignificant data.ResultsAn algorithm for detecting knowledge base patterns in real time was developed, taking into account the possibility of a time shift in observing long sequences of identical values of the observed value. The possibility of partial data overlapping corresponding to triplets of different patterns is taken into account. The proposed hybrid pattern makes it possible to accelerate the detection of temporal regularities in the data.ConclusionScientific results are presented by the developed structure for storing information on temporal regularities in data, based on a singly linked linear list, as well as an algorithm for finding regularities in observational data using a set of OLS-patterns. The advantage of this structure and algorithm in comparison with the known ways of storing and analysing temporal data is a reduction in the amount of memory necessary for storing templates in the knowledge base, as well as the possibility of applying OLS patterns for decisionmaking purposes.

GridDS: a hybrid data structure for residue computation in point set matching

Registration of 3D point clouds is a problem that arises in a variety of research areas such as computer vision, computer graphics and computational geometry. This situation causes most papers in the area to focus on solving practical problems by using data structures often developed in theoretical contexts. Consequently, discrepancies arise between asymptotic cost and experimental performance. The point cloud registration or matching problem encompasses many different steps. Among them, the computation of the distance between two point sets (often refereed to as residue computation) is crucial and can be seen as an aggregate of range searching or nearest neighbor searching. In this paper, we aim at providing theoretical analysis and experimental performance of range searching and nearest neighbor data structures applied to 3D point cloud registration. Performance of widely used data structures such as compressed octrees, KDtrees, BDtrees and regular grids is reported. Additionally, we present a new hybrid data structure named GridDS, which combines a regular grid with some preexisting “inner” data structure in order to meet the best asymptotic bounds while also obtaining the best performance. The experimental evaluation in both synthetic and real data demonstrates that the hybrid data structures built using GridDS improve the running times of the single data structures. Thus, as we have studied the performances of the state-of-the-art techniques managing to improve their respective running times thanks to GridDS, this paper presents the best running time for point cloud residue computation up to date.

Grid k-d tree approach for point location in polyhedral data sets – application to explicit MPC

ABSTRACT Explicit model predictive control (EMPC) moves the online computational burden of linear model predictive control (MPC) to offline computation by using multi-parametric programming which produces control laws defined over a set of polyhedral regions in the state space. The online computation of EMPC is to find the corresponding control law according to a given state, this is called the point location problem. This paper deals with efficient point location in larger polyhedral data sets. The authors propose a hybrid data structure, grid k-d tree (GKDT), which is constructed by the k-dimensional tree (k-d tree), hash table and binary search tree (BST). The main part of GKDT is a multiple branch tree which constructs subtrees by splitting the polyhedral region into several equal grids based on the k-d tree and is traversed by the hash function on each level. GKDT has a high search efficiency, even though it needs much more storage memory. A complexity analysis of the approach in the runtime and storage requirements is provided. Advantages of the method are supported by two examples in the paper.

K-d tree based approach for point location problem in explicit model predictive control

Explicit Model Predictive Control (EMPC) produces control laws defined over a set of polyhedral regions in the state space, and the online computation of EMPC is to find the corresponding control law according to a given state by searching in a lookup table, called point location problem. This paper presents an approach of constructing a hybrid data structure called constructed k-d tree(CKDT), which combines the k-dimensional tree (k-d tree) with the binary search tree (BST) for point location in such polyhedral sets. To maintain a ‘full’ and balanced constructed tree the number of affine control laws is used as the basis for choosing the candidate hyperplanes (HPs) during the main construction process of CKDT, thus increasing offline efficiency by reducing the number of candidate HPs requiring computation. This methodology can be applied to the EMPC of high dimensional problems as the k-d tree – a main part of the CKDT approach - has already been successfully used to solve high dimensional problems in the field of computer science and engineering. The method involves a trade-off between memory storage requirement and online efficiency. A complexity analysis of the approach in the runtime and storage requirements is provided. The advantages of the method are supported by two examples in the paper.

Model driven development of hybrid databases using lightweight metamodel extensions

ABSTRACTThe transition from traditional to Big Data technologies has put new challenges in front of existing database modelling techniques. The ‘Design then implement’ approach to relational database modelling, could not be fully applied to ‘Discover and analyze’ databases with rapid data growth. In the context of Big Data, models have preserved a descriptive role: they are used to analyze and understand the architecture of hybrid data structures. From the Model-Driven Engineering point of view, several questions emerge: Could Big Data and hybrid database models represent a formal specification for a solution, which can be transformed to an executable specification for a specific target platform? Does the refining mechanism have its place in the process of hybrid data structure development? An approach to modelling hybrid databases, based on empirically validated concepts of Model-Driven Engineering, is presented in this paper. The approach was applied during the implementation of Precision Agriculture Based Information Service (PAIS) application – a web-based service for the collection and archiving of diverse data on crops, and their presentation to farmers.

Mobile web-based lightweight and real-time roaming algorithm for large-scale WebBIM scenes

With the popularity of mobile terminals, including smart phones, there is a great push for users to share large-scale data on phone webpages. This study considers building information modeling (BIM) as the core smart construction technology and leverages, its classical data format — industry foundation classes (IFC) as the input data format, and presents a mobile & web-based lightweight and real-time roaming algorithm for large-scale WebBIM scenes. To comply with the semantics information in IFC files and with a buildings structure, our method focuses on creating a more lightweight product, by not relying exclusively on geometry faces as ordinary visualization programs do. Our main idea for optimizing mobile, web-based, lightweight and real-time roaming is threefold: reduce redundancies, generate an efficient hybrid data structure and design dynamic load/unload strategies. In the study, a prototype system was integrated and leveraged for experimentation. The final performance analysis shows that we can significantly reduce the redundancy to about 70 percent of the raw BIM scenes. Meanwhile, our method can maintain the FPS at around 30 and memory consumption at about 400 MB with an increasing scene scale.

Practical Path Guiding for Efficient Light‐Transport Simulation

AbstractWe present a robust, unbiased technique for intelligent light‐path construction in path‐tracing algorithms. Inspired by existing path‐guiding algorithms, our method learns an approximate representation of the scene's spatio‐directional radiance field in an unbiased and iterative manner. To that end, we propose an adaptive spatio‐directional hybrid data structure, referred to as SD‐tree, for storing and sampling incident radiance. The SD‐tree consists of an upper part—a binary tree that partitions the 3D spatial domain of the light field—and a lower part—a quadtree that partitions the 2D directional domain. We further present a principled way to automatically budget training and rendering computations to minimize the variance of the final image. Our method does not require tuning hyperparameters, although we allow limiting the memory footprint of the SD‐tree. The aforementioned properties, its ease of implementation, and its stable performance make our method compatible with production environments. We demonstrate the merits of our method on scenes with difficult visibility, detailed geometry, and complex specular‐glossy light transport, achieving better performance than previous state‐of‐the‐art algorithms.

IEEE Access Special Section Editorial: Intelligent Sensing On Mobile and Social Media Analytics

The proliferation of mobile devices has led to an increasingly interconnected era, and a recent trend is the migration of multimedia search and analysis from the desktop to the cloud. This is, perhaps, partly due to the ease of capturing images and videos on a mobile device and associating them with social and contextual metadata such as comments and GPS tags. Such a hybrid data structure can benefit a wide variety of potential (mobile) multimedia applications, such as location recognition, landmark search, augmented reality, and commercial recommendations. One intrinsic potential solution is to explore large-scale social multimedia to assist and facilitate location related applications on the go and in real-time. With the combination of social and mobile cues, several problems that are previously difficult to be tackled in multimedia content analysis have also become more tractable. Current research in Intelligent Sensing is addressing the following: