Compact Data Structure Research Articles

Photon mapping to accelerate daylight simulation with high-resolution, data-driven fenestration models

Data-driven modelling provides a general means to represent optically complex fenestration in daylight simulation by its Bidirectional Scattering Distribution Function (BSDF). Radiance employs the tensor tree as a compact data structure to store the BSDF at high directional resolution. The application of such models under sunny sky conditions is, however, computationally demanding, since the density of stochastic backward samples must match the BSDF resolution. The bidirectional Photon Map is proposed to rapidly forward-sample the BSDF, starting from the known sun direction. Its exemplary application shows a potential speed-up of ⩾ 98% when compared to backward ray-tracing.

Using Predictive and Differential Methods with K2-Raster Compact Data Structure for Hyperspectral Image Lossless Compression

This paper proposes a lossless coder for real-time processing and compression of hyperspectral images. After applying either a predictor or a differential encoder to reduce the bit rate of an image by exploiting the close similarity in pixels between neighboring bands, it uses a compact data structure called k 2 -raster to further reduce the bit rate. The advantage of using such a data structure is its compactness, with a size that is comparable to that produced by some classical compression algorithms and yet still providing direct access to its content for query without any need for full decompression. Experiments show that using k 2 -raster alone already achieves much lower rates (up to 55% reduction), and with preprocessing, the rates are further reduced up to 64%. Finally, we provide experimental results that show that the predictor is able to produce higher rates reduction than differential encoding.

SPATIAL ADJACENCY ANALYSIS OF CITYGML BUILDINGS VIA 3D TOPOLOGICAL DATA STRUCTURE

Abstract. Adjacencies between objects provides the most basic connectivity information of objects. This connectivity information provides support for more complex 3D spatial analysis such as 3D navigation, nearest neighbour and others. In 3D models, the connectivity information is maintained by building a comprehensive 3D topology. As the international standard for 3D city models, CityGML employs a simple XML links mechanism that references related entities to each other as a means of maintaining topological information. This method fulfils the purpose of relating connected entities but, it does not describe how the entities are related or in other words its adjacencies. In this study, a 3D topological data structure was utilised to preserve topological primitives and maintain connectivity information for CityGML datasets of buildings in LoD2. The adjacencies tested in this study were based on the topological links maintained by the Compact Abstract Cell Complexes 3D topological data structure. Four types of adjacencies were tested which are Point-to-Line, Line-to-Surface, Surface-to-Surface and Volume-to-Volume adjacency. As a result, all adjacencies were able to be executed for both datasets which consisted of two connected buildings and disjointed buildings. It was found that the ability of the 3D topological data structure to preserve topological primitives and build topological links supported the maintenance of connectivity information between buildings. The maintenance of connectivity information was also not limited to objects of the same dimension and could extend to connectivity between building elements in different dimensions.

The negative skycube

Multidimensional data analysis has attracted a lot of research efforts during past years. One of the aspects that has been addressed so far is that to allow users to analyze their data from different perspectives, each of which corresponds to a selected subset of dimensions. To optimize these analysis queries, precomputation, and materialization, are among most studied solutions. In the context of skyline analysis, the skycube structure has been proposed as an optimization structure to allow users to ask for the non dominated records with respect to every selected dimensions set. More precisely, given a set of dimensions D={D1,…,Dd} and a relation T(id,D), the Skycube of T is the set of all skylines obtained by considering each of the subsets of D (subspaces). To make the Skycube practically useful, two lines of research have been pursued so far: the first one aims to propose efficient algorithms for computing it. Note that the number of these skylines is exponential w.r.t. |D|. Hence, both execution time and storage space make these solutions struggling with even moderately large datasets, say |D| larger than 10 and the number of tuples greater than 106 . This motivated the second line of researches which propose Skycube summarization techniques to reduce both time and space consumption. Both lines of research, store the whole or a summary of the following information: “for every tuple t, keep track of the dimensions subsets X (subspaces) where t belongs to the respective skyline”. In this paper, we consider the complementary statement, i.e., “for every tuple t, we store a compact data structure encoding the subspaces X with respect to which, tis dominated”. This is what we call the negative skycube. Despite the apparent equivalence between the two statements (dominated vs not dominated), our analysis and extensive experiments show that these two points of view do not lead to the same behavior of the related algorithms. More specifically, our proposal shows that: (i) the negative summary can be obtained much faster than state of the art techniques for positive summaries, (ii) in general, it consumes less memory space, (iii) skyline queries evaluation using this summary is much faster, (iv) the positive Skycube can be obtained more rapidly than state of the art algorithms especially designed for this purpose, and (v) it is highly effective with respect to insertions and deletions.

Sequence tube maps: making graph genomes intuitive to commuters.

MotivationCompared to traditional haploid reference genomes, graph genomes are an efficient and compact data structure for storing multiple genomic sequences, for storing polymorphisms or for mapping sequencing reads with greater sensitivity. Further, graphs are well-studied computer science objects that can be efficiently analyzed. However, their adoption in genomic research is slow, in part because of the cognitive difficulty in interpreting graphs.ResultsWe present an intuitive graphical representation for graph genomes that re-uses well-honed techniques developed to display public transport networks, and demonstrate it as a web tool.Availability and implementation Code: https://github.com/vgteam/sequenceTubeMap.Demonstration https://vgteam.github.io/sequenceTubeMap/.Supplementary information Supplementary data are available at Bioinformatics online.

Exact Representations and Geometric Queries for Lattice Structures with Quador Beams

Objects with architected internal structure often consist of a lattice of beams that are interconnected at nodes. Compared to the conventional cylindrical and conical beams, quador-beams are bounded by quadrics-of-revolution. We decompose the lattice into an assembly of hubs, wherein a hub is the union of a node (ball) with the half-beams incident upon it. In this paper, we propose (1) a hybrid representation of a quador-hub (hub with quador half-beams) consisting of exact CSG + CST + Boundary models, (2) a compact data structure and associated operators to work with the hybrid representation, and (3) algorithms to perform fundamental geometric queries on the hub. We provide the operators to query the half-beams of the hub, to inquire the trimming planes of a surface, or to traverse the faces, edges and vertices of the hub. We suggest efficient algorithms, that take advantage of the hybrid representation, for geometric queries such as point membership classification, planar slicing, surface meshing, and area/volume computation. We outline a possible approach for computing exact answers to all tests that are required for computing the exact topology of the proposed representation of the hub. Finally, we show that the simplified geometry of quador hubs leads to smaller data sizes and improved performance in a commercial modeler.

HUOPM: High-Utility Occupancy Pattern Mining.

Mining useful patterns from varied types of databases is an important research topic, which has many real-life applications. Most studies have considered the frequency as sole interestingness measure to identify high-quality patterns. However, each object is different in nature. The relative importance of objects is not equal, in terms of criteria, such as the utility, risk, or interest. Besides, another limitation of frequent patterns is that they generally have a low occupancy, that is, they often represent small sets of items in transactions containing many items and, thus, may not be truly representative of these transactions. To extract high-quality patterns in real-life applications, this paper extends the occupancy measure to also assess the utility of patterns in transaction databases. We propose an efficient algorithm named high-utility occupancy pattern mining (HUOPM). It considers user preferences in terms of frequency, utility, and occupancy. A novel frequency-utility tree and two compact data structures, called the utility-occupancy list and frequency-utility table, are designed to provide global and partial downward closure properties for pruning the search space. The proposed method can efficiently discover the complete set of high-quality patterns without candidate generation. Extensive experiments have been conducted on several datasets to evaluate the effectiveness and efficiency of the proposed algorithm. Results show that the derived patterns are intelligible, reasonable, and acceptable, and that HUOPM with its pruning strategies outperforms the state-of-the-art algorithm, in terms of runtime and search space, respectively.

Energy Consumption in Compact Integer Vectors: A Study Case

In the field of algorithms and data structures analysis and design, most of the researchers focus only on the space/time trade-off, and little attention has been paid to energy consumption. Moreover, most of the efforts in the field of Green Computing have been devoted to hardware-related issues, being green software in its infancy. Optimizing the usage of computing resources, minimizing power consumption or increasing battery life are some of the goals of this field of research. As an attempt to address the most recent sustainability challenges, we must incorporate the energy consumption as a first-class constraint when designing new compact data structures. Thus, as a preliminary work to reach that goal, we first need to understand the factors that impact on the energy consumption and their relation with compression. In this work, we study the energy consumption required by several integer vector representations. We execute typical operations over datasets of different nature. We can see that, as commonly believed, energy consumption is highly related to the time required by the process, but not always. We analyze other parameters, such as number of instructions, number of CPU cycles, memory loads, among others.

Accelerating Sequence Alignments Based on FM-Index Using the Intel KNL Processor.

FM-index is a compact data structure suitable for fast matches of short reads to large reference genomes. The matching algorithm using this index exhibits irregular memory access patterns that cause frequent cache misses, resulting in a memory bound problem. This paper analyzes different FM-index versions presented in the literature, focusing on those computing aspects related to the data access. As a result of the analysis, we propose a new organization of FM-index that minimizes the demand for memory bandwidth, allowing a great improvement of performance on processors with high-bandwidth memory, such as the second-generation Intel Xeon Phi (Knights Landing, or KNL), integrating ultra high-bandwidth stacked memory technology. As the roofline model shows, our implementation reaches 95 percent of the peak random access bandwidth limit when executed on the KNL and almost all of the available bandwidth when executed on other Intel Xeon architectures with conventional DDR memory. In addition, the obtained throughput in KNL is much higher than the results reported for GPUs in the literature.

The Book Review Column

In this column we review these three books: 1. Compact Data Structures - A Practical Approach, by Gonzalo Navarro. An unusual approach focussing on data structures that use as little space as is theoretically possible, while maintaining practicality. Review by Laszlo Kozma. 2. Power Up: Unlocking the Hidden Mathematics in Video Games, by Matthew Lane. A revealing look at the mathematics, physics, and computer science that underlies (good) video games, with special attention to their potential valuable role in education. Review by S. V. Nagaraj. 3. Probability and Computing: Randomization and Probabilistic Techniques in Algorithms and Data Analysis (Second Edition), by Michael Mitzenmacher and Eli Upfal. The second edition2 of a distinguished text on a rich area that is central to computer and data science. Review by Aravind Srinivasan. Many thanks to these and all reviewers over the years who have graciously volunteered this valuable service to the community! Interested in reviewing? I hope so. Remember the perks: Learning (more?) about a field that interests you, a review on these pages, and a free book! The books listed on the next two pages are available, but the list is not exhaustive: I'm open to suggestions.

Review of Compact Data Structures - a practical approach by Gonzalo Navarro

The book studies fundamental data structures and representations of combinatorial objects, including arrays, bitvectors, permutations, trees, graphs, etc. There are, of course, a great number of existing textbooks and lecture notes about data structures. The perspective of this book is, however, different from the usual presentations. In some sense, this book starts exactly where others leave off saying "we omit the low-level details of representation", and concerns itself mostly with the low-level representation. Specifically, it looks at how to organize and store data structures using as little space as possible while supporting efficient queries and operations. For the considered structures, the book explores the trade-off between reduced space and speed of operation. The analyses of the presented solutions go beyond the usual asymptotics, and also consider exact constant factors (particularly for the leading term of the space complexity); useful practical considerations are also discussed.

Path queries on functions

Let f:[1..n]→[1..n] be a function, and ℓ:[1..n]→[1..σ] indicate a label assigned to each element of the domain. We design several compact data structures that answer various kinds of summary queries on the labels of paths in f. For example, we can find either the minimum label in fk(i) for a given i and any k≥0 in a given range [k1..k2], or the minimum label in f−k(i) for a given i and k>0, using nlg⁡n+nlg⁡σ+o(nlg⁡n) bits and time O(α(n)), the inverse Ackermann function. Within similar space we can count, in time O(lg⁡n/lg⁡lg⁡n), the number of labels within a range, and report each element with such labels in O(lg⁡n/lg⁡lg⁡n) additional time. Several other tradeoffs and possible queries are considered, such as selection, top-r queries and τ-majorities. Finally, we consider queries that allow us navigate on the graph of the function, such as the nearest common successor of two elements, or the nearest successor or predecessor of an element within a range of labels.

Compact and efficient representation of general graph databases

In this paper, we propose a compact data structure to store labeled attributed graphs based on the k2-tree, which is a very compact data structure designed to represent a simple directed graph. The idea we propose can be seen as an extension of the k2-tree to support property graphs. In addition to the static approach, we also propose a dynamic version of the storage representation, which allows exible schemas and insertion or deletion of data. We provide an implementation of a basic set of operations, which can be combined to form complex queries over these graphs with attributes. We evaluate the performance of our proposal with existing graph database systems and prove that our compact attributed graph representation obtains also competitive time results.

A Compact Face-Based Topological Data Structure for Triangle Mesh Representation

A Compact Face-Based Topological Data Structure for Triangle Mesh Representation

Compact Data Structures to Represent and Query Data Warehouses into Main Memory

In this paper we propose the use of compact data structures to represent and process Data Warehouses (DWs) into main memory. Compact data structures are data structures that allow compacting the data without losing the capacity of querying the data in their compact form. A DW is a data repository to store historical data for decision support, and consists of dimensions and facts. The dimensions are abstract concepts that groups data with a similar meaning, usually, they are modelled as hierarchies of levels, which contain elements. The facts are quantitative data associated to dimensions. A data cube is a way to retrieve facts at different levels of granularity, which is achieved by navigation on dimensions hierarchies. Since a DW can store terabytes of data, the efficient processing of data cubes is key in OLAP (On-line Analytical Processing). We show that by using a compact representation of DWs we can improve the use of space in main memory, and achieve better performance for query processing. In this paper we extend a previous work to process aggregate queries with aggregate functions MAX, MIN, COUNT and AVG.

An Efficient Representation for Filtrations of Simplicial Complexes

A filtration over a simplicial complex K is an ordering of the simplices of K such that all prefixes in the ordering are subcomplexes of K . Filtrations are at the core of Persistent Homology, a major tool in Topological Data Analysis. To represent the filtration of a simplicial complex, the entire filtration can be appended to any data structure that explicitly stores all the simplices of the complex such as the Hasse diagram or the recently introduced Simplex Tree [Algorithmica’14]. However, with the popularity of various computational methods that need to handle simplicial complexes, and with the rapidly increasing size of the complexes, the task of finding a compact data structure that can still support efficient queries is of great interest. This direction has been recently pursued for the case of maintaining simplicial complexes. For instance, Boissonnat et al. [Algorithmica’17] considered storing the simplices that are maximal with respect to inclusion and Attali et al. [IJCGA’12] considered storing the simplices that block the expansion of the complex. Nevertheless, so far there has been no data structure that compactly stores the filtration of a simplicial complex, while also allowing the efficient implementation of basic operations on the complex. In this article, we propose a new data structure called the Critical Simplex Diagram (CSD), which is a variant of the Simplex Array List [Algorithmica’17]. Our data structure allows one to store in a compact way the filtration of a simplicial complex and allows for the efficient implementation of a large range of basic operations. Moreover, we prove that our data structure is essentially optimal with respect to the requisite storage space. Finally, we show that the CSD representation admits fast construction algorithms for Flag complexes and relaxed Delaunay complexes.

Accelerated partial decoding in wavelet trees

A Wavelet Tree is a compact data structure which is used in order to perform various well defined operations directly on the compressed form of a file. As random access is one of these operations, the underlying file is not needed anymore, and is often discarded because it can be restored, when necessary, by repeated accesses. This paper concentrates on cases in which partial decoding of a contiguous portion of the file, or even its full decoding, is still needed. We show how to accelerate the decoding relative to repeatedly performing random accesses on the consecutive indices. Experiments on partial and full decoding support the effectiveness of our approach, and present an improvement of about 50% of the run-time for full decoding, and about 30% or more for partial decoding of large enough ranges.

GPU-accelerated generation and rendering of multi-level voxel representations of solid models

Solid models traditionally use boundary-representation (B-rep) to define and model their geometry. However, performing modeling operations such as Boolean operations or computing point membership classification with B-rep is computationally intensive, since B-reps do not have volumetric information. Voxelized representations, on the other hand, can be extended to include volumetric information of solid models. However, in order to use voxelized representations for solid modeling, efficient methods for voxelizing a B-rep solid model needs to be developed. In this paper, we present GPU-accelerated methods for creating and rendering a multi-level voxelization of a solid model that can be used for modeling operations. We describe two GPU-accelerated algorithms; one for creating a multi-level voxelization given a B-rep of a solid model and another for ray casting to render the multi-level voxelization of the solid model. We describe compact and flat data structures that can be used to store the multi-level voxelization data and can be efficiently retrieved in parallel by GPU-algorithms for rendering and modeling operations. The GPU-accelerated multi-level voxelization method can generate models with an effective voxel count of up to 8 billion voxels. In addition, the GPU voxelization algorithm is more than 40x faster than the CPU implementation in generating the voxelization. Finally, we outline a few applications for the voxel representation, which include fast point-membership classification, volume computation, and collision detection.

Using Compressed Suffix-Arrays for a compact representation of temporal-graphs

Temporal graphs represent binary relationships that change along time. They can model the dynamism of, for example, social and communication networks. Temporal graphs are defined as sets of contacts that are edges tagged with the temporal intervals when they are active. This work explores the use of the Compressed Suffix Array (CSA), a well-known compact and self-indexed data structure in the area of text indexing, to represent large temporal graphs. The new structure, called Temporal Graph CSA (TGCSA), is experimentally compared with the most competitive compact data structures in the state-of-the-art, namely, EdgeLog and CET. The experimental results show that TGCSA obtains a good space-time trade-off. It uses a reasonable space and is efficient for solving complex temporal queries. Furthermore, TGCSA has wider expressive capabilities than EdgeLog and CET, because it is able to represent temporal graphs where contacts on an edge can temporally overlap.

Constraint-Based Inference in Probabilistic Logic Programs

AbstractProbabilistic Logic Programs (PLPs) generalize traditional logic programs and allow the encoding of models combining logical structure and uncertainty. In PLP, inference is performed by summarizing the possible worlds which entail the query in a suitable data structure, and using this data structure to compute the answer probability. Systems such as ProbLog, PITA, etc., use propositional data structures like explanation graphs, BDDs, SDDs, etc., to represent the possible worlds. While this approach saves inference time due to substructure sharing, there are a number of problems where a more compact data structure is possible. We propose a data structure called Ordered Symbolic Derivation Diagram (OSDD) which captures the possible worlds by means of constraint formulas. We describe a program transformation technique to construct OSDDs via query evaluation, and give procedures to perform exact and approximate inference over OSDDs. Our approach has two key properties. Firstly, the exact inference procedure is a generalization of traditional inference, and results in speedup over the latter in certain settings. Secondly, the approximate technique is a generalization of likelihood weighting in Bayesian Networks, and allows us to perform sampling-based inference with lower rejection rate and variance. We evaluate the effectiveness of the proposed techniques through experiments on several problems.