Space-efficient Representation Research Articles

LinSets.zip: Compressing Linear Set Diagrams.

Linear diagrams are used to visualize set systems by depicting set memberships as horizontal line segments in a matrix, where each set is represented as a row and each element as a column. Each such line segment of a set is shown in a contiguous horizontal range of cells of the matrix indicating that the corresponding elements in the columns belong to the set. As each set occupies its own row in the matrix, the total height of the resulting visualization is as large as the number of sets in the instance. Such a linear diagram can be visually sparse and intersecting sets containing the same element might be represented by distant rows. To alleviate such undesirable effects, we present LinSets.zip, a new approach that achieves a more space-efficient representation of linear diagrams. First, we minimize the total number of gaps in the horizontal segments by reordering columns, a criterion that has been shown to increase readability in linear diagrams. The main difference of LinSets.zip to linear diagrams is that multiple non-intersecting sets can be positioned in the same row of the matrix. Furthermore, we present several different rendering variations for a matrix-based representation that utilize the proposed row compression. We implemented the different steps of our approach in a visualization pipeline using integer-linear programming, and suitable heuristics aiming at sufficiently fast computations in practice. We conducted both a quantitative evaluation and a small-scale user experiment to compare the effects of compressing linear diagrams.

Symbolic Loop Compilation for Tightly Coupled Processor Arrays

Tightly Coupled Processor Arrays (TCPAs), a class of massively parallel loop accelerators, allow applications to offload computationally expensive loops for improved performance and energy efficiency. To achieve these two goals, executing a loop on a TCPA requires an efficient generation of specific programs as well as other configuration data for each distinct combination of loop bounds and number of available processing elements (PEs). Since both these parameters are generally unknown at compile time—the number of available PEs due to dynamic resource management, and the loop bounds, because they depend on the problem size—both the programs and configuration data must be generated at runtime. However, pure just-in-time compilation is impractical, because mapping a loop program onto a TCPA entails solving multiple NP-complete problems. As a solution, this article proposes a unique mixed static/dynamic approach called symbolic loop compilation. It is shown that at compile time, the NP-complete problems (modulo scheduling, register allocation, and routing) can still be solved to optimality in a symbolic way resulting in a so-called symbolic configuration , a space-efficient intermediate representation parameterized in the loop bounds and number of PEs. This phase is called symbolic mapping . At runtime, for each requested accelerated execution of a loop program with given loop bounds and known number of available PEs, a concrete configuration , including PE programs and configuration data for all other components, is generated from the symbolic configuration according to these parameter values. This phase is called instantiation . We describe both phases in detail and show that instantiation runs in polynomial time with its most complex step, program instantiation, not directly depending on the number of PEs and thus scaling to arbitrary sizes of TCPAs. To validate the efficiency of this mixed static/dynamic compilation approach, we apply symbolic loop compilation to a set of real-world loop programs from several domains, measuring both compilation time and space requirements. Our experiments confirm that a symbolic configuration is a space-efficient representation suited for systems with little memory—in many cases, a symbolic configuration is smaller than even a single concrete configuration instantiated from it—and that the times for the runtime phase of program instantiation and configuration loading are negligible and moreover independent of the size of the available processor array. To give an example, instantiating a configuration for a matrix-matrix multiplication benchmark takes equally long for 4× 4 and 32× 32 PEs.

Space-efficient representations of raster time series

Raster time series, a.k.a. temporal rasters, are collections of rasters covering the same region at consecutive timestamps. These data have been used in many different applications ranging from weather forecast systems to monitoring of forest degradation or soil contamination. Many different sensors are generating this type of data, which makes such analyses possible, but also challenges the technological capacity to store and retrieve the data. In this work, we propose a space-efficient representation of raster time series that is based on Compact Data Structures (CDS). Our method uses a strategy of snapshots and logs to represent the data, in which both components are represented using CDS. We study two variants of this strategy, one with regular sampling and another one based on a heuristic that determines at which timestamps should the snapshots be created to reduce the space redundancy. We perform a comprehensive experimental evaluation using real datasets. The results show that the proposed strategy is competitive in space with alternatives based on pure data compression, while providing much more efficient query times for different types of queries.

Pebbles: Leveraging Sketches for Processing Voluminous, High Velocity Data Streams

Voluminous, time-series data streams originating in continuous sensing environments pose data ingestion and processing challenges. We present a holistic methodology centered around data sketching to address both challenges. We introduce an order-preserving sketching algorithm that we have designed for space-efficient representation of multi-feature streams with native support for stream processing related operations. Observational streams are preprocessed at the edges of the network generating sketched streams to reduce data transfer costs and energy consumption. Ingested sketched streams are then processed using sketch-aware extensions to existing stream processing APIs delivering improved performance. Our benchmarks with real-world datasets show up to a ~8× reduction in data volumes transferred and a ~27× improvement in throughput.

SeqRepo: A system for managing local collections of biological sequences.

MotivationAccess to biological sequence data, such as genome, transcript, or protein sequence, is at the core of many bioinformatics analysis workflows. The National Center for Biotechnology Information (NCBI), Ensembl, and other sequence database maintainers provide methods to access sequences through network connections. For many users, the convenience and currency of remotely managed data are compelling, and the network latency is non-consequential. However, for high-throughput and clinical applications, local sequence collections are essential for performance, stability, privacy, and reproducibility.ResultsHere we describe SeqRepo, a novel system for building a local, high-performance, non-redundant collection of biological sequences. SeqRepo enables clients to use primary database identifiers and several digests to identify sequences and sequence alises. SeqRepo provides a native Python interface and a REST interface, which can run locally and enables access from other programming languages. SeqRepo also provides an alternative REST interface based on the GA4GH refget protocol.SeqRepo provides fast random access to sequence slices. We provide results that demonstrate that a local SeqRepo sequence collection yields significant performance benefits of up to 1300-fold over remote sequence collections. In our use case for a variant validation and normalization pipeline, SeqRepo improved throughput 50-fold relative to use with remote sequences. SeqRepo may be used with any species or sequence type. Regular snapshots of Human sequence collections are available.It is often convenient or necessary to use a computed digest as a sequence identifier. For example, a digest-based identifier may be used to refer to proprietary reference genomes or segments of a graph genome, for which conventional identifiers will not be available. Here we also introduce a convention for the application of the SHA-512 hashing algorithm with Base64 encoding to generate URL-safe identifiers. This convention, sha512t24u, combines a fast digest mechanism with a space-efficient representation that can be used for any object. Our report includes an analysis of timing and collision probabilities for sha512t24u. SeqRepo enables clients to use sha512t24u as identifiers, thereby seamlessly integrating public and private sequence sets.AvailabilitySeqRepo is released under the Apache License 2.0 and is available on github and PyPi. Docker images and database snapshots are also available. See https://github.com/biocommons/biocommons.seqrepo.

Space efficient representations of finite groups

The Cayley table representation of a group uses O(n2) words for a group of order n and answers multiplication queries in time O(1). It is interesting to ask if there is a o(n2) space representation of groups that still has O(1) query-time. We show that for any δ, 1/log⁡n≤δ≤1, there is an O(n1+δ/δ) space representation for groups of order n with O(1/δ) query-time. We also show that for Dedekind groups, simple groups and several group classes defined in terms of semidirect product, there are O(n) space representations with logarithmic query-time where n is the size of the group. Farzan and Munro (ISSAC'06) defined a model for group representation and gave a succinct data structure for abelian groups with constant query-time. They asked if their result can be extended to categorically larger group classes. We construct data structures in their model for Hamiltonian groups and some other classes of groups with constant query-time.

Dynamic construction of pan-genome subgraphs

AbstractMarcus et al. (Bioinformatics 2014) proposed to use a compressed de Bruijn graph as a description of a pan-genome, comprising the genomes of many individuals/strains of the same or closely related species. Subsequent work improved the construction of the compressed de Bruijn graph in terms of run-time and memory consumption. According to the Computational Pan-Genomics Consortium (Briefings in Bioinformatics 2016), a pan-genome data structure should support the following functionality: “All information within a data structure should be easily accessible for human eyes by visualization support on different scales.” However, a pan-genome graph can have thousands to millions of nodes and such an amount of information is certainly not easily accessible for human eyes. Thus, the possibility to construct pangenome subgraphs on demand would be quite valuable. In this article, we use the space-efficient representation of the compressed de Bruijn graph devised by Beller and Ohle-busch (Algorithms for Molecular Biology 2016) to construct pan-genome subgraphs on the fly. The user can specify a region in one of the genomes and the software tool will build a subgraph that contains the path corresponding to that region and all paths that are in the neighborhood of that path. The size of the neighborhood can be controlled by the user.

Efficient Representation of Geometric Tree Models with Level-of-Detail Using Compressed 3D Chain Code.

In the paper, we present a method for space-efficient representation of geometric tree models, which are provided as skeletons with radii attached to individual branch segments. The proposed approach uses a new differential 3D chain code to encode orientation changes of consecutive branch segments, which allows optimizing chain code generation for increased compressibility while maintaining control over the model reconstruction error. The presented method is the first to encode the complete branching geometry including the branch radii and provides level-of-detail construction directly from the chain code. It is demonstrated that by using interpolative encoding of the resulting tree descriptors and radii sequences, the storage requirements for geometric description of a mixed all-aged forest can be reduced to less than 15 percent of its raw size while preserving the structural fidelity of tree models.

Network-based characterization of drug-protein interaction signatures with a space-efficient approach

BackgroundCharacterization of drug-protein interaction networks with biological features has recently become challenging in recent pharmaceutical science toward a better understanding of polypharmacology.ResultsWe present a novel method for systematic analyses of the underlying features characteristic of drug-protein interaction networks, which we call “drug-protein interaction signatures” from the integration of large-scale heterogeneous data of drugs and proteins. We develop a new efficient algorithm for extracting informative drug-protein interaction signatures from the integration of large-scale heterogeneous data of drugs and proteins, which is made possible by space-efficient representations for fingerprints of drug-protein pairs and sparsity-induced classifiers.ConclusionsOur method infers a set of drug-protein interaction signatures consisting of the associations between drug chemical substructures, adverse drug reactions, protein domains, biological pathways, and pathway modules. We argue the these signatures are biologically meaningful and useful for predicting unknown drug-protein interactions and are expected to contribute to rational drug design.

A novel verifiable and unconditionally secure (m, t, n)-threshold multi-secret sharing scheme using overdetermined systems of linear equations over finite Galois fields

Threshold multi-secrets sharing schemes allow sharing a set of m secrets among n participants, while secrets can be revealed only if t or more participants collude. Although many multi-secret sharing schemes have been proposed, several improvements remain essential in order to cope with actual effectiveness and security requirements, including computational performances and compliance for large-scale data. In this paper, we present a novel multi-secrets (m, t, n)-threshold scheme using overdetermined systems of linear equations defined over finite Galois fields. The scheme provides unconditional security, linear sharing/reconstructing complexities and holds secure verifiability and t-consistence. By considering both secrets and shares as elements over finite Galois fields GF(2r), optimal and space-efficient representation is ensured compared to recent sharing schemes. In addition, the scheme provides dynamic secrets sharing, forgery/cheating detection and robustness against common attacks, while lower computational overhead is required.

A novel verifiable and unconditionally secure (m, t, n)-threshold multi-secret sharing scheme using overdetermined systems of linear equations over finite Galois fields

Threshold multi-secrets sharing schemes allow sharing a set of m secrets among n participants, while secrets can be revealed only if t or more participants collude. Although many multi-secret sharing schemes have been proposed, several improvements remain essential in order to cope with actual effectiveness and security requirements, including computational performances and compliance for large-scale data. In this paper, we present a novel multi-secrets (m, t, n)-threshold scheme using overdetermined systems of linear equations defined over finite Galois fields. The scheme provides unconditional security, linear sharing/reconstructing complexities and holds secure verifiability and t-consistence. By considering both secrets and shares as elements over finite Galois fields GF(2r), optimal and space-efficient representation is ensured compared to recent sharing schemes. In addition, the scheme provides dynamic secrets sharing, forgery/cheating detection and robustness against common attacks, while lower computational overhead is required.

Symbolic Synthesis of Timed Models with Strict 2-Phase Fault Recovery

In this article, we focus on efficient synthesis of fault-tolerant timed models from their fault-intolerant version. Although the complexity of the synthesis problem is known to be polynomial time in the size of the time-abstract bisimulation of the input model, the state of the art currently lacks synthesis algorithms that can be efficiently implemented. This is in part due to the fact that synthesis is in general a challenging problem and its complexity is significantly magnified in the context of timed systems. We propose an algorithm that takes as input a timed automaton, a set of fault actions, and a set of safety and bounded-time response properties, and utilizes a space-efficient symbolic representation of the timed automaton (called zone graph) to synthesize a fault-tolerant timed automaton as output. The output automaton satisfies strict phased recovery, where it is guaranteed that the output model behaves similarly to the input model in the absence of faults and in the presence of faults, fault recovery is achieved in two phases, each satisfying certain safety and timing constraints.

Managing variants of a personalized product

A product variant table is a table that lists legal combinations of product features. Variant tables can be used to constrain the variability offered for a personalized product. The concept of such a table is easy to understand. Hence, variant tables are natural to use when ensuring the completeness and correctness of a quote/order for a customizable product. They are also used to filter out inadmissible choices for features in an interactive specification (configuration) process. Variant tables can be maintained as relational (database) tables, using spreadsheets, or in proprietary ways offered by the product modeling environment. Variant tables can become quite large. A way of compressing them is then sought that supports a space-efficient representation and a time-efficient evaluation. The motivation of this work is to develop a simple approach to compress/compile a variant table into an easy to read, but possibly hard to write form that can be deployed in a business setting at acceptable cost and risk in a similar manner as a database. The main result is a simple compression and evaluation scheme for an individual variant table called a Variant Decomposition Diagram (VDD). A VDD supports efficient consistency checks, the filtering of inadmissible features, and iteration over the table. A simple static heuristic for decomposition order is proposed that suggests itself from a “column oriented viewpoint”. This heuristic is not always optimal, but it has the advantage of allowing fast compilation of a variant table into a VDD. Compression results for a publicly available model of a Renault Megane are given. With the proposed heuristic the VDD is a specialization of a Zero-suppressed (binary) Decision Diagram (ZDD) (Knuth 2011) and also maps to a Multi-valued Decision Diagram (MDD) (Andersen et al. 2007; Berndt et al. 2012).

Compact representations of automata for regular expression matching

Glushkov's nondeterministic finite automaton leads to efficient regular expression matching. But it is memory greedy for long regular expressions. We develop space efficient representations for the deterministic finite automata obtained from Glushkov automata. The approach reduces the space of the DFA from O(m2m) bits to O(m2k) bits where k is the number of strings in the regular expression, m is the number of characters excluding operators. The average space usage is O(m(1+1/σ)k) bits where σ is the size of the alphabet. The state transition function runs in constant time when the length of a word is not less than m. Experiments show that our method is as efficient as the previous method and uses less space.

A representation of a compressed de Bruijn graph for pan-genome analysis that enables search.

BackgroundRecently, Marcus et al. (Bioinformatics 30:3476–83, 2014) proposed to use a compressed de Bruijn graph to describe the relationship between the genomes of many individuals/strains of the same or closely related species. They devised an O(nlog g) time algorithm called splitMEM that constructs this graph directly (i.e., without using the uncompressed de Bruijn graph) based on a suffix tree, where n is the total length of the genomes and g is the length of the longest genome. Baier et al. (Bioinformatics 32:497–504, 2016) improved their result.ResultsIn this paper, we propose a new space-efficient representation of the compressed de Bruijn graph that adds the possibility to search for a pattern (e.g. an allele—a variant form of a gene) within the pan-genome. The ability to search within the pan-genome graph is of utmost importance and is a design goal of pan-genome data structures.

Space-efficient representation of truncated suffix trees, with applications to Markov order estimation

Suffix trees (ST) are useful in many text processing applications, for example, to determine the number of occurrences of patterns of arbitrary length in an input string x. If the length n, of x, is large, the memory required to represent the ST may become a practical performance bottleneck. This problem can be alleviated, in cases where a nontrivial upper bound is known on the lengths of the patterns of interest, by using a truncated ST (TST). However, conventional TST implementations still require Ω(n) bits of memory, since they store x. We describe a new TST representation that avoids this limitation by storing all the information necessary to reconstruct the TST edge labels in a string y that is often much shorter than x. We apply TSTs to the implementation of Markov order estimators, where an upper bound kn on the estimated order can be derived or it is imposed (for consistency, for example). The new representation allows for estimator implementations with sublinear space complexity in some cases of interest. In other cases we show, experimentally, that even when the new representation does not have an asymptotic advantage, it still achieves very significant memory savings in practice.

Near neighbor searching with K nearest references

Proximity searching is the problem of retrieving, from a given database, those objects closest to a query. To avoid exhaustive searching, data structures called indexes are built on the database prior to serving queries. The curse of dimensionality is a well-known problem for indexes: in spaces with sufficiently concentrated distance histograms, no index outperforms an exhaustive scan of the database.In recent years, a number of indexes for approximate proximity searching have been proposed. These are able to cope with the curse of dimensionality in exchange for returning an answer that might be slightly different from the correct one.In this paper we show that many of those recent indexes can be understood as variants of a simple general model based on K-nearest reference signatures. A set of references is chosen from the database, and the signature of each object consists of the K references nearest to the object. At query time, the signature of the query is computed and the search examines only the objects whose signature is close enough to that of the query.Many known and novel indexes are obtained by considering different ways to determine how much detail the signature records (e.g., just the set of nearest references, or also their proximity order to the object, or also their distances to the object, and so on), how the similarity between signatures is defined, and how the parameters are tuned. In addition, we introduce a space-efficient representation for those families of indexes, making it possible to search very large databases in main memory. Small indexes are cache friendly, inducing faster queries.We perform exhaustive experiments comparing several known and new indexes that derive from our framework, evaluating their time performance, memory usage, and quality of approximation. The best indexes outperform the state of the art, offering an attractive balance between all these aspects, and turn out to be excellent choices in many scenarios. Our framework gives high flexibility to design new indexes.

SlaMEM: efficient retrieval of maximal exact matches using a sampled LCP array

Maximal exact matches, or just MEMs, are a powerful tool in the context of multiple sequence alignment and approximate string matching. The most efficient algorithms to collect them are based on compressed indexes that rely on longest common prefix array-centered data structures. However, their space-efficient representations make use of encoding techniques that are expensive from a computational point of view. With the deluge of data generated by high-throughput sequencing, new approaches need to be developed to deal with larger genomic sequences. In this work, we have developed a new longest common prefix array-sampled representation, optimized to work with the backward search method inherently used by the FM-Index. Unlike previous implementations that sacrifice running time to have smaller space, ours lead to both a fast and a space-efficient approach. This implementation was used by the new software slaMEM, developed to efficiently retrieve MEMs. The results show that the new algorithm is competitive against existing state-of-the-art approaches. The software is implemented in C and is operating system independent. The source code is freely available for download at http://github.com/fjdf/slaMEM/ under the GPLv3 license.

Space-efficient representations of rectangle datasets supporting orthogonal range querying

The increasing use of geographic search engines manifests the interest of Internet users in geo-located resources and, in general, in geographic information. This has emphasized the importance of the development of efficient indexes over large geographic databases. The most common simplification of geographic objects used for indexing purposes is a two-dimensional rectangle. Furthermore, one of the primitive operations that must be supported by every geographic index structure is the orthogonal range query, which retrieves all the geographic objects that have at least one point in common with a rectangular query region. In this work, we study several space-efficient representations of rectangle datasets that can be used in the development of geographic indexes supporting orthogonal range queries.

VDVR: Verifiable Volume Visualization of Projection-Based Data

Practical volume visualization pipelines are never without compromises and errors. A delicate and often-studied component is the interpolation of off-grid samples, where aliasing can lead to misleading artifacts and blurring, potentially hiding fine details of critical importance. The verifiable visualization framework we describe aims to account for these errors directly in the volume generation stage, and we specifically target volumetric data obtained via computed tomography (CT) reconstruction. In this case the raw data are the X-ray projections obtained from the scanner and the volume data generation process is the CT algorithm. Our framework informs the CT reconstruction process of the specific filter intended for interpolation in the subsequent visualization process, and this in turn ensures an accurate interpolation there at a set tolerance. Here, we focus on fast trilinear interpolation in conjunction with an octree-type mixed resolution volume representation without T-junctions. Efficient rendering is achieved by a space-efficient and locality-optimized representation, which can straightforwardly exploit fast fixed-function pipelines on GPUs.