Memory-efficient Representation Research Articles

Pan-genome de Bruijn graph using the bidirectional FM-index

BackgroundPan-genome graphs are gaining importance in the field of bioinformatics as data structures to represent and jointly analyze multiple genomes. Compacted de Bruijn graphs are inherently suited for this purpose, as their graph topology naturally reveals similarity and divergence within the pan-genome. Most state-of-the-art pan-genome graphs are represented explicitly in terms of nodes and edges. Recently, an alternative, implicit graph representation was proposed that builds directly upon the unidirectional FM-index. As such, a memory-efficient graph data structure is obtained that inherits the FM-index’ backward search functionality. However, this representation suffers from a number of shortcomings in terms of functionality and algorithmic performance.ResultsWe present a data structure for a pan-genome, compacted de Bruijn graph that aims to address these shortcomings. It is built on the bidirectional FM-index, extending the ability of its unidirectional counterpart to navigate and search the graph in both directions. All basic graph navigation steps can be performed in constant time. Based on these features, we implement subgraph visualization as well as lossless approximate pattern matching to the graph using search schemes. We demonstrate that we can retrieve all occurrences corresponding to a read within a certain edit distance in a very efficient manner. Through a case study, we show the potential of exploiting the information embedded in the graph’s topology through visualization and sequence alignment.ConclusionsWe propose a memory-efficient representation of the pan-genome graph that supports subgraph visualization and lossless approximate pattern matching of reads against the graph using search schemes. The C++ source code of our software, called Nexus, is available at https://github.com/biointec/nexus under AGPL-3.0 license.

BiocMAP: a Bioconductor-friendly, GPU-accelerated pipeline for bisulfite-sequencing data

BackgroundBisulfite sequencing is a powerful tool for profiling genomic methylation, an epigenetic modification critical in the understanding of cancer, psychiatric disorders, and many other conditions. Raw data generated by whole genome bisulfite sequencing (WGBS) requires several computational steps before it is ready for statistical analysis, and particular care is required to process data in a timely and memory-efficient manner. Alignment to a reference genome is one of the most computationally demanding steps in a WGBS workflow, taking several hours or even days with commonly used WGBS-specific alignment software. This naturally motivates the creation of computational workflows that can utilize GPU-based alignment software to greatly speed up the bottleneck step. In addition, WGBS produces raw data that is large and often unwieldy; a lack of memory-efficient representation of data by existing pipelines renders WGBS impractical or impossible to many researchers.ResultsWe present BiocMAP, a Bioconductor-friendly methylation analysis pipeline consisting of two modules, to address the above concerns. The first module performs computationally-intensive read alignment using Arioc, a GPU-accelerated short-read aligner. Since GPUs are not always available on the same computing environments where traditional CPU-based analyses are convenient, the second module may be run in a GPU-free environment. This module extracts and merges DNA methylation proportions—the fractions of methylated cytosines across all cells in a sample at a given genomic site. Bioconductor-based output objects in R utilize an on-disk data representation to drastically reduce required main memory and make WGBS projects computationally feasible to more researchers.ConclusionsBiocMAP is implemented using Nextflow and available at http://research.libd.org/BiocMAP/. To enable reproducible analysis across a variety of typical computing environments, BiocMAP can be containerized with Docker or Singularity, and executed locally or with the SLURM or SGE scheduling engines. By providing Bioconductor objects, BiocMAP’s output can be integrated with powerful analytical open source software for analyzing methylation data.

Low-Complexity Lossless Coding for Memory-Efficient Representation of Event Camera Frames

A novel low-complexity lossless coding framework is proposed for efficient memory representation of event frames (EFs) suitable for hardware implementation in the low-cost event signal processing (ESP) chip. The proposed framework adapts the run-length encoding scheme and Elias coding for EF coding. A first contribution proposes a novel low-complexity codec for fast encoding of large EF sets using a binary map of accumulated spatial information and a vector of concatenated time intervals. Another contribution proposes a novel memory-efficient representation codec for random access to any group of pixels. The experimental evaluation demonstrates that the proposed codecs provide a much smaller runtime and a close performance compared with the state-of-the-art lossless image and video codecs, designed for storage optimization. To our knowledge, this letter proposes the first low-complexity lossless coding framework of EFs suitable for integration into low-cost ESP chips.

Efficient and interpretable graph network representation for angle-dependent properties applied to optical spectroscopy

Graph neural networks are attractive for learning properties of atomic structures thanks to their intuitive graph encoding of atoms and bonds. However, conventional encoding does not include angular information, which is critical for describing atomic arrangements in disordered systems. In this work, we extend the recently proposed ALIGNN (Atomistic Line Graph Neural Network) encoding, which incorporates bond angles, to also include dihedral angles (ALIGNN-d). This simple extension leads to a memory-efficient graph representation that captures the complete geometry of atomic structures. ALIGNN-d is applied to predict the infrared optical response of dynamically disordered Cu(II) aqua complexes, leveraging the intrinsic interpretability to elucidate the relative contributions of individual structural components. Bond and dihedral angles are found to be critical contributors to the fine structure of the absorption response, with distortions that represent transitions between more common geometries exhibiting the strongest absorption intensity. Future directions for further development of ALIGNN-d are discussed.

OctAttention: Octree-Based Large-Scale Contexts Model for Point Cloud Compression

In point cloud compression, sufficient contexts are significant for modeling the point cloud distribution. However, the contexts gathered by the previous voxel-based methods decrease when handling sparse point clouds. To address this problem, we propose a multiple-contexts deep learning framework called OctAttention employing the octree structure, a memory-efficient representation for point clouds. Our approach encodes octree symbol sequences in a lossless way by gathering the information of sibling and ancestor nodes. Expressly, we first represent point clouds with octree to reduce spatial redundancy, which is robust for point clouds with different resolutions. We then design a conditional entropy model with a large receptive field that models the sibling and ancestor contexts to exploit the strong dependency among the neighboring nodes and employ an attention mechanism to emphasize the correlated nodes in the context. Furthermore, we introduce a mask operation during training and testing to make a trade-off between encoding time and performance. Compared to the previous state-of-the-art works, our approach obtains a 10%-35% BD-Rate gain on the LiDAR benchmark (e.g. SemanticKITTI) and object point cloud dataset (e.g. MPEG 8i, MVUB), and saves 95% coding time compared to the voxel-based baseline. The code is available at https://github.com/zb12138/OctAttention.

A microstructurally motivated constitutive description of collagenous soft biological tissue towards the description of their non-linear and time-dependent properties

A versatile constitutive model for load-carrying soft biological tissue should incorporate salient microstructural deformation mechanisms and be able to reliably predict complex non-linear viscoelastic behavior. The advancement of treatment and rehabilitation strategies for soft tissue injuries is inextricably linked to our understanding of the underlying tissue microstructure and how this defines its macroscopic material properties. Towards this long-term objective, we present a generalized multiscale constitutive framework based on a novel description of collagen, the most mechanically significant extracellular matrix protein. The description accounts for the gradual recruitment of undulated collagen fibrils and introduces proteoglycan mediated time-dependent fibrillar sliding. Crucially, the proteoglycan deformation allows for the reduction of overstressed fibrils towards a preferential homeostatic stress. An implicit Finite Element implementation of the model uses an interpolation strategy towards collagen fiber stress determination and results in a memory-efficient representation of the model. A number of test cases, including patient-specific geometries, establish the efficiency of the description and demonstrate its ability to explain qualitative properties reported from macroscopic experimental studies of tendon and vascular tissue.

ColMap: A memory-efficient occupancy grid mapping framework

In order to possess a significant degree of autonomy, a robot must be able to perceive its environment and store a representation of that environment for use in tasks such as localisation, navigation, collision avoidance, and higher decision making. It must do this subject to constraints on memory and processing power typical of the embedded computer systems commonly found on small robotic devices. These constraints are particularly important for flying robots (i.e. unmanned aerial vehicles), for which weight must be minimised. The challenge of storing a detailed map of a large area on a small embedded computer has led to the development of many algorithms that exploit the sparsity of typical maps to create a more memory-efficient representation. In this paper, we demonstrate that the verticality of both natural and man-made structures can be exploited to create a framework that can store occupancy grid maps efficiently, without causing additional computational burden. The new framework achieves an order-of-magnitude reduction in memory footprint relative to widely-used occupancy grid mapping software, while also achieving a slight speed-up in map insertion and access times. We also make available LIDAR scans taken from a hexacopter of an indoor flight arena that can be used to assist in evaluating future mapping and SLAM developments.

UFOMap: An Efficient Probabilistic 3D Mapping Framework That Embraces the Unknown

3D models are an essential part of many robotic applications. In applications where the environment is unknown a-priori, or where only a part of the environment is known, it is important that the 3D model can handle the unknown space efficiently. Path planning, exploration, and reconstruction all fall into this category. In this letter we present an extension to OctoMap which we call UFOMap. UFOMap uses an explicit representation of all three states in the map, i.e., unknown, free, and occupied. This gives, surprisingly, a more memory efficient representation. We provide methods that allow for significantly faster insertions into the octree. Furthermore, UFOMap supports fast queries based on occupancy state using so called indicators and based on location by exploiting the octree structure and bounding volumes. This enables real-time colored octree mapping at high resolution (below 1 cm). UFOMap is contributed as a C++ library that can be used standalone but is also integrated into ROS.

Time- and memory-efficient representation of complex mesoscale potentials

We apply the modern technique of approximation of multivariate functions – tensor train cross approximation – to the problem of the description of physical interactions between complex-shaped bodies in a context of computational nanomechanics. In this note we showcase one particular example – van der Waals interactions between two cylindrical bodies – relevant to modeling of carbon nanotube systems. The potential is viewed as a tensor (multidimensional table) which is represented in compact form with the help of tensor train decomposition. The described approach offers a universal solution for the description of van der Waals interactions between complex-shaped nanostructures and can be used within the framework of such systems of mesoscale modeling as recently emerged mesoscopic distinct element method (MDEM).

HESML: A scalable ontology-based semantic similarity measures library with a set of reproducible experiments and a replication dataset

This work is a detailed companion reproducibility paper of the methods and experiments proposed by Lastra-Díaz and García-Serrano in (2015, 2016) [56–58], which introduces the following contributions: (1) a new and efficient representation model for taxonomies, called PosetHERep, which is an adaptation of the half-edge data structure commonly used to represent discrete manifolds and planar graphs; (2) a new Java software library called the Half-Edge Semantic Measures Library (HESML) based on PosetHERep, which implements most ontology-based semantic similarity measures and Information Content (IC) models reported in the literature; (3) a set of reproducible experiments on word similarity based on HESMLand ReproZip with the aim of exactly reproducing the experimental surveys in the three aforementioned works; (4) a replication framework and dataset, called WNSimRep v1, whose aim is to assist the exact replication of most methods reported in the literature; and finally, (5) a set of scalability and performance benchmarks for semantic measures libraries. PosetHERep and HESML are motivated by several drawbacks in the current semantic measures libraries, especially the performance and scalability, as well as the evaluation of new methods and the replication of most previous methods. The reproducible experiments introduced herein are encouraged by the lack of a set of large, self-contained and easily reproducible experiments with the aim of replicating and confirming previously reported results. Likewise, the WNSimRep v1 dataset is motivated by the discovery of several contradictory results and difficulties in reproducing previously reported methods and experiments. PosetHERepproposes a memory-efficient representation for taxonomies which linearly scales with the size of the taxonomy and provides an efficient implementation of most taxonomy-based algorithms used by the semantic measures and IC models, whilst HESML provides an open framework to aid research into the area by providing a simpler and more efficient software architecture than the current software libraries. Finally, we prove the outperformance of HESML on the state-of-the-art libraries, as well as the possibility of significantly improving their performance and scalability without caching using PosetHERep.

A compact random-access representation for urban modeling and rendering

We propose a highly memory-efficient representation for modeling and rendering urban buildings composed predominantly of rectangular block structures, which can be used to completely or partially represent most modern buildings. With the proposed representation, the data size required for modeling most buildings is more than two orders of magnitude less than using the conventional mesh representation. In addition, it substantially reduces the dependency on conventional texture maps, which are not space-efficient for defining visual details of building facades. The proposed representation can be stored and transmitted as images and can be rendered directly without any mesh reconstruction. A ray-casting based shader has been developed to render buildings thus represented on the GPU with a high frame rate to support interactive fly-by as well as street-level walk-through. Comparisons with standard geometric representations and recent urban modeling techniques indicate the proposed representation performs well when viewed from a short and long distance.

Data Analysis with Intersection Graphs

This paper presents a new framework for multivariate data analysis, based on graph theory, using intersection graphs [1]. We have named this approach DAIG – Data Analysis with Intersection Graphs. This new framework represents data vectors as paths on a graph, which has a number of advantages over the classical table representation of data. To do so, each node represents an atom of information, i.e. a pair of a variable and a value, associated with the set of observations for which that pair occurs. An edge exists between a pair of nodes whenever the intersection of their respective sets is not empty. We show that this representation of data as an intersection graph allows an easy and intuitive geometric interpretation of data observations, groups of observations, and results of multivariate data analysis techniques such as biplots, principal components, cluster analysis, or multidimensional scaling. These will appear as paths on the graph, relating variables, values and observations. This approach allows for a compact and memory efficient representation of data that contains many missing values or multi-valued attributes. The basic principles and advantages of this approach are presented with an example of its application to a simple toy problem. The main features of this methodology are illustrated with the aid software specifically developed for this purpose.

Compact Hilbert indices: Space-filling curves for domains with unequal side lengths

In this paper we define a new compact Hilbert index which, while maintaining all of the advantages of the standard Hilbert curve, permits spaces with unequal dimension cardinalities. The compact Hilbert index can be used in any application that would have previously relied on Hilbert curves but, in the case of unequal side lengths, provides a more memory efficient representation. This advantage is particularly important in distributed applications (Parallel, P2P and Grid), in which not only is memory space saved but communication volume is significantly reduced.

The disjunctive graph machine representation of the job shop scheduling problem

The disjunctive graph is one of the most popular models used for describing instances of the job shop scheduling problem, which has been very intensively explored. In this paper, a new time and memory efficient representation of the disjunctive graph is proposed. It has the form of a graph matrix and combines advantages of a few classical graph representations, enabling easy operating on the problem data. Computational experiments have proved higher efficiency of the proposed approach over the classical ones.

Numerical analysis of superposed GSPNs

The numerical analysis of various modeling formalisms profits from a structured representation for the generator matrix Q of the underlying continuous-time Markov chain, where Q is described by a sum of tensor (Kronecker) products of much smaller matrices. In this paper, we describe such a representation for the class of superposed generalized stochastic Petri nets (GSPNs), which is less restrictive than in previous work. Furthermore a new iterative analysis algorithm is proposed. It pays special attention to a memory-efficient representation of iteration vectors as well as to a memory-efficient structured representation of Q in consequence the new algorithm is able to solve models which have state spaces with several million states, where other exact numerical methods become impracticable on a common workstation.