Memory-efficient Data Structure Research Articles

Efficient Construction and Utilization of k-Ordered FM-indexes with kISS for Ultra-Fast Read Mapping in Large Genomes.

The Full-text index in Minute space (FM-index) is a memory-efficient data structure widely used in bioinformatics for solving the fundamental pattern-matching task of searching for short patterns within a long reference. With the demand for short query patterns, the k-ordered concept has been proposed for FM-indexes. However, few construction algorithms in the state of the art fully exploit this idea to achieve significant speedups in the pan-genome era. We introduce the k-ordered Induced Suffix Sorting (kISS) for efficient construction and utilization of k-ordered FM-indexes. We present an algorithmic workflow for building k-ordered suffix arrays, incorporating two novel strategies to improve time and memory efficiency. We also demonstrate the compatibility of integrating k-ordered FM-indexes with locate operations in FMtree. Experiments show that kISS can improve the construction time, and the generated k-ordered suffix array can also be applied to FMtree without any additional in computation or memory usage. https://github.com/jhhung/kISS. Supplementary data are available at Bioinformatics online.

Partitioned 2D Set-Pruning Segment Trees with Compressed Buckets for Multi-Dimensional Packet Classification

Abstract Multi-dimensional packet classification is one of the most important functions to support various services in next generation routers. Both the memory-efficient data structure to support larger rule tables and the hardware architecture to achieve a higher throughput are desired. In this paper, we propose a parallel and pipelined architecture called Set-Pruning Segment Trees with Buckets (SPSTwB) for multi-dimensional packet classification. SPSTwB significantly reduces rule duplication based on a novel partitioning scheme and an efficient bucket merging scheme. The key feature of our proposed architecture is that memory consumption is reduced significantly regardless of the characteristics of various rule tables. In addition, the logic complexity of each pipeline stage is simplified by not storing rule IDs and priorities and thus it can run at a high clock rate. The proposed scheme needs less than 20 bytes per rule for various 100 K rule tables generated by ClassBench. In addition, the proposed scheme supports fast incremental rule update. The proposed pipelined architecture can achieve a throughput of 134 Gbps from the implementation on Xilinx Virtex-7 FPGA device with dual-ported Block RAM.

Evaluation Performance of Bloom Filter in Blockchain Network

A blockchain is a secret, scalability and decentralized p2p network in which all nodes follow similar protocols, preventing any single node from controlling the basic structure. In blockchain, Bloom filters are used to preserve the privacy of lightweight nodes. Bloom filter is a memory efficient randomized data structure to represent a set in order to support associate queries. Bloom filters offer a trade-off between (elements size in the network and bandwidth) and privacy metrics in untrusted environments. This paper proposes an analysis to evaluate the performance of Bloom filters. The evaluation results are based on the statistical distribution, standard deviation, entropy and y-deniability that are used by the attacker to analysis the leakage of the Bloom filter algorithm. Experimental results show that the degree of privacy in the network needs a large amount of elements and more bandwidth using the Bloom filter algorithm.

Design and Evaluation of Memory Efficient Data Structure Scheme for Energy Drainage Attacks in Wireless Sensor Networks

Design and Evaluation of Memory Efficient Data Structure Scheme for Energy Drainage Attacks in Wireless Sensor Networks

SparkEC: speeding up alignment-based DNA error correction tools

BackgroundIn recent years, huge improvements have been made in the context of sequencing genomic data under what is called Next Generation Sequencing (NGS). However, the DNA reads generated by current NGS platforms are not free of errors, which can affect the quality of downstream analysis. Although error correction can be performed as a preprocessing step to overcome this issue, it usually requires long computational times to analyze those large datasets generated nowadays through NGS. Therefore, new software capable of scaling out on a cluster of nodes with high performance is of great importance.ResultsIn this paper, we present SparkEC, a parallel tool capable of fixing those errors produced during the sequencing process. For this purpose, the algorithms proposed by the CloudEC tool, which is already proved to perform accurate corrections, have been analyzed and optimized to improve their performance by relying on the Apache Spark framework together with the introduction of other enhancements such as the usage of memory-efficient data structures and the avoidance of any input preprocessing. The experimental results have shown significant improvements in the computational times of SparkEC when compared to CloudEC for all the representative datasets and scenarios under evaluation, providing an average and maximum speedups of 4.9times and 11.9times, respectively, over its counterpart.ConclusionAs error correction can take excessive computational time, SparkEC provides a scalable solution for correcting large datasets. Due to its distributed implementation, SparkEC speed can increase with respect to the number of nodes in a cluster. Furthermore, the software is freely available under GPLv3 license and is compatible with different operating systems (Linux, Windows and macOS).

Sarve: synthetic data and local differential privacy for private frequency estimation

The collection of user attributes by service providers is a double-edged sword. They are instrumental in driving statistical analysis to train more accurate predictive models like recommenders. The analysis of the collected user data includes frequency estimation for categorical attributes. Nonetheless, the users deserve privacy guarantees against inadvertent identity disclosures. Therefore algorithms called frequency oracles were developed to randomize or perturb user attributes and estimate the frequencies of their values. We propose Sarve, a frequency oracle that used Randomized Aggregatable Privacy-Preserving Ordinal Response (RAPPOR) and Hadamard Response (HR) for randomization in combination with fake data. The design of a service-oriented architecture must consider two types of complexities, namely computational and communication. The functions of such systems aim to minimize the two complexities and therefore, the choice of privacy-enhancing methods must be a calculated decision. The variant of RAPPOR we had used was realized through bloom filters. A bloom filter is a memory-efficient data structure that offers time complexity of O(1). On the other hand, HR has been proven to give the best communication costs of the order of log(b) for b-bits communication. Therefore, Sarve is a step towards frequency oracles that exhibit how privacy provisions of existing methods can be combined with those of fake data to achieve statistical results comparable to the original data. Sarve also implemented an adaptive solution enhanced from the work of Arcolezi et al. The use of RAPPOR was found to provide better privacy-utility tradeoffs for specific privacy budgets in both high and general privacy regimes.

Dynamically Allocated Bloom Filter-Based PIT Architectures

As a key component in implementing Named Data Networking (NDN), Pending Interest Table (PIT) requires an efficient exact-matching algorithm for a scalable and fast PIT lookup. A Bloom filter (BF) is a memory-efficient data structure for performing exact matching operations. In this paper, three different BF-based PIT architectures are proposed: PIT using functional Bloom filters (FBF-PIT), PIT using counting Bloom filters with return values (rCBF-PIT), and a refined rCBF-PIT with signatures (R-rCBF-PIT). The proposed BF-based PITs incrementally allocate a new BF for storing multiple incoming faces of Interest packets with the same content name. For a Data packet lookup, the proposed PIT architectures simultaneously access every BF structure to find matching faces and delete the faces (i.e., matching Interest packet information). The functional Bloom filter (FBF) used in an FBF-PIT is a key-value data structure that stores values only without keys. However, because the number of non-reusable conflict cells in the FBF increases as the number of stored packets increases in the FBF-PIT, the indeterminable rate increases. To decrease the indeterminable rate, we propose the rCBF-PIT, which uses counting Bloom filters with return values (rCBFs), allowing reusable conflict cells. False positives for Interest packets lead to incorrect deletions that can cause false negatives for incoming Data packets. Because most of the false positives occur in the first BF structure, we finally propose the R-rCBF-PIT, in which the first rCBF is replaced with an rCBF with a signature field. The proposed PITs also provide an aging mechanism using a valid bit and a hit bit for entry expiration. Simulation results show that rCBF-PIT and R-rCBF-PIT both reduce the indeterminable rate by more than 81% compared with FBF-PIT. The results also show that R-rCBF-PIT resolves false negatives caused by incorrect deletions by including the signature fields in the first rCBF.

An Optimized FPGA-Based Real-Time NDT for 3D-LiDAR Localization in Smart Vehicles

Light detection and ranging (LiDAR) based normal distribution transform (NDT) is a popular localization algorithm in smart vehicles for its robustness and accuracy. To meet the requirement of higher speed of smart vehicles, LiDAR point clouds become increasingly large due to the increase of not only the number of laser channels but also their frame rates. However, previous NDT implementations fail to achieve real-time performance when handling large point clouds because they require an enormous number of search operations and perform intensive computations per search operation. To address this issue, we propose an optimized NDT algorithm and its FPGA implementation. First, we effectively speed up each search operation by proposing a new non-recursive and memory-efficient data structure, called occupation-aware-voxel-structure (OAVS). Second, we significantly reduce the number of search operations by removing redundant searches with an optimized semantic-assisted OAVS-based NDT algorithm (SEO-NDT). Third, we further improve the real-time and energy efficiency by proposing a streaming FPGA accelerator architecture for SEO-NDT. Compared with the state-of-the-art embedded CPU and GPU implementations, our FPGA implementation achieves up to 35.85x and 2.44x execution time speed up, respectively.

A Performance Evaluation of DRAM Access for In-Memory Databases

The latest CPUs(computer cpu processors) employ multiple cores, massively superscalar pipelines, out-of-order execution of tons of instructions, and advanced SIMD capabilities, which can hide the memory access latency. And most of recent memory-oriented data structures have already benefit from these features. However, due to the complexity of data organization, these CPUs do not always work well in memory resident database systems (MMDBs), particularly regarding storing data in dynamic random-access memory (DRAM). This article studies memory-efficient data structures by analyzing the run time, access latency, cache misses, instructions per cycle (IPC), and DRAM reads (bytes). Then, we design and implement two data organization schemas in the main memory database: dispersing data block organization and clustering data block organization. Using algorithmic engineering and careful attention to internal parallelism, cache alignment can hide the memory access latency. However, we find that these data structures work well in some cases, though they have been eclipsed in the face of complex access paths. To determine the reasons, we study the impact of database techniques on memory access latency, such as data partitioning, storage models, and by processing algorithms. With the specific main memory database system, we estimate the performance of each data organization schema based on DRAM DDR4 and the latest Intel Haswell microarchitecture. In conclusion, this work will make DRAM access applicable in real-world situations by implementing the schema to systems, such as in-memory databases.

Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Per-flow measurement is a critical issue in computer networks, and one of its key tasks is to count the number of packets in each flow (for big streaming data). The literature has demonstrated that sketch is the most memory-efficient data structure for the counting task, and is widely used in distributed systems. Existing sketches often use many counters that are of the same size to record the number of packets in a flow, thus the counters are forced to be large enough to accommodate the size of the largest flow. Unfortunately, as most flows are small (i.e., mice flows) and only a very few flows are large (i.e., elephant flows), many counters represent very small values, which is a waste of memory. Sketches are often stored in fast but expensive memory (e.g., SRAM), thus it is critical to achieve high memory efficiency. To address this issue, we propose a novel sketch, namely the Diamond sketch. The Diamond sketch is composed of atom sketches, and each atom sketch uses small counters. The key idea of Diamond is to dynamically assign an appropriate number of atom sketches to each flow on demand, thus optimizing memory efficiency . Experimental results show that the Diamond sketch outperforms the best of the five typical sketches by up to 508.3 times in terms of relative error while keeping comparable speed. We made the source code of all the six sketches available on GitHub [1] .

A memory-efficient canonical data structure for decimal floating point arithmetic systems modeling and verification

Decimal floating point (DFP) number representation was proposed in IEEE-754-2008 in order to overcome binary floating point inaccuracy. Neglecting binary floating point verification has resulted in significant validity and economic losses. Formal verification can be a solution to similar DFP design problems. Verification techniques aiming at DFP are limited to functional methods whereas formal approaches have been neglected and traditional decision diagrams cannot model DFP representation complexity. In this paper, we propose an efficient canonical data structure that can model DFP properties. Our novel data structure models coefficient, exponent, sign, and bias of a DFP number. We will prove mathematically that our data structure is canonical. We will also show that the size of the proposed data structure will grow linearly for a DFP number for all format lengths, so our data structure can be built with a reasonable amount of memory and run time. Experimental results support our mathematical discussion for linear growth of the DFP data structure. Moreover, comparison of our data structure with integer decision diagrams reveals that our model is also efficient for modeling integer functions when utilized as an integer level diagram.

Near-Optimal Data Structure for Approximate Range Emptiness Problem in Information-Centric Internet of Things

The approximate range emptiness problem requires a memory-efficient data structure D to approximately represent a set S of n distinct elements chosen from a large universe U= {0,1,⋯,N-1} and answer an emptiness query of the form “S∩[a;b]=0?” for an interval [a;b] of length L (a,b∈U), with a false positive rate e. The designed D for this problem can be kept in high-speed memory and quickly determine approximately whether a query interval is empty or not. Thus, it is crucial for facilitating online query processing in the information-centric Internet of Things applications, where the IoT data are continuously generated from a large number of resource-constrained sensors or readers and then are processed in networks. However, the existing works on the approximate range emptiness problem only consider the simple case when the set S is static, rendering them unsuitable for the continuously generated IoT data. In this paper, we study the approximate range emptiness problem over sliding windows in the IoT Data streams, denoted by e-ARESD-problem, where both insertion and deletion are allowed. We first prove that, given a sliding window size n and an interval length L, the lower bound of memory bits needed in any data structure for e-ARESD-problem is n log 2 (nL/e)+Θ(n). Then, a data structure is proposed and proved to be within a factor of 1.33 of the lower bound. The extensive simulation results demonstrate the advantage of the efficiency of our data structure over the baseline approach.

An accurate estimation algorithm for big data streams

Sketch is a memory-efficient data structure, and is used to store and query the frequency of any item in a given multiset. As it can achieve fast query and update, it has been applied to various fields. Different sketches have different advantages and disadvantages. Sketches are originally proposed for estimation of flow size in network measurement. The key factor of sketches for network measurement is the insertion speed and accuracy. In this paper, we propose a new sketch, which can significantly improve the insertion speed while improving the accuracy. Our key methods include on-chip/off-chip separation and partial update algorithm. Extensive experimental results show that our sketch significantly outperforms the state-of-the-art both in terms of accuracy and speed.

An efficient dynamic superset bit-vector approach for mining frequent closed itemsets and their lattice structure

Fast discovery of association rules from millions of transactions in a variety of large databases has now become a major challenge in data mining domain. Frequent itemsets and frequent closed itemsets are key sources of mining association rules. Association rules can be mined efficiently from lattice of itemsets. Non-redundancy, accuracy, time efficiency and memory usage are the factors those need to be considered while developing algorithms in order to extract meaningful association rules. In this paper, we propose an efficient bit-vector approach that exploits dual properties, tidset and superset information of an itemset in order to mine frequent closed itemsets with their lattice structure. We introduce a new memory efficient data structure called dynamic superset bit-vector to establish the relationship among frequent closed itemsets in a lattice. The novelty of our approach is that it effectively uses dual data structures called a dynamic bit-vector and a dynamic superset bit-vector jointly in order to reduce the search space and eliminate the generators of non-closed itemsets. Our proposed approach efficiently builds up the subset-superset relationship among the closed itemsets of the lattice structure in a bottom-up manner. Extensive experiments using real-life datasets and pervasive performance comparison with the existing works prove the efficiency and scalability of our proposed approach.

Me-CLOCK:A Memory-Efficient Framework to Implement Replacement Policies for Large Caches

Solid State Drives (SSDs) have been extensively deployed as the cache of hard disk-based storage systems. The SSD-based cache generally supplies ultra-large capacity, whereas managing so large a cache introduces excessive memory overhead, which in turn makes the SSD-based cache neither cost-effective nor energy-efficient. This work targets to reduce the memory overhead introduced by the replacement policy of SSD-based cache. Traditionally, data structures involved in cache replacement policy reside in main memory. While these in-memory data structures are not suitable for SSD-based cache any more since the cache is much larger than ever. We propose a memory-efficient framework which keeps most data structures in SSD while just leaving the memory-efficient data structure (i.e., a new bloom proposed in this work) in main memory. Our framework can be used to implement any LRU-based replacement policies under negligible memory overhead. We evaluate our proposals via theoretical analysis and prototype implementation. Experimental results demonstrate that, our framework is practical to implement most replacement policies for large caches, and is able to reduce the memory overhead by about $10 \times$ .

Tree-based string pattern matching on FPGAs

Network intrusion detection systems (NIDSs) monitor Internet Protocol (IP) traffic to detect anomalous and malicious activities on a network. Despite the plethora of studies in this field, hardware-based string matching engines that can accommodate the advancements in optical networking technology are still in high demand. Furthermore, memory efficient data structures to store intrusion patterns have recently received a great deal of research attention. In this paper, we introduce a tree-based pattern matching (TPM) scheme that comprises a forest of Binary Search Tree (BST) data structures and an accommodating high-throughput multi-pipelined architecture for scalable string matching on hardware. To improve the resource efficiency in hardware implementations, we enhanced TPM scheme (extended-TPM) with two novel tree structures, namely BST-epsilon (BST∊) and hierarchical BST (H-BST). Our entire design accomplishes a memory efficiency of 1.07 bytes/char for the latest Snort dictionary. Utilizing a state-of-the-art Field Programmable Gate Arrays (FPGAs), TPM architecture can sustain a throughput of 2.7 Gbps.

Konnector v2.0: pseudo-long reads from paired-end sequencing data.

BackgroundReading the nucleotides from two ends of a DNA fragment is called paired-end tag (PET) sequencing. When the fragment length is longer than the combined read length, there remains a gap of unsequenced nucleotides between read pairs. If the target in such experiments is sequenced at a level to provide redundant coverage, it may be possible to bridge these gaps using bioinformatics methods. Konnector is a local de novo assembly tool that addresses this problem. Here we report on version 2.0 of our tool.ResultsKonnector uses a probabilistic and memory-efficient data structure called Bloom filter to represent a k-mer spectrum - all possible sequences of length k in an input file, such as the collection of reads in a PET sequencing experiment. It performs look-ups to this data structure to construct an implicit de Bruijn graph, which describes (k-1) base pair overlaps between adjacent k-mers. It traverses this graph to bridge the gap between a given pair of flanking sequences.ConclusionsHere we report the performance of Konnector v2.0 on simulated and experimental datasets, and compare it against other tools with similar functionality. We note that, representing k-mers with 1.5 bytes of memory on average, Konnector can scale to very large genomes. With our parallel implementation, it can also process over a billion bases on commodity hardware.

Efficient computation of Lyapunov functions for Morse decompositions

We present an efficient algorithm for constructing piecewise constant Lyapunov functions for dynamics generated by a continuous nonlinear map defined on a compact metric space.We provide a memory efficient data structure for storing nonuniform grids on which the Lyapunov function is defined and give bounds on the complexity of the algorithm for both time and memory.We prove that if the diameters of the grid elements go to zero, then the sequence of piecewise constant Lyapunov functions generated by our algorithm converge to a continuous Lyapunov function for the dynamics generated the nonlinear map.We conclude by applying these techniques to two problems from population biology.

A Fast and Simple Online Synchronous Context Free Grammar Extractor

Abstract Hierarchical phrase-based machine translation systems rely on the synchronous context free grammar formalism to learn and use translation rules containing gaps. The grammars learned by such systems become unmanageably large even for medium sized parallel corpora. The traditional approach of preprocessing the training data and loading all possible translation rules into memory does not scale well for hierarchical phrase-based systems. Online grammar extractors address this problem by constructing memory efficient data structures on top of the source sideof the parallel data (often based on suffix arrays), which are usedto efficiently match phrases in the corpus and to extract translation rules on the fly during decoding. This paper describes an open source implementation of an online synchronous context free grammar extractor. Our approach builds on the work of Lopez (2008a) and introduces a new technique for extending the lists of phrase matches for phrases containing gaps that reduces the extraction time by a factor of 4. Our extractor is available as part of the cdec toolkit1 (Dyer et al., 2010).

Real-time 3D reconstruction at scale using voxel hashing

Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. The basic problem takes live overlapping depth maps as input and incrementally fuses these into a single 3D model. This is challenging particularly when real-time performance is desired without trading quality or scale. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure. Our system uses a simple spatial hashing scheme that compresses space, and allows for real-time access and updates of implicit surface data, without the need for a regular or hierarchical grid data structure. Surface data is only stored densely where measurements are observed. Additionally, data can be streamed efficiently in or out of the hash table, allowing for further scalability during sensor motion. We show interactive reconstructions of a variety of scenes, reconstructing both fine-grained details and large scale environments. We illustrate how all parts of our pipeline from depth map pre-processing, camera pose estimation, depth map fusion, and surface rendering are performed at real-time rates on commodity graphics hardware. We conclude with a comparison to current state-of-the-art online systems, illustrating improved performance and reconstruction quality.