Space-efficient Data Structure Research Articles

UltraLogLog: A Practical and More Space-Efficient Alternative to HyperLogLog for Approximate Distinct Counting

Since its invention HyperLogLog has become the standard algorithm for approximate distinct counting. Due to its space efficiency and suitability for distributed systems, it is widely used and also implemented in numerous databases. This work presents UltraLogLog, which shares the same practical properties as HyperLogLog. It is commutative, idempotent, mergeable, and has a fast guaranteed constant-time insert operation. At the same time, it requires 28% less space to encode the same amount of distinct count information, which can be extracted using the maximum likelihood method. Alternatively, a simpler and faster estimator is proposed, which still achieves a space reduction of 24%, but at an estimation speed comparable to that of HyperLogLog. In a non-distributed setting where martingale estimation can be used, UltraLogLog is able to reduce space by 17%. Moreover, its smaller entropy and its 8-bit registers lead to better compaction when using standard compression algorithms. All this is verified by experimental results that are in perfect agreement with the theoretical analysis which also outlines potential for even more space-efficient data structures. A production-ready Java implementation of UltraLogLog has been released as part of the open-source Hash4j library.

A Learned Cuckoo Filter for Approximate Membership Queries over Variable-sized Sliding Windows on Data Streams

Designing a space-efficient data structure to answer membership queries while ensuring high accuracy and real-time response is a challenging task in the field of stream processing. Many techniques have been developed to answer these queries in a sliding windows manner. However, assuming the user will conduct the query with the presupposed window size is not always practical. In this paper, we introduce a novel data structure called Learned Cuckoo Filter (LCF). It can provide satisfactory results for the approximate membership query on data streams, regardless of the user-defined query windows. LCF operates by adaptively maintaining cuckoo filters with the assistance of a well-trained oracle that learned the frequency feature of the data within the stream. To further enhance memory utilization, we develop a compact version of LCF (denoted by LCF_C), which selectively removes redundant information to reduce space consumption without compromising query accuracy. Furthermore, we conduct a thorough theoretical analysis of query accuracy and provide detailed guidelines for optimal parameter selection (denoted by LCF_O). Extensive experimental studies on synthetic and real-world datasets demonstrate the superiority of the proposed methods in terms of both space consumption and accuracy. Compared to the state-of-the-art algorithms, LCF_O can reduce up to 61% of space cost at the same error level, and achieve up to 12× improved accuracy with the same space cost.

Succinct data structure for path graphs

We consider the problem of designing a succinct data structure for path graphs, that generalizes interval graphs, on n vertices while efficiently supporting degree, adjacency, and neighbourhood queries. We provide the following two solutions for this problem:1.an nlog⁡n+o(nlog⁡n)-bit succinct data structure that supports adjacency query in O(log⁡n) time, neighbourhood query in O(dlog⁡n) time and finally, degree query in min⁡{O(log2⁡n),O(dlog⁡n)} time where d is the degree of the queried vertex.2.an O(nlog2⁡n)-bit space-efficient data structure that supports adjacency, neighborhood, and degree queries optimally. Central to our data structures is the usage of the heavy path decomposition, followed by careful bookkeeping using an orthogonal range search data structure using wavelet trees among others, which may be of independent interest for designing succinct data structures for other graph classes.

Logical labeling schemes

A labeling scheme is a space-efficient data structure for encoding graphs from a particular graph class. The idea is to assign each vertex of a graph a short label s.t. adjacency of two vertices can be algorithmically determined from their labels. For instance, planar and interval graphs have labeling schemes. The algorithm used to determine adjacency—called label decoding algorithm—should be of low complexity since the time it takes to execute corresponds to the time to query an edge in that representation.What graph classes have a labeling scheme if the label decoding algorithm must be very efficient, e.g. computable in constant time? In order to investigate this question we introduce logical labeling schemes where the label decoding algorithm is expressed as a first-order formula and consider their properties such as the relation to labeling schemes defined in terms of classical complexity classes. Additionally, we introduce a notion of reduction between graph classes in terms of boolean formulas and show completeness results.

EBF: an enhanced Bloom Filter for intrusion detection in IoT

Intrusion Detection is essential to identify malicious incidents and continuously alert many users of the Internet of Things (IoT). The constant monitoring of events generated from many devices connected to the IoT and the extensive analysis of every event based on predefined security policies consumes enormous resources. Accordingly, performance enhancement is a crucial concern of Intrusion Detection in IoT and other massive Big Data Applications to ensure a secure environment. Like many Big Data Applications, the Intrusion Detection system of the IoT needs to employ the fast membership filter, Bloom Filter, to quickly identify possible attacks. Bloom Filter is an admiringly fast and space-efficient data structure that quickly handles elements of extensive datasets in small memory space. However, the trade-off between the query performance, the number of hash functions, memory space, and false positive probability remains an issue of Bloom Filter. Thus, this article presents an enhanced Bloom Filter (eBF) that remarkably improves memory efficiency and introduces new techniques to accelerate the filtering of malicious URLs. We experimentally show the efficacy of eBF using a real Intrusion Detection dataset. The experimental result shows that the proposed filter is remarkably memory efficient, faster, and more accurate than the state-of-the-art filters. eBF requires 15.6x, 13x, and 8x less memory compared with Standard Bloom Filter, Cuckoo filter, and robustBF, respectively. Therefore, this new system significantly enhances the performance of Intrusion Detection of IoT that concurrently monitors several billion events crosschecking with the defined security policies.

SplinterDB and Maplets: Improving the Tradeoffs in Key-Value Store Compaction Policy

A critical aspect of modern key-value stores is the interaction between compaction policy and filters. Aggressive compaction reduces the on-disk footprint of a key-value store and can improve query performance, but can reduce insertion throughput because it is I/O and CPU expensive. Filters can mitigate the query costs of lazy compaction, but only if they fit in RAM, limiting the scalability of queries with lazy compaction. And, with fast storage devices, the CPU costs of querying filters in a lazy compacting system can be significant. In this work, we present Mapped SplinterDB, a key-value store that achieves excellent insertion performance, query performance, space efficiency, and scalability by replacing filters with maplets, space-efficient data structures that act as lossy maps with false positives. Critically, we use quotient maplets, which can be merged and resized without access to the underlying data, enabling us to decouple compaction of the data from compaction of the quotient maplets. Thus Mapped SplinterDB can compact data lazily and quotient maplets aggressively, so that each level has multiple sorted runs of data but only one quotient maplet. Quotient maplets are so small that compacting them aggressively is still cheaper than compacting the (much larger) data lazily, so overall we get the insertion performance of a lazily compacted system. And, since there is only one quotient maplet to query on each level, we get the query performance of an aggressively compacted system. Furthermore, quotient maplets can accelerate queries even when they don't fit in RAM, improving scalability to huge datasets. We also show how to use quotient maplets to estimate when a compaction could resolve a high density of updates, enabling Mapped SplinterDB to perform targeted compactions for space recovery. In our benchmarks, Mapped SplinterDB matches the insertion performance of SplinterDB, a state-of-the-art lazily compacted system, and beats RocksDB, an aggressive compacting system, by up to 9×. On queries, Mapped SplinterDB outperforms SplinterDB and RocksDB by up to 89% and 83%, respectively, and scales gracefully to huge datasets. Mapped SplinterDB is able to dynamically trade update performance for space efficiency, resulting in space overheads on update-heavy workloads as low as 15-61%, whereas RocksDB had 80-117% and SplinterDB had up to 137% space overhead.

Smash: Flexible, Fast, and Resource-efficient Placement and Lookup of Distributed Storage

Large-scale distributed storage systems, such as object stores, usually apply hashing-based placement and lookup methods to achieve scalability and resource efficiency. However, when object locations are determined by hash values, placement becomes inflexible, failing to optimize or satisfy application requirements such as load balance, failure tolerance, parallelism, and network/system performance. This work presents a novel solution to achieve the best of two worlds: flexibility while maintaining cost-effectiveness and scalability. The proposed method Smash is an object placement and lookup method that achieves full placement flexibility, balanced load, low resource cost, and short latency. Smash utilizes a recent space-efficient data structure and applies it to object-location lookups. We implement Smash as a prototype system and evaluate it in a public cloud. The analysis and experimental results show that Smash achieves full placement flexibility, fast storage operations, fast recovery from node dynamics, and lower DRAM cost (<60%) compared to existing hash-based solutions such as Ceph and MapX.

Compact representations of spatial hierarchical structures with support for topological queries

Among different spatial data models, the topological model for spatial regions explicitly represents common boundaries. This model pursues the efficiency of topology-related queries and the elimination of data redundancy. This paper proposes several space-efficient data structures to support access to the topological representation of two-dimensional regions that are organized in a multi-granular or hierarchical structure, such as the political and administrative partition of a country. In the context of these hierarchies, we focus on queries that search for inclusion, disjointness, and adjacency between regions. The proposed structures build upon compact planar graph embeddings, which show to have a good trade-off between space and time.

False Event Message Detection Robust to Burst Attacks in Wireless Sensor Networks

A sensor device can be used to detect target events at low cost. Moreover, there is a significant risk of sensor nodes being compromised or captured within large wireless sensor networks (WSNs). The transmission of valid event messages to users by a WSN can be hindered by network congestion due to false event messages in a compromised node. Several methods are available for detecting false event messages. However, they rely on long bits of authentication codes and hence do not provide a fundamental solution to prevent network congestion. In the proposed method, various hashing vectors, which are space-efficient data structures that can determine whether the given data are an element of a set, are created based on the correct combination of authentication codes and placed in each node in advance. An event message contains an XOR of the authentication codes, and each node verifies it based on its hashing vector. If a node is acquired illegally, the information of the hashing vector and the XOR information of the authentication codes assigned to the correct event message is compromised, so we propose an algorithm to update the information securely. Compared to existing research, the number of hops required to detect a false event message increases by only about one hop, but the amount of traffic that a malicious node can generate can be reduced by about 60% or more. In other words, the proposed method effectively reduces the amount of traffic an attacker can generate with false event messages, which also reduces the overall network congestion.

Succinct dynamic de Bruijn graphs.

The de Bruijn graph is one of the fundamental data structures for analysis of high throughput sequencing data. In order to be applicable to population-scale studies, it is essential to build and store the graph in a space- and time-efficient manner. In addition, due to the ever-changing nature of population studies, it has become essential to update the graph after construction, e.g. add and remove nodes and edges. Although there has been substantial effort on making the construction and storage of the graph efficient, there is a limited amount of work in building the graph in an efficient and mutable manner. Hence, most space efficient data structures require complete reconstruction of the graph in order to add or remove edges or nodes. In this article, we present DynamicBOSS, a succinct representation of the de Bruijn graph that allows for an unlimited number of additions and deletions of nodes and edges. We compare our method with other competing methods and demonstrate that DynamicBOSS is the only method that supports both addition and deletion and is applicable to very large samples (e.g. greater than 15 billion k-mers). Competing dynamic methods, e.g. FDBG cannot be constructed on large scale datasets, or cannot support both addition and deletion, e.g. BiFrost. DynamicBOSS is publicly available at https://github.com/baharpan/dynboss. Supplementary data are available at Bioinformatics online.

Improving the Run-Time of Space-Efficient n-Gram Data Structures Using Apache Spark.

Storing information in memory efficiently is one of the most significant challenges in computer science. The two main factors that consist an efficient data structure is the reduction of space and time consumption. There is a plethora of different tools able to reduce the run-time of a process, and Apache Spark is one of these; it is a computing framework that is using clusters to execute a process. There are two key features in this software, a directed acyclic graph (DAG) that maps the execution process and the resilient distributed datasets (RDD), which allow large in-memory computations. In order to construct a data structure, which is space- and time-efficient, we have to utilize the corresponding framework. A comparison of the run-time improvement with the use of Spark is also provided. Finally, to prove the efficacy of this software tool, we construct a space-efficient data structure and compare the run-time with and without its use.

An index for moving objects with constant-time access to their compressed trajectories

ABSTRACT As the number of vehicles and devices equipped with GPS technology has grown explosively, an urgent need has arisen for time- and space-efficient data structures to represent their trajectories. The most commonly desired queries are the following: queries about an object’s trajectory, range queries, and nearest neighbor queries. In this paper, we consider that the objects can move freely and we present a new compressed data structure for storing their trajectories, based on a combination of logs and snapshots, with the logs storing sequences of the objects’ relative movements and the snapshots storing their absolute positions sampled at regular time intervals. We call our data structure ContaCT because it provides Constant- time access to Compressed Trajectories. Its logs are based on a compact partial-sums data structure that returns cumulative displacement in constant time, and allows us to compute in constant time any object’s position at any instant, enabling a speedup when processing several other queries. We have compared ContaCT experimentally with another compact data structure for trajectories, called GraCT, and with a classic spatio-temporal index, the MVR-tree. Our results show that ContaCT outperforms the MVR-tree by orders of magnitude in space and also outperforms the compressed representation in time performance.

Range closest-pair search in higher dimensions

Range closest-pair (RCP) search is a range-search variant of the classical closest-pair problem, which aims to store a given set S of points into some space-efficient data structure such that when a query range Q is specified, the closest pair in S∩Q can be reported quickly. RCP search has received attention over years, but the primary focus was only on R2. In this paper, we study RCP search in higher dimensions. We give the first nontrivial RCP data structures for orthogonal, simplex, halfspace, and ball queries in Rd for any constant d. Furthermore, we prove a conditional lower bound for orthogonal RCP search for d≥3.

PHISHSTORM: Detection Mechanism for Finding the Phishing and Legitimate Websites by Implementing the Search Engine

This paper proposes software for detecting the phishing and legitimate websites. In that software, we build and explain the mechanism behind the detection of the phishing and legitimate websites. Phishing is defined as a fraudulent attempt, used to obtain confidential information through which an attacker emulates an actual person or institution by disguising them as an official person or entity through e-mail or other communication mediums. We created this project to safeguard the users from the phishing websites so that they won’t lose their personal details like usernames, passwords and credit card details. Here, we combined Storm in the architecture, thereby detecting the phishing and legitimate URL’s manually. A bloomfilter is a space-efficient data structure used to test the element either present in the set or may not be in the set. The simulation is carried out in the search engine, entering the search word, viewing the phishing and legitimate websites, downloading the files, give feedback about the websites and transformation is done wherever it is needed.

Graph Filter: Enabling Efficient Topology Calibration

The topology of a network may change inevitably, due to dynamic behaviors of nodes and links, and failures of hardware and software. Many protocols and applications must be aware of the up-to-date topology of the underlying network. This triggers the topology calibration problem, which means to deduce those different nodes and links between two topologies. The Bloom filter and its variants are efficient to represent and calibrate two general sets. They, however, fail to represent all links and nodes in a topology simultaneously, and thus remain inapplicable to the topology calibration problem. In this paper, we design the graph filter, a novel space-efficient data structure to record not only the node set but also the link set of any given topology. Accordingly, given two topologies we aim to represent them via two respective graph filters, and thereafter deduce those different links in an invertible manner. To this end, we design three essential operations for graph filter, i.e., encoding, subtracting and decoding. Although such operations are sufficient to solve the topology calibration problem, two challenging issues still remain open. First, the XOR traps which occur with low probability at the encoding stage may result in a few miscalculations at the decoding stage. Thus, we propose another augmented decoding algorithm to lessen the impact of XOR traps via terminating illegal decodings. Second, several different links may form cycles in the worst case; hence, we further design a cycle destruction algorithm to make such different links decodable. We implement the graph filter and the associated topology calibration method. Comprehensive evaluations indicate that our method finishes the topology calibration task efficiently with high probability, incurs the least space overhead, and supports invertible decoding reasonably.

Vectored-Bloom Filter for IP Address Lookup: Algorithm and Hardware Architectures

The Internet Protocol (IP) address lookup is one of the most challenging tasks for Internet routers, since it requires to perform packet forwarding at wire-speed for tens of millions of incomming packets per second. Efficient IP address lookup algorithms have been widely studied to satisfy this requirement. Among them, Bloom filter-based approach is attractive in providing high performance. This paper proposes a high-speed and flexible architecture based on a vectored-Bloom filter (VBF), which is a space-efficient data structure that can be stored in a fast on-chip memory. An off-chip hash table is infrequently accessed, only when the VBF fails to provide address lookup results. The proposed architecture has been evaluated through both a behavior simulation with C language and a timing simulation with Verilog. The hardware implementation result shows that the proposed architecture can achieve the throughput of 5 million packets per second in a field programmable gate array (FPGA) operated at 100 MHz.

Decision Support Based Resource Allocation for Cost Reduction in Cloud Storage using Big Data Analytics

Provision of highly efficient storage for dynamically growing data is considered problem to be solved in data mining. Few research works have been designed for big data storage analytics. However, the storage efficiency using conventional techniques was not sufficient as where data duplication and storage overhead problem was not addressed. In order to overcome such limitations, Tanimoto Regressive Decision Support Based Blake2 Hashing Space Efficient Quotient Data Structure (TRDS-BHSEQDS) Model is proposed. Initially, TRDS-BHSEQDS technique gets larger number of input data as input. Then, TRDS-BHSEQDS technique computes 512 bits Blake2 hash value for each data to be stored. Consequently, TRDS-BHSEQDS technique applies Tanimoto Regressive Decision Support Model (TRDSM) where it carried outs regression analysis with application of Tanimoto similarity coefficient. During this process, proposed TRDS-BHSEQDS technique finds relationship between hash values of data by determining Tanimoto similarity coefficient value. If similarity value is ‘+1’, then TRDS-BHSEQDS technique considered that input data is already stored in BHSEQF memory. TRDS-BHSEQDS technique enhances the storage efficiency of big data when compared to state-of-the-art works. The performance of TRDS-BHSEQDS technique is measured in terms of storage efficiency, time complexity and space complexity and storage overhead with respect to different numbers of input big data.

Bayesian localization of CNV candidates in WGS data within minutes

BackgroundFull Bayesian inference for detecting copy number variants (CNV) from whole-genome sequencing (WGS) data is still largely infeasible due to computational demands. A recently introduced approach to perform Forward–Backward Gibbs sampling using dynamic Haar wavelet compression has alleviated issues of convergence and, to some extent, speed. Yet, the problem remains challenging in practice.ResultsIn this paper, we propose an improved algorithmic framework for this approach. We provide new space-efficient data structures to query sufficient statistics in logarithmic time, based on a linear-time, in-place transform of the data, which also improves on the compression ratio. We also propose a new approach to efficiently store and update marginal state counts obtained from the Gibbs sampler.ConclusionsUsing this approach, we discover several CNV candidates in two rat populations divergently selected for tame and aggressive behavior, consistent with earlier results concerning the domestication syndrome as well as experimental observations. Computationally, we observe a 29.5-fold decrease in memory, an average 5.8-fold speedup, as well as a 191-fold decrease in minor page faults. We also observe that metrics varied greatly in the old implementation, but not the new one. We conjecture that this is due to the better compression scheme. The fully Bayesian segmentation of the entire WGS data set required 3.5 min and 1.24 GB of memory, and can hence be performed on a commodity laptop.

A diversity-based search-and-routing approach for named-data networking

Nowadays, the Internet is mostly used for content generation, sharing and access. Users are no longer interested in connecting their computers and mobiles to an end system located at somewhere in the network border, but at obtaining pieces of content. This paradigm shift in Internet usage has motivated the proposal of Information-Centric Networking (ICN) architectures. ICN architectures replace the current connection oriented architecture for a general data oriented one. Contents are identified by their names and the architecture uses those names for routing and retrieving contents. The routers temporarily store copies of requested contents. Therefore, ICN introduces a largely distributed in-networking caching infrastructure. When content’s copies are highly spread across the network routers, end-to-end routing algorithms between client and servers are no longer the most effective way of retrieving contents. In this work, we extend DIVER, a routing algorithm that explores the network in order to search and retrieve router’s copies that are closer to the clients. DIVER uses probe packets to find routers copies, and the routers, upon receiving the probe packets, answer them by inserting their availability information in a space-efficient data structure. DIVER keeps the most diversified as possible the acquired availability information in the explorations in order to raise the probability of satisfying future requests for the same content. Simulation results show that DIVER is able to retrieve four times more chunks from routers compared to the end-to-end traditional routing.

Provably Secure Private Set Intersection With Constant Communication Complexity

Electronic information is increasingly shared among unreliable entities. In this context, one interesting problem involves two parties that secretly want to determine an intersection of their respective private data sets while none of them wish to disclose the whole set to the other. One can adopt a Private Set Intersection (PSI) protocol to address this problem preserving the associated security and privacy issues. In this article, the authors present the first PSI protocol that incurs constant (p(k)) communication complexity with linear computation overhead and is fast even for the case of large input sets, where p(k) is a polynomial in security parameter k. Security of this scheme is proven in the standard model against semi-honest entities. The authors combine somewhere statistically binding (SSB) hash function with indistinguishability obfuscation (iO) and space-efficient probabilistic data structure Bloom filter to design the scheme.