Data Structure For Membership Queries Research Articles

Scalable sequence database search using partitioned aggregated Bloom comb trees.

The Sequence Read Archive public database has reached 45 petabytes of raw sequences and doubles its nucleotide content every 2 years. Although BLAST-like methods can routinely search for a sequence in a small collection of genomes, making searchable immense public resources accessible is beyond the reach of alignment-based strategies. In recent years, abundant literature tackled the task of finding a sequence in extensive sequence collections using k-mer-based strategies. At present, the most scalable methods are approximate membership query data structures that combine the ability to query small signatures or variants while being scalable to collections up to 10000 eukaryotic samples. Results. Here, we present PAC, a novel approximate membership query data structure for querying collections of sequence datasets. PAC index construction works in a streaming fashion without any disk footprint besides the index itself. It shows a 3-6 fold improvement in construction time compared to other compressed methods for comparable index size. A PAC query can need single random access and be performed in constant time in favorable instances. Using limited computation resources, we built PAC for very large collections. They include 32000 human RNA-seq samples in 5 days, the entire GenBank bacterial genome collection in a single day for an index size of 3.5TB. The latter is, to our knowledge, the largest sequence collection ever indexed using an approximate membership query structure. We also showed that PAC's ability to query 500000 transcript sequences in less than an hour. PAC's open-source software is available at https://github.com/Malfoy/PAC.

A Multi-hashing Index for hybrid DRAM-NVM memory systems

Hybrid memory systems composed of DRAM and Non-Volatile Memory (NVM) promise the capacity benefits of NVM and the low-latency benefits of DRAM. Most existing hash-based indexes are designed for NVM only and do not exploit the benefits of DRAM. In this paper, we proposed a novel hybrid DRAM-NVM persistent and concurrent hashing index, named Multi-Hashing Index (MuHash). MuHash uses a multi-hash function scheme to solve the cascading write problem of open-addressed hash-based indexes in NVM. It employs a Cuckoo Filter, an approximate membership query data structure, to prune unnecessary NVM accesses for improving read performance. To maximize throughput in multi-thread environments, MuHash also includes a fine-grained concurrency control mechanism. We implemented MuHash for Intel Optane DC Persist Memory, and single-core experiments shows that MuHash achieves up to 90% higher read throughput compared to state-of-the-art hash-based indexes. On multicore experiments, MuHash achieves near-linear scalability for all operations.

QCF for deep packet inspection

Deep packet inspection (DPI) acts as a tool to control and classify incoming network traffic depending on users,content, applications and becomes a very important aspect of every network today. Instead of utilising only incoming packet header information, Internet service providers utilise DPI for security purposes, flow management, and routing. The high-performance challenges that face DPI techniques to manage line-speed packet processing with restricted available memory are the increasing traffic volumes and link rates. Bloom (BF) and quotient (QF) filters are effective randomised data structures for performing approximate membership queries. They have been employed in many networking applications including DPI. Nevertheless, they consume more calculations which degrade the system performance. Consequently, in this paper, the authors propose a new variant of membership query data structure, called quotient-based Cuckoo filter (QCF), that reflects a merging process between QF and CF to minimise the calculation overhead and employing it in a DPI system. QCF employs only two hash functions and consumes less calculation overhead than BF, QF, and CF. The implementation results show that using QCF as an identification tool in a DPI system result in a time improvement up to 77% over CF and up to 98% over BF and QF.

Squeakr: an exact and approximate k-mer counting system.

k-mer-based algorithms have become increasingly popular in the processing of high-throughput sequencing data. These algorithms span the gamut of the analysis pipeline from k-mer counting (e.g. for estimating assembly parameters), to error correction, genome and transcriptome assembly, and even transcript quantification. Yet, these tasks often use very different k-mer representations and data structures. In this article, we show how to build a k-mer-counting and multiset-representation system using the counting quotient filter, a feature-rich approximate membership query data structure. We introduce the k-mer-counting/querying system Squeakr (Simple Quotient filter-based Exact and Approximate Kmer Representation), which is based on the counting quotient filter. This off-the-shelf data structure turns out to be an efficient (approximate or exact) representation for sets or multisets of k-mers. Squeakr takes 2×-4.3× less time than the state-of-the-art to count and perform a random-point-query workload. Squeakr is memory-efficient, consuming 1.5×-4.3× less memory than the state-of-the-art. It offers competitive counting performance. In fact, it is faster for larger k-mers, and answers point queries (i.e. queries for the abundance of a particular k-mer) over an order-of-magnitude faster than other systems. The Squeakr representation of the k-mer multiset turns out to be immediately useful for downstream processing (e.g. de Bruijn graph traversal) because it supports fast queries and dynamic k-mer insertion, deletion, and modification. https://github.com/splatlab/squeakr available under BSD 3-Clause License. ppandey@cs.stonybrook.edu. Supplementary data are available at Bioinformatics online.

Enhancing Counting Bloom Filters Through Huffman-Coded Multilayer Structures

Bloom Filters are efficient randomized data structures for membership queries on a set with a certain known false positive probability. Counting Bloom Filters (CBFs) allow the same operation on dynamic sets that can be updated via insertions and deletions with larger memory requirements. This paper first presents a simple tight upper bound for counters overflow probability in CBFs, which is adopted in the design of more efficient CBFs. On the basis of such theoretical achievements, we introduce the idea of a hierarchical structure as well as the use of Huffman code to improve standard CBFs in terms of fast access and limited memory consumption (up to 50% of memory saving). The target could be the implementation of the compressed data structures in the small (but fast) local memory or on-chip SRAM of devices such as network processors. As an application of our algorithms, an anti-evasion system is finally proposed.

Hierarchical packet classification using a Bloom filter and rule-priority tries

Packet classification techniques have received significant attention in the network literature over the past 10 years, due to its fundamental role in the Internet routers. In recent years, Bloom filter, which is an efficient data structure for membership queries, becomes popular in the network applications. Though Bloom filter allows an error called “false positives,” the efficiency and the space saving overweigh this drawback when the false positive rate is properly controlled. In this paper, we proposed a packet classification algorithm based on a hierarchical approach. While the same data structure is used both for the source and the destination prefix fields in most of other hierarchical packet classification algorithms, our proposed hierarchical packet classification algorithm uses a Bloom filter for the source prefix field and a trie structure for the destination prefix field. The Bloom filter is primarily employed to pre-filter the sub-strings of the source address which have no match for the source prefixes of a given rule set. For the sub-strings with a positive result from the Bloom filter, rule-priority tries constructed based on a destination prefix field determine the highest priority rule matching the input packet for entire rule fields. Since the Bloom filter requires a small amount of memory, it is implemented with an on-chip memory or a fast cache, and hence the off-chip memory accesses are not occurred in the first stage of the hierarchical approach in the proposed algorithm. The proposed packet classification algorithm also provides incremental update. To compare the performance of the proposed packet classification algorithm with other related algorithms, extensive simulations for various algorithms are performed. The simulation result shows that the proposed algorithm renders a better performance in terms of average and worst-case search performance and memory requirement.

On the false-positive rate of Bloom filters

Bloom filters are a randomized data structure for membership queries dating back to 1970. Bloom filters sometimes give erroneous answers to queries, called false positives. Bloom analyzed the probability of such erroneous answers, called the false-positive rate, and Bloom's analysis has appeared in many publications throughout the years. We show that Bloom's analysis is incorrect and give a correct analysis.

Longest prefix matching using bloom filters

We introduce the first algorithm that we are aware of to employ Bloom filters for longest prefix matching (LPM). The algorithm performs parallel queries on Bloom filters, an efficient data structure for membership queries, in order to determine address prefix membership in sets of prefixes sorted by prefix length. We show that use of this algorithm for Internet Protocol (IP) routing lookups results in a search engine providing better performance and scalability than TCAM-based approaches. The key feature of our technique is that the performance, as determined by the number of dependent memory accesses per lookup, can be held constant for longer address lengths or additional unique address prefix lengths in the forwarding table given that memory resources scale linearly with the number of prefixes in the forwarding table. Our approach is equally attractive for Internet Protocol Version 6 (IPv6) which uses 128-bit destination addresses, four times longer than IPv4. We present a basic version of our approach along with optimizations leveraging previous advances in LPM algorithms. We also report results of performance simulations of our system using snapshots of IPv4 BGP tables and extend the results to IPv6. Using less than 2 Mb of embedded RAM and a commodity SRAM device, our technique achieves average performance of one hash probe per lookup and a worst case of two hash probes and one array access per lookup.