Approximate Membership Queries Research Articles

A Micro-architecture that supports the Fano–Elias encoding and a hardware accelerator for approximate membership queries

We present the first hardware design that supports operations over the Fano–Elias encoding (FE-encoding). Our design is a combinational circuit (i.e., single clock cycle) that supports insertions, deletions, and queries. FE-encoding allows one to store f binary strings, each of length ℓ+logm using a string that is m+f+fℓ bits long (rather than f(ℓ+logm)). The asymptotic gate-count of the circuit is Θ((m+f)⋅lgm+f⋅ℓ). The asymptotic delay is Θ(lgm+lgf+lgℓ). We implemented our design on an FPGA with four combinations of parameters in which the FE-encoding fits in 512 or 1024 bits.We present the first hardware design for a dynamic filter that maintains a set subject to insertions, deletions, and approximate membership queries. The design contains four main blocks: two memory banks that store FE-encodings and two combinational circuits for FE-encoding. Additional logic deals with double buffering and forwarding.We implemented the dynamic filter on an FPGA with the following parameters: (1) Elements in the dataset are 32-bit strings. (2) The supported dataset can contain up to nmax=45⋅214=737,280 elements. (3) The latency is 2-4 clock cycles. (4) Fixed (i.e., constant and stable) throughput. A new operation can be issued every clock cycle. (5) We prove that the probability of a false-positive error is bounded by 0.385⋅10−2. (6) We prove that the expected number of insertion failures is less than 1 for every 75 million insertions.Synthesis of our filter on a Xilinx Alveo U250 FPGA achieves a clock rate of 100 MHz (the critical path is due to the memory access). We measure a fixed throughput of 97.7 million operations per second (the loss of 2.3% in the throughput is due to instabilities in the bandwidth of the AXI4 Lite I/O channel).A unique feature of our filter implementation is that the throughput is stable and constant for all benchmarks and loads. Namely, the combination of operations does not influence the throughput and the throughput does not depend on the number of elements in the dataset (as long as the cardinality of the dataset is bounded by nmax). Previous dynamic filter implementations in software (implemented on x86 or GPU’s) do not exhibit stable and constant throughputs.

A Learned Cuckoo Filter for Approximate Membership Queries over Variable-sized Sliding Windows on Data Streams

Designing a space-efficient data structure to answer membership queries while ensuring high accuracy and real-time response is a challenging task in the field of stream processing. Many techniques have been developed to answer these queries in a sliding windows manner. However, assuming the user will conduct the query with the presupposed window size is not always practical. In this paper, we introduce a novel data structure called Learned Cuckoo Filter (LCF). It can provide satisfactory results for the approximate membership query on data streams, regardless of the user-defined query windows. LCF operates by adaptively maintaining cuckoo filters with the assistance of a well-trained oracle that learned the frequency feature of the data within the stream. To further enhance memory utilization, we develop a compact version of LCF (denoted by LCF_C), which selectively removes redundant information to reduce space consumption without compromising query accuracy. Furthermore, we conduct a thorough theoretical analysis of query accuracy and provide detailed guidelines for optimal parameter selection (denoted by LCF_O). Extensive experimental studies on synthetic and real-world datasets demonstrate the superiority of the proposed methods in terms of both space consumption and accuracy. Compared to the state-of-the-art algorithms, LCF_O can reduce up to 61% of space cost at the same error level, and achieve up to 12× improved accuracy with the same space cost.

Scalable sequence database search using partitioned aggregated Bloom comb trees.

The Sequence Read Archive public database has reached 45 petabytes of raw sequences and doubles its nucleotide content every 2 years. Although BLAST-like methods can routinely search for a sequence in a small collection of genomes, making searchable immense public resources accessible is beyond the reach of alignment-based strategies. In recent years, abundant literature tackled the task of finding a sequence in extensive sequence collections using k-mer-based strategies. At present, the most scalable methods are approximate membership query data structures that combine the ability to query small signatures or variants while being scalable to collections up to 10000 eukaryotic samples. Results. Here, we present PAC, a novel approximate membership query data structure for querying collections of sequence datasets. PAC index construction works in a streaming fashion without any disk footprint besides the index itself. It shows a 3-6 fold improvement in construction time compared to other compressed methods for comparable index size. A PAC query can need single random access and be performed in constant time in favorable instances. Using limited computation resources, we built PAC for very large collections. They include 32000 human RNA-seq samples in 5 days, the entire GenBank bacterial genome collection in a single day for an index size of 3.5TB. The latter is, to our knowledge, the largest sequence collection ever indexed using an approximate membership query structure. We also showed that PAC's ability to query 500000 transcript sequences in less than an hour. PAC's open-source software is available at https://github.com/Malfoy/PAC.

Hierarchical Interleaved Bloom Filter: enabling ultrafast, approximate sequence queries

We present a novel data structure for searching sequences in large databases: the Hierarchical Interleaved Bloom Filter (HIBF). It is extremely fast and space efficient, yet so general that it could serve as the underlying engine for many applications. We show that the HIBF is superior in build time, index size, and search time while achieving a comparable or better accuracy compared to other state-of-the-art tools. The HIBF builds an index up to 211 times faster, using up to 14 times less space, and can answer approximate membership queries faster by a factor of up to 129.

Fimpera: drastic improvement of Approximate Membership Query data-structures with counts.

High throughput sequencing technologies generate massive amounts of biological sequence datasets as costs fall. One of the current algorithmic challenges for exploiting these data on a global scale consists in providing efficient query engines on these petabyte-scale datasets. Most methods indexing those datasets rely on indexing words of fixed length k, called k-mers. Many applications, such as metagenomics, require the abundance of indexed k-mers as well as their simple presence or absence, but no method scales up to petabyte-scaled datasets. This deficiency is primarily because storing abundance requires explicit storage of the k-mers in order to associate them with their counts. Using counting Approximate Membership Queries (cAMQ) data structures, such as counting Bloom filters, provides a way to index large amounts of k-mers with their abundance, but at the expense of a sensible false positive rate. We propose a novel algorithm, called fimpera, that enables the improvement of any cAMQ performance. Applied to counting Bloom filters, our proposed algorithm reduces the false positive rate by two orders of magnitude and it improves the precision of the reported abundances. Alternatively, fimpera allows for the reduction of the size of a counting Bloom filter by two orders of magnitude while maintaining the same precision. fimpera does not introduce any memory overhead and may even reduces the query time. https://github.com/lrobidou/fimpera. Supplementary data are available at Bioinformatics online.

Cuckoo filter based IP packet filtering using M-tree

Internet protocol (IP) packet filtering as a firewall (FW) technology is one of the most widely researched networks functions over the past two decades. IP packet filtering is the process of filtering incoming and outgoing network packets by matching several packet headers fields with thousands of predefined filters known as filter-set. With the development of modern network technologies such as software-defined networking (SDN) and the increase in attacks threatening network security, attention has become focused on IP packet filtering. With the growing size and number of filter-sets, it becomes a challenge to perform IP packet filtering at wire-speed. In this paper, a new method is proposed for IP packet filtering, where two data structures were combined to produce a new data structure suitable for IP packet filtering with high performance and support dynamic access to filters as well as support approximate membership query. Experimental results show that the proposed method has a high throughput of 10.8 mega packets per second (MPpS) with high filtering accuracy and low memory requirements to working on big filter-sets (up to 1 mega filters).

Stretching your data with taffy filters

AbstractPopular approximate membership query structures such as Bloom filters and cuckoo filters are widely used in databases, security, and networking. These structures represent sets approximately, and support at least two operations—insert and lookup; lookup always returns true on elements in the structure; it also returns true with some probability on elements not in the structure. These latter elements are called false positives. Compensatory for these false positives, filters can be much smaller than hash tables that represent the same set. However, unlike hash tables, cuckoo filters, and Bloom filters must be initialized with the intended number of maximum entries, and cannot grow larger—inserts beyond this number fail or significantly increase the false positive probability. This article presents designs and implementations of filters than can grow without inserts failing and without meaningfully increasing the false positive probability, even if the filters are created with a small initial size. The resulting code is available on GitHub under a permissive open source license.

효율적 공간 활용을 위한 슬롯 공유 필터의 설계 및 구현

대용량 데이터를 저장하기 위해 사용되는 데이터베이스, 스토리지 시스템 등의 분야에서 특정 데이터의 존재 여부 (existence)를 효율적으로 판단하기 위해 approximate membership query (AMQ) 자료구조가 널리 사용된다. AMQ는 적은 양의 메모리만을 사용하며 빠른 접근 속도로 데이터 처리 시간을 단축한다. 대표적 AMQ 자료구조인 쿠쿠 필터 (Cuckoo Filter)는 빠른 조회와 삭제 기능을 제공해주지만, 데이터 적재율 (load factor) 이 높아질수록 연쇄적인 원소 제거 발생 빈도가 높아짐에 따라 삽입 성능이 저하되는 문제가 존재한다. 이를 해결하기 위해 본 논문에서는 버킷 간 슬롯 공유를 통해 버킷의 슬롯이 소진되는 시점을 지연시킨다. 버킷의 슬롯 상태를 관리함으로써 연쇄적인 원소 제거 발생 비율을 최소화하여 이로 인한 삽입 성능 저하를 방지한다. 또한 슬롯을 공유하여 사용함으로써 첫 번째 버킷에서 특정 원소의 조회 성공 비율을 높일 수 있게 된다. 실험을 통해 제안하는 슬롯 공유 기법이 기존의 쿠쿠 필터 대비 데이터 적재율이 50% 이상인 상황에서 첫 번째 버킷으로의 데이터 삽입 성공율을 10% 이상 향상시키며, 연쇄적인 원소 제거 횟수를 5배 이상 감소시킴을 확인하였다. 또한, 첫 번째 버킷에서의 조회 성공 비율을 최대 17% 높일 수 있음을 확인하였다.

A Multi-hashing Index for hybrid DRAM-NVM memory systems

Hybrid memory systems composed of DRAM and Non-Volatile Memory (NVM) promise the capacity benefits of NVM and the low-latency benefits of DRAM. Most existing hash-based indexes are designed for NVM only and do not exploit the benefits of DRAM. In this paper, we proposed a novel hybrid DRAM-NVM persistent and concurrent hashing index, named Multi-Hashing Index (MuHash). MuHash uses a multi-hash function scheme to solve the cascading write problem of open-addressed hash-based indexes in NVM. It employs a Cuckoo Filter, an approximate membership query data structure, to prune unnecessary NVM accesses for improving read performance. To maximize throughput in multi-thread environments, MuHash also includes a fine-grained concurrency control mechanism. We implemented MuHash for Intel Optane DC Persist Memory, and single-core experiments shows that MuHash achieves up to 90% higher read throughput compared to state-of-the-art hash-based indexes. On multicore experiments, MuHash achieves near-linear scalability for all operations.

Prefix filter

Many applications of approximate membership query data structures, or filters , require only an incremental filter that supports insertions but not deletions. However, the design space of incremental filters is missing a "sweet spot" filter that combines space efficiency, fast queries, and fast insertions. Incremental filters, such as the Bloom and blocked Bloom filter, are not space efficient. Dynamic filters (i.e., supporting deletions), such as the cuckoo or vector quotient filter, are space efficient but do not exhibit consistently fast insertions and queries. In this paper, we propose the prefix filter , an incremental filter that addresses the above challenge: (1) its space (in bits) is similar to state-of-the-art dynamic filters; (2) query throughput is high and is comparable to that of the cuckoo filter; and (3) insert throughput is high with overall build times faster than those of the vector quotient filter and cuckoo filter by 1.39X--1.46X and 3.2X--3.5X, respectively. We present a rigorous analysis of the prefix filter that holds also for practical set sizes (i.e., n = 2 25 ). The analysis deals with the probability of failure, false positive rate, and probability that an operation requires accessing more than a single cache line.

Towards Capacity-Adjustable and Scalable Quotient Filter Design for Packet Classification in Software-Defined Networks

Software defined networking (SDN), which can provide a dynamic and configurable network architecture for resource allocation, have been widely employed for efficient massive data traffic management. To accelerate the packet classification process in SDN, the hash-based filters which can support fast approximate membership query have been widely employed. However, the existing Quotient Filters are limited to fixed size and the number of elements has to be provided in advance. Thus, in this paper, we investigate the first capacity adjustable and scalable quotient filter for dynamic packet classification in SDN. Firstly, a novel Index Independent Quotient Filter (IIQF) is designed, which can adjust its capacity in a more precise level to support dynamic set representation. The algorithms for the operations of insertion, querying, deletion and capacity adjustment of IIQF are also given. Secondly, on the basis of IIQF, a Scalable Index Independent Quotient Filter (SIIQF) is designed to ensure the consistency of the designed quotient filter when adjusting its size. The theoretical performance of the proposed SIIQF, including the error rate, probability of collisions, and the time and space complexity are all analyzed. An instance of employing SIIQF for packet classification with tuple space searching algorithm is also introduced. Finally, the extensive simulations demonstrate the performance gains achieved by the proposed SIIQF compared with the baseline methods.

Xor Filters

The Bloom filter provides fast approximate set membership while using little memory. Engineers often use these filters to avoid slow operations such as disk or network accesses. As an alternative, a cuckoo filter may need less space than a Bloom filter and it is faster. Chazelle et al. proposed a generalization of the Bloom filter called the Bloomier filter. Dietzfelbinger and Pagh described a variation on the Bloomier filter that can answer approximate membership queries over immutable sets. It has never been tested empirically, to our knowledge. We review an efficient implementation of their approach, which we call the xor filter. We find that xor filters can be faster than Bloom and cuckoo filters while using less memory. We further show that a more compact version of xor filters (xor+) can use even less space than highly compact alternatives (e.g., Golomb-compressed sequences) while providing speeds competitive with Bloom filters.

Vacuum filters

We present vacuum filters, a type of data structures to support approximate membership queries. Vacuum filters cost the smallest space among all known AMQ data structures and provide higher insertion and lookup throughput in most situations. Hence they can be used as the replacement of the widely used Bloom filters and cuckoo filters. Similar to cuckoo filters, vacuum filters also store item fingerprints in a table. The memory-efficiency and throughput improvements are from the innovation of a table insertion and fingerprint eviction strategy that achieves both high load factor and data locality without any restriction of the table size. In addition, we propose a new update framework to resolve two difficult problems for AMQ structures under dynamics, namely duplicate insertions and set resizing. The experiments show that vacuum filters can achieve 25% less space in average and similar throughput compared to cuckoo filters, and 15% less space and >10x throughput compared to Bloom filters, with same false positive rates. AMQ data structures are widely used in various layers of computer systems and networks and are usually hosted in platforms where memory is limited and precious. Hence the improvements brought by vacuum filters can be considered significant.

Sketching and Sublinear Data Structures in Genomics

Large-scale genomics demands computational methods that scale sublinearly with the growth of data. We review several data structures and sketching techniques that have been used in genomic analysis methods. Specifically, we focus on four key ideas that take different approaches to achieve sublinear space usage and processing time: compressed full-text indices, approximate membership query data structures, locality-sensitive hashing, and minimizers schemes. We describe these techniques at a high level and give several representative applications of each.

QCF for deep packet inspection

Deep packet inspection (DPI) acts as a tool to control and classify incoming network traffic depending on users,content, applications and becomes a very important aspect of every network today. Instead of utilising only incoming packet header information, Internet service providers utilise DPI for security purposes, flow management, and routing. The high-performance challenges that face DPI techniques to manage line-speed packet processing with restricted available memory are the increasing traffic volumes and link rates. Bloom (BF) and quotient (QF) filters are effective randomised data structures for performing approximate membership queries. They have been employed in many networking applications including DPI. Nevertheless, they consume more calculations which degrade the system performance. Consequently, in this paper, the authors propose a new variant of membership query data structure, called quotient-based Cuckoo filter (QCF), that reflects a merging process between QF and CF to minimise the calculation overhead and employing it in a DPI system. QCF employs only two hash functions and consumes less calculation overhead than BF, QF, and CF. The implementation results show that using QCF as an identification tool in a DPI system result in a time improvement up to 77% over CF and up to 98% over BF and QF.

Squeakr: an exact and approximate k-mer counting system.

k-mer-based algorithms have become increasingly popular in the processing of high-throughput sequencing data. These algorithms span the gamut of the analysis pipeline from k-mer counting (e.g. for estimating assembly parameters), to error correction, genome and transcriptome assembly, and even transcript quantification. Yet, these tasks often use very different k-mer representations and data structures. In this article, we show how to build a k-mer-counting and multiset-representation system using the counting quotient filter, a feature-rich approximate membership query data structure. We introduce the k-mer-counting/querying system Squeakr (Simple Quotient filter-based Exact and Approximate Kmer Representation), which is based on the counting quotient filter. This off-the-shelf data structure turns out to be an efficient (approximate or exact) representation for sets or multisets of k-mers. Squeakr takes 2×-4.3× less time than the state-of-the-art to count and perform a random-point-query workload. Squeakr is memory-efficient, consuming 1.5×-4.3× less memory than the state-of-the-art. It offers competitive counting performance. In fact, it is faster for larger k-mers, and answers point queries (i.e. queries for the abundance of a particular k-mer) over an order-of-magnitude faster than other systems. The Squeakr representation of the k-mer multiset turns out to be immediately useful for downstream processing (e.g. de Bruijn graph traversal) because it supports fast queries and dynamic k-mer insertion, deletion, and modification. https://github.com/splatlab/squeakr available under BSD 3-Clause License. ppandey@cs.stonybrook.edu. Supplementary data are available at Bioinformatics online.

Multi-Granularity Locality-Sensitive Bloom Filter

In many applications, such as homeland security, image processing, social network, and bioinformatics, it is often required to support an approximate membership query (AMQ) to answer a question like “is an (query) object q near to at least one of the objects in the given data set $\Omega$ ?” However, existing techniques for processing AMQs require a key parameter, i.e., the distance value, to be defined in advance for the query processing. In this paper, we propose a novel filter, called multi-granularity locality-sensitive Bloom filter (MLBF), which can process AMQs with multiple distance granularities. Specifically, the MLBF is composed of two Bloom filters (BF), one is called basic multi-granularity locality-sensitive BF (BMLBF), and the other is called multi-granularity verification BF (MVBF). The BMLBF is used to store the data objects. It adopts an alignable locality-sensitive hashing (LSH) function family to support multiple granularities. The MVBF is used to reduce the false positive rate of the MLBF. The false negative rate of the MLBF is reduced by applying AND-constructions followed by an OR-construction. In addition, based on the MLBF structure, we suggest a more space-effective variant, called the MLBF*, to further reduce space cost. Theoretical analyses for estimating false positive/negative rates of the MLBF/MLBF* are given. Experiments using synthetic and real data show that the theoretical estimates are quite accurate, and the MLBF/MLBF* technique can handle AMQs with low false positive and negative rates for multiple distance granularities.

Locality-Sensitive Bloom Filter for Approximate Membership Query

In many network applications, Bloom filters are used to support exact-matching membership query for their randomized space-efficient data structure with a small probability of false answers. In this paper, we extend the standard Bloom filter to Locality-Sensitive Bloom Filter (LSBF) to provide Approximate Membership Query (AMQ) service. We achieve this by replacing uniform and independent hash functions with locality-sensitive hash functions. Such replacement makes the storage in LSBF to be locality sensitive. Meanwhile, LSBF is space efficient and query responsive by employing the Bloom filter design. In the design of the LSBF structure, we propose a bit vector to reduce False Positives (FP). The bit vector can verify multiple attributes belonging to one member. We also use an active overflowed scheme to significantly decrease False Negatives (FN). Rigorous theoretical analysis (e.g., on FP, FN, and space overhead) shows that the design of LSBF is space compact and can provide accurate response to approximate membership queries. We have implemented LSBF in a real distributed system to perform extensive experiments using real-world traces. Experimental results show that LSBF, compared with a baseline approach and other state-of-the-art work in the literature (SmartStore and LSB-tree), takes less time to respond AMQ and consumes much less storage space.

False Negative Problem of Counting Bloom Filter

Bloom filter is effective, space-efficient data structure for concisely representing a data set and supporting approximate membership queries. Traditionally, researchers often believe that it is possible that a Bloom filter returns a false positive, but it will never return a false negative under well-behaved operations. By investigating the mainstream variants, however, we observe that a Bloom filter does return false negatives in many scenarios. In this work, we show that the undetectable incorrect deletion of false positive items and detectable incorrect deletion of multiaddress items are two general causes of false negative in a Bloom filter. We then measure the potential and exposed false negatives theoretically and practically. Inspired by the fact that the potential false negatives are usually not fully exposed, we propose a novel Bloom filter scheme, which increases the ratio of bits set to a value larger than one without decreasing the ratio of bits set to zero. Mathematical analysis and comprehensive experiments show that this design can reduce the number of exposed false negatives as well as decrease the likelihood of false positives. To the best of our knowledge, this is the first work dealing with both the false positive and false negative problems of Bloom filter systematically when supporting standard usages of item insertion, query, and deletion operations.

The Dynamic Bloom Filters

A Bloom filter is an effective, space-efficient data structure for concisely representing a set, and supporting approximate membership queries. Traditionally, the Bloom filter and its variants just focus on how to represent a static set and decrease the false positive probability to a sufficiently low level. By investigating mainstream applications based on the Bloom filter, we reveal that dynamic data sets are more common and important than static sets. However, existing variants of the Bloom filter cannot support dynamic data sets well. To address this issue, we propose dynamic Bloom filters to represent dynamic sets, as well as static sets and design necessary item insertion, membership query, item deletion, and filter union algorithms. The dynamic Bloom filter can control the false positive probability at a low level by expanding its capacity as the set cardinality increases. Through comprehensive mathematical analysis, we show that the dynamic Bloom filter uses less expected memory than the Bloom filter when representing dynamic sets with an upper bound on set cardinality, and also that the dynamic Bloom filter is more stable than the Bloom filter due to infrequent reconstruction when addressing dynamic sets without an upper bound on set cardinality. Moreover, the analysis results hold in stand-alone applications, as well as distributed applications.