Traditional Bloom Filters Research Articles

Blockchain Traceability Process for Hairy Crab Based on Cuckoo Filter

With the rapid development of global information technology, smart agriculture has gradually become an important force to promote agricultural modernisation. Taking Yangcheng Lake hairy crabs as the research object, this paper constructs a full-chain traceability process for hairy crabs integrating blockchain and Internet of Things technology in response to the demand for the enhancement of its brand value and market competitiveness. The model covers all aspects of breeding, processing, and transport to sales; realises end-to-end transparent regulation; effectively guarantees the authenticity and non-tamperability of information; and enhances consumers’ trust in the brand. At the technical level, this paper designs the ECMI (Enhanced Cuckoo Merkle Index) model, optimises the data storage and retrieval mechanism, and experimentally verifies the advantages of cuckoo filters over traditional Bloom filters in reducing false positives and improving query efficiency. This paper not only provides an efficient and safe traceability solution for the hairy crab industry in Yangcheng Lake but also provides technical support and practical cases for the further application of blockchain technology in smart agriculture, which is of great value for the realisation of the national rural revitalisation strategy.

Two-layer partitioned and deletable deep bloom filter for large-scale membership query

The recently proposed Learned Bloom Filter (LBF) provides a new perspective on large-scale membership queries by using machine learning to replace the traditional bloom filter. However, reducing the false positive rate (FPR) of the learned model with small memory usage, and supporting deletion efficiently become the new issues. In this paper, we propose a novel Two-layer Partitioned and Deletable Deep Bloom Filter (PDDBF) for large-scale membership query, which can reduce the FPR with small memory usage and support deletion efficiently. The proposed PDDBF consists of three main parts: (1) Data partition. To improve the classification accuracy of the learned model, the K-means cluster with the elbow method is used for the data partition. (2) Deep Bloom Filter. To reduce the FPR, deep learning models are used to construct multiple independent learning mechanisms, which correspond to the clusters obtained by part1. (3) Partitioned backup filter. To support deletion under the premise of ensuring low FPR and reducing query time consumption, combine the perfect hash (PH) table and counting bloom filters (CBFs) on the basis of the partition bloom filter. Experiments show that the proposed PDDBF reduces the FPR 87.13% with the same memory usage compared with the state-of-the-art PLBF on real-world URLs data set. Moreover, the PDDBF reduces the FPR 99.68% with the same memory usage and reduces the query time consumption to 2.61x that of the PLBF after data deletion, respectively.

Search mechanism for data contents based on bloom filter and tree hybrid structure in system wide information management

AbstractThe system wide information management (SWIM) system infrastructure layer uses information‐centric networking (ICN) to implement the cache routing and sharing of air traffic information data. The SWIM network searches and forwards routes on the basis of content names. The variable length, hierarchical name structure, and routing updates caused by the frequent publication and deletion of content make the implementation of fast name routing lookup algorithms a significant but arduous task. To address this problem, this study designed a name matching mechanism on the basis of the hybrid structure of the Bloom filter and the Tree. Compared with the traditional Bloom filter search scheme, this search mechanism reduces the quantity of Bloom filter insertion entries and reduces the possibility of hash collisions. Compared with the traditional Tree search scheme, the proposed mechanism can effectively solve the problem of numerous memory accesses caused by extremely long name prefixes, improving the query efficiency. In addition, the proposed composite structure is compared with several methods, such as DIPIT and NPT, by analyzing the effects of layering, global delay and search speed. The experimental results show that the proposed structure has a favourable performance in terms of storage overhead and search speed.

Optimizing Learned Bloom Filters: How Much Should Be Learned?

The learned Bloom filter (LBF) combines a machine learning model (learner) with a traditional Bloom filter to improve the false positive rate (FPR) that can be achieved for a given memory budget. The LBF has recently been generalized by making use of the full spectrum of the learner’s prediction score. However, in all those designs, the machine learning model is fixed. In this letter, for the first time, the design of LBFs is proposed and evaluated by considering the machine learning model as one of the variables in the process. In detail, for a given memory budget, several LBFs are constructed using different machine learning models and the one with the lowest FPR is selected. We demonstrate that our approach can achieve much better performance than existing LBF designs providing reductions of the FPR of up to 90% in some settings.

DH-SVRF: A Reconfigurable Unicast/Multicast Forwarding for High-Performance Packet Forwarding Engines

High-performance multicast-enabled packet forwarding engines (PFEs), as an essential component of high-end switches, use a polynomial-time membership query algorithm to determine which port(s) the data packet should be forwarded. The currently widely used query algorithm is Bloom Filter (BF), which has been proven to have many fatal flaws. Another error-free membership query algorithm includes Scalar-pair Vectors Routing Forwarding (SVRF), Fractional- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> Scalar-pair Vectors Routing Forwarding (Frac- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> SVRF), and the Per-Port Prime Filter Array (P <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> FA) also have some shortcomings in space and time efficiencies. In this paper, we proposed a hybrid strategy: Divaricate Heterogeneous SVRF (DH-SVRF) scheme, which based on the P <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> FA and Frac- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> SVRF, which randomly divides all member ships into <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> groups, and each group has the same structure and is independent of each other to obtain higher time efficiency and space utilization. Finally, we also discussed the selection of the optimal egress-diversity threshold. Through mathematical modeling and simulation, we validate that the proposed DH-SVRF scheme is superior to the SVRF/Frac <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-N</i> SVRF and traditional BF in terms of scalability, space utilization, and time efficiency in specific conditions such as appropriate egress-diversity thresholds.

The Impact of Check Bits on the Performance of Bloom Filter

Bloom filter (BF) is a space-and-time efficient probabilistic technique that helps answer membership queries. However, BF faces several issues. The problems with traditional BF are generally two. Firstly, a large number of false positives can return wrong content when the data is queried. Secondly, the large size of BF is a bottleneck in the speed of querying and thus uses large memory. In order to solve the above two issues, in this article, we propose the check bits concept. From the implementation perspective, in the check bits approach, before saving the content value in the BF, we obtain the binary representation of the content value. Then, we take some bits of the content value, we call these the check bits. These bits are stored in a separate array such that they point to the same location as the BF. Finally, the content value (data) is stored in the BF based on the hash function values. Before retrieval of data from BF, the reverse process of the steps ensures that even if the same hash functions output has been generated for the content, the check bits make sure that the retrieval does not depend on the hash output alone. This thus helps in the reduction of false positives. In the experimental evaluation, we are able to reduce more than 50% of false positives. In our proposed approach, the false positives can still occur, however, false positives can only occur if the hash functions and check bits generate the same value for a particular content. The chances of such scenarios are less, therefore, we get a reduction of approximately more than 50% false positives in all cases. We believe that the proposed approach adds to the state of the art and opens new directions as such.

Succinct range filters

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries, such as range counts. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the performance of state-of-the-art order-preserving indexes, while consuming only 10 bits per trie node---a space close to the minimum required by information theory. Our experiments show that SuRF speeds up range queries in a widely used database storage engine by up to 5×.

Bloom Filter With a False Positive Free Zone

Bloom filters and their variants are widely used as space-efficient probabilistic data structures for representing sets and are very popular in networking applications. They support fast element insertion and deletion, along with membership queries with the drawback of false positives. Bloom filters can be designed to match the false positive rates that are acceptable for the application domain. However, in many applications, a common engineering solution is to set the false positive rate very small and ignore the existence of the very unlikely false positive answers. This article is devoted to close the gap between the two design concepts of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unlikely</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">not having</i> false positives. We propose a data structure called EGH filter that supports the Bloom filter operations, and besides, it can guarantee false positive free operations for a finite universe and a restricted number of elements stored in the filter. We refer to the limited universe and filter size as the false positive free zone of the filter. We describe necessary conditions for the false-positive free zone of a filter. We then generalize the filter to support the listing of the elements through the use of counters rather than bits. We detail networking applications of the filter and discuss potential generalizations. We evaluate the performance of the filter in comparison with the traditional Bloom filters. We also evaluate the price in terms of memory that needs to be paid to guarantee real false positive-free operations for having a deterministic Bloom filter-like behavior. Our data structure is based on recently developed combinatorial group testing techniques.

Succinct Range Filters

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries: open-range queries, closed-range queries, and range counts. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the point and range query performance of state-of-the-art order-preserving indexes, while consuming only 10 bits per trie node. The false-positive rates in SuRF for both point and range queries are tunable to satisfy different application needs. We evaluate SuRF in RocksDB as a replacement for its Bloom filters to reduce I/O by filtering requests before they access on-disk data structures. Our experiments on a 100-GB dataset show that replacing RocksDB’s Bloom filters with SuRFs speeds up open-seek (without upper-bound) and closed-seek (with upper-bound) queries by up to 1.5× and 5× with a modest cost on the worst-case (all-missing) point query throughput due to slightly higher false-positive rate.

Succinct Range Filters

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the point and range query performance of state-of-the-art order-preserving indexes, while consuming only 10 bits per trie node. The false positive rates in SuRF for both point and range queries are tunable to satisfy different application needs. We evaluate SuRF in RocksDB as a replacement for its Bloom filters to reduce I/O by filtering requests before they access on-disk data structures. Our experiments on a 100 GB dataset show that replacing RocksDB's Bloom filters with SuRFs speeds up open-seek (without upper-bound) and closed-seek (with upper-bound) queries by up to 1.5× and 5× with a modest cost on the worst-case (all-missing) point query throughput due to slightly higher false positive rate.

A neural data structure for novelty detection

Novelty detection is a fundamental biological problem that organisms must solve to determine whether a given stimulus departs from those previously experienced. In computer science, this problem is solved efficiently using a data structure called a Bloom filter. We found that the fruit fly olfactory circuit evolved a variant of a Bloom filter to assess the novelty of odors. Compared with a traditional Bloom filter, the fly adjusts novelty responses based on two additional features: the similarity of an odor to previously experienced odors and the time elapsed since the odor was last experienced. We elaborate and validate a framework to predict novelty responses of fruit flies to given pairs of odors. We also translate insights from the fly circuit to develop a class of distance- and time-sensitive Bloom filters that outperform prior filters when evaluated on several biological and computational datasets. Overall, our work illuminates the algorithmic basis of an important neurobiological problem and offers strategies for novelty detection in computational systems.

A Bloom Filter for High Dimensional Vectors

Regardless of the type of data, traditional Bloom filters treat each element of a set as a string, and by iterating every character of the string, they discretize all data randomly and uniformly. However, with the data size and dimension increases, these variants are inefficient. To better discretize vectors with high numerical dimensions, this paper improves the string hashes to integer hashes. Based on the integer hashes and a counter array, we propose a new variant—high-dimensional bloom filter (HDBF)—to extend the Bloom filter into high-dimensional spaces, which can represent and query numerical vectors of a big set with a low false positive probability. This paper theoretically analyzes the feasibility of the integer hashes on discretizing data and discusses the relationship of parameters of the HDBF. The experiments illustrate that, in high-dimensional numerical spaces, the HDBF shows better randomness on distribution and entropy than that of the counting Bloom filter. Compared with the parallel Bloom filters, for a fixed false positive probability, the HDBF displays time-space overheads, and is more suitable to deal with the numerical vectors with high dimensions.

Dynamically Updatable Ternary Segmented Aging Bloom Filter for OpenFlow-Compliant Low-Power Packet Processing

OpenFlow, the main protocol for software-defined networking, requires large-sized rule tables and frequent updating. For fast packet processing, rule tables are often implemented with ternary content-addressable memory (TCAM) in the OpenFlow. To deal with TCAM power problems, many network applications employ bloom filters (BFs) to reduce the redundant operations of table-lookup and for low power consumption. However, applying traditional BFs to an OpenFlow switch leads to problems, such as unsupported dynamic update, large space overhead, and the rule-set expansion of ternary data. In this paper, we propose a dynamically updatable ternary segmented aging bloom filter (TSA-BF). The TSA-BF consists of two parts: a segmented aging BF algorithm (SA-BF) and a ternary prefix-tagging encoder (TPE). First, in the SA-BF, we develop an automatic update scheme using the mechanisms of content-aging and buffer-segmenting. The SA-BF ages and deletes its contents automatically, thus eliminating the costly communication overhead and enabling dynamic updating. It also achieves space efficiency by the developed partial-deletion mechanism. Second, in the TPE, we encode ternary prefix-rules into uniquely decodable binary code words. The TPE prevents the rule-set expansion of ternary-data in the OpenFlow environment. Simulation results show that the SA-BF alone can save 37% of space overhead, compared with state-of-the-art techniques. In an environment with the ternary prefix-rules, the TSA-BF can save another 93% of space overhead, compared with the best-performance scheme. Hence, the proposed TSA-BF is highly suited to the requirements of emerging high-performance TCAM-based packet processing in the OpenFlow, which considers dynamic update and power efficiency.

A distributed incremental information acquisition model for large-scale text data

Timely discovering and acquiring information from incremental data on the Internet is a hot topic in a big data era. This paper presents a distributed incremental information acquisition model for large-scale text data. To obtain a lower false positive rate and higher efficiency of the traditional Bloom filter, a distributed multidimensional Bloom filter is designed and proposed to cope with the deduplication of large-scale Web URL text data. Three methods related to Bloom filter were compared based on the false positive rate and response efficiency. The results show that the distributed incremental information acquisition model for large-scale text data can achieve a high duplicate removal rate with a lower false positive rate.

Idempotent distributed counters using a forgetful bloom filter

Distributed key-value stores power the backend of high-performance web services and cloud computing applications. Key-value stores such as Cassandra rely heavily on counters to track occurrences of various kinds of events. However, modern implementations of counters do not provide exactly-once semantics. E.g., a client may request a counter increment, time out waiting for a response, and create a duplicate request resulting in a double increment at the server. In this paper, we address this problem by presenting, analyzing, and evaluating a novel server-side data structure called the forgetful bloom filter (FBF). Like a traditional Bloom filter, an FBF is a compact representation of a set of elements (e.g., client requests). However, an FBF: (i) can forget older elements (e.g., requests that are too old to apply), and (ii) adapts itself to meet a desired false positive rate under a varying workload. We present experimental results from a prototype implementation of FBFs and an implementation of FBFs in the Cassandra key-value store. Our results show that the FBF is highly accurate in maintaining correct counter values.

Implementing error detection in fast counting Bloom filters

Bloom filters have found numerous applications in computing and networking systems. They are used to determine whether a given element is present in a set. Counting Bloom filters (CBFs) are an extension of Bloom filters that supports the removal of elements from the set. Traditional Bloom filters require several memory accesses to determine whether an element is present in the set. Recently, fast CBFs that can complete a search operation with only one memory access have been presented. Modern electronic systems are prone to soft errors. These errors can corrupt the contents of memories, causing system failures. In the case of Bloom filters, errors can cause failures where an element that is in the set is classified as not being in the set and the other way around. To avoid those failures, a per-word parity bit is commonly added to detect errors in memories. It is shown that error detection can be implemented in fast CBFs without adding any parity bit. This is achieved by exploiting the properties of the filters to implement error detection.

Improving the performance of Invertible Bloom Lookup Tables

Invertible Bloom Lookup Tables (IBLTs) have been recently introduced as an extension of traditional Bloom filters. IBLTs store key-value pairs. Unlike traditional Bloom filters, IBLTs support both a lookup operation (given a key, return a value) and an operation that lists out all the key-value pairs stored. One issue with IBLTs is that there is a probability that a lookup operation will return “not found” for a key. In this paper, a technique to reduce this probability without affecting the storage requirement and only moderately increasing the search time is presented and evaluated. The results show that it can significantly reduce the probability of not returning a value that is actually stored in the IBLT. The overhead of the modified search procedure, compared to the standard IBLT search procedure, is small and has little impact on the average search time.

Impact of Bloom Filter on Infection Rate in Epidemic Forwarding for ICNs

Mobile ad hoc routing protocols fails in intermittently connected networks (ICN) (i.e. characterized by short-range communication and absence of connected path from source to destination). However, Epidemic routing techniques ensures eventual message delivery from source to destination even where there is never a connected path or when a network partition exists at the origin of message. Epidemic Routing uses random pair-wise messages exchanges between nodes with goals to maximize message delivery rate, minimize message latency, and the total resources consumed in message delivery. Epidemic routing uses summary vector to avoid useless transmission and redundancy. Further,to make summery vector efficient, epidemic routing can use bloom filter to significantly reduce the useless transmissions associated with the summary vector. However, the challenge for epidemic routing remains opens is to optimal design of summary vector size for finite buffer while keeping the benefits of infinite buffer space. This paper proposes an improved scheme of Bloom filter (named it modified bloom filter MBLF), which is tailored according to epidemic routing. We performed simulation to support our clam and observed that delivery ratio of MBLF with epidemic routing is 19 % higher then the traditional bloom filter. In this paper, we have proposed a bloom filter based epidemic forwarding for ICNs.

Implementation of Improved Hash and Mapping Modified Low Power Parallel Bloom Filter Design

In this article, the authors present an investigation on bloom filters and introduce a new improved variant, which uses a secure modified hash function and suggested improved mapping scheme with an efficient parallel architecture. This novel architecture provides efficient, relatively fast membership querying and compact information representation with negligible false positive. This is relatively a low power and secure design with very less false positive ratio when compared with the traditional bloom filters. The design has been evaluated and tested using Xilinx 65 nm Virtex-5 field-programmable gate array as the target technology. The performance matrices are false positive ratio, power, speed and compactness.

The Research of Technology of Bloom Filter Realization in Hardware

Bloom filter is a space-efficient data with a certain probability of false positive . We present a reusable hardware implementation framework, define a module interface to provide users with a customize module, and introduce the constraints of hardware resources in the analysis of false positive rate against the traditional Bloom filter hardware design and analysis of the Bloom filter false positives. Finally, we make verification and analysis of our design combined with the the NetMagic platform.