Dynamic Bloom Filter Research Articles

PSBF: p-adic Integer Scalable Bloom Filter.

Given the challenges associated with the dynamic expansion of the conventional bloom filter's capacity, the prevalence of false positives, and the subpar access performance, this study employs the algebraic and topological characteristics of p-adic integers to introduce an innovative approach for dynamically expanding the p-adic Integer Scalable Bloom Filter (PSBF). The proposed method involves converting the target element into an integer using a string hash function, followed by the conversion of said integer into a p-adic integer through algebraic properties. This process automatically establishes the topological tree access structure of the PSBF. The experiment involved a comparison of access performance among the standard bloom filter, dynamic bloom filter, and scalable bloom filter. The findings indicate that the PSBF offers advantages such as avoidance of a linear storage structure, enhanced efficiency in element insertion and query, improved storage space utilization, and reduced likelihood of false positives. Consequently, the PSBF presents a novel approach to the dynamic extensibility of bloom filters.

An improved data integrity validation model for cloud storage

Data integrity validation is considered to be an important tool to solve the problem that cloud subscribers cannot accurately know whether there are non-subjective changes in the data they upload to cloud servers. In this paper, a data integrity verification model based on dynamic successor tree index structure, Bloom filter and Merkle tree is proposed. The block labels generated according to the features of the dynamic successor tree index structure can sense whether changes have been made to the user's data, while the Merkle tree can track the cha*nged data blocks, enabling the user to effectively verify the integrity of the data stored in the cloud server and provide more effective protection for data.

Par-BF: A parallel partitioned Bloom filter for dynamic data sets

Compared with a hash table, a Bloom filter (BF) is more space efficient for supporting fast matching through a controllable and acceptable false positive probability. The space size of the basic BF is predetermined based on the expected number of elements to be stored. However, we cannot predict the space scale of a BF for dynamic sets. It is still challenging for the two existing solutions, scalable BF (SBF) and dynamic BF (DBF), to manipulate dynamic data sets with low memory overhead but achieving high performance. This article presents a partitioned BF (Par-BF) for dynamic data sets. Compared with DBF and SBF, Par-BF is able to leverage a sweet spot between high performance and low overhead by a group of formulas to support fast concurrent matching. Specifically, the size and the range of the false positive probability in Par-BF can be deliberately derived. From our trace-driven experimental results, the input/output operations per second of Par-BF outperforms that of DBF and SBF by 10× to 14× and by 3× to 8×, respectively. Besides, through our proposed garbage collection policy, Par-BF consumes less than half of the memory usage of SBF.

Dynamic Wear Leveling for Phase-Change Memories With Endurance Variations

Phase change memory (PCM) has a write endurance problem. This problem is exacerbated due to endurance variations (EVs) when using advanced process technology (e.g., sub-20 nm), where PCM is expected to provide scaling benefits over dynamic random access memory (RAM). Wear leveling can solve this problem by dynamically changing the mapping from memory addresses to PCM physical addresses such that all PCM cells are evenly written, thereby extending the effective lifetime of such devices. PCM permits fine-grained writes, i.e., even bit level updates are allowed. To allow fine-grained wear leveling, this capability must be exploited. However, previous wear leveling approaches do not fully exploit fine-grained writes since fine-grained writes cause them to suffer from high data copy (called swap) overhead for address remapping, and/or high area and runtime overhead for the management of write frequency and address mapping information. This paper proposes a dynamic wear leveling method for PCMs that addresses all of these issues. The method: 1) uses bloom filters to enable low-cost write counters for fine-grained writes and 2) exploits the EV of PCM cells to avoid mapping hot data onto weak cells. To improve the effectiveness of the bloom filters, dynamic bloom filter management (write counts, hash functions, and write counter thresholds) and hot-cold address lists are used. The proposed method was evaluated using simulations and a hardware implementation. Using a small amount of PCM capacity overhead (0.3%), the proposed method extended the lifetime of a PCM device by 2.8–4.6 times over the existing methods when there were significant EVs.

Security Enhancement in Distributed Networks Using Link-Based Mapping Scheme for Network Intrusion Detection with Enhanced Bloom Filter

To prevent and monitor the unauthorized usage of data access, security on the network is implemented by authenticating the data. Network intrusion detection system monitors the network traffic and measures the information to identify the suspicious activities. In distributed networks, the network administrator has to authorize the user data access. When large data set is concerned in network applications the two complex issues to be solved are the organization of information and decision making. To address these issues, a space efficient data structure, called the bloom filter is used which effectively organizes and decides the presence of reliability. However, using advanced filtering techniques, the intruders easily hack the authorized data for unauthorized operations. At the same time, when processing the information, it is difficult to access the data in a secured manner using the standard bloom filters. To enhance the security over the user data access from the intruders, an enhanced bloom filter technique is presented to represent the large set of data in secure manner applied in distributed applications like web caching, peer networks etc. Additionally, to restrict the unauthorized access over the dataset from malicious activities by intruders, the enhanced bloom filter is applied with an upper bound on the false-positive probability by increasing its capacity as the packet data size increases. The occurrence of network data traffic is cleared by mapping the set of data elements to the appropriate setting in the database using hash functions, minimizing the number of resets created and at the same time improving the mean hit ratio. An experimental evaluation is done with the KDD cup 1999 dataset extracted from UCI repository to estimate the performance of the proposed link-based mapping for network intrusion detection system with enhanced bloom filters. Performance evaluation is measured in terms of false positive probability, false negative probability, mean hit ratio, scalability, number of resets created and security. The experimental results reveals that security over the packet data achieves 42.5 % higher against existing dynamic bloom filter approach.

Identifying Large Flows Based on SDBF Algorithm in High Speed Network Measurement

It is very important in developing effective and efficient traffic engineering schemes for identifying large flows. In this paper, according to the characteristic of network flows, a new algorithm which combined by scalable bloom filter and dynamic bloom filter is proposed to realize elephant flows identification. This method can improve the performance of network measurement and the efficiency of the resource usage of measurement system by handling flows with different strategy according to its different feature. The theoretical analysis and the simulation result indicate that under the condition of existing some tolerable measurement error about the length of flows, our algorithm can identify elephant flows accurately, which reduce the storage space and improve the processing speed efficiently.

Efficiently Representing Membershipfor Variable Large Data Sets

Cloud computing has raised new challenges for the membership representation scheme of storage systems that manage very large data sets. This paper proposes DBA, a dynamic Bloom filter array aimed at representing membership for variable large data sets in storage systems in a scalable way. DBA consists of dynamically created groups of space-efficient Bloom filters (BFs) to accommodate changes in set sizes. Within a group, BFs are homogeneous and the data layout is optimized at the bit level to enable parallel access and thus achieve high query performance. DBA can effectively control its query accuracy by partially adjusting the error rate of the constructing BFs, where each BF only represents an independent subset to help locate elements and confirm membership. Further, DBA supports element deletion by introducing a lazy update policy. We prototype and evaluate our DBA scheme as a scalable fast index in the MAD2 deduplication storage system. Experimental results reveal that DBA (with 64 BFs per group) shows significantly higher query performance than the state-of-the-art approach while scaling up to 160 BFs. DBA is also shown to excel in scalability, query accuracy, and space efficiency by theoretical analysis and experimental evaluation.

The DBF-based semantic service discovery for mobile ad hoc networks

Service discovery is a significant area of research in mobile ad hoc networks, where mobile nodes are allowed to exchange services and utilize each other's services. Furthermore, in ad hoc environments, the service discovery process is enhanced by making use of semantic descriptions of services. Semantic-based user requests are used to carry out discovery process to satisfy the requester's service requirements. In this paper, we exploit the OWL-DL to describe services and the dynamic bloom filter (DBF) to improve the discovery process. The integration of the DBF and distributed service directories, and the utilization of the hybrid service matchmaking approach provide an adaptive, flexible and efficient discovery of services. Simulations were carried out to evaluate a DBF-based service discovery model in terms of the service discovery overhead, the Dynamic Bloom filter's efficiency, the network scalability, the service discovery success ratio and the average response time.

The Dynamic Bloom Filters

A Bloom filter is an effective, space-efficient data structure for concisely representing a set, and supporting approximate membership queries. Traditionally, the Bloom filter and its variants just focus on how to represent a static set and decrease the false positive probability to a sufficiently low level. By investigating mainstream applications based on the Bloom filter, we reveal that dynamic data sets are more common and important than static sets. However, existing variants of the Bloom filter cannot support dynamic data sets well. To address this issue, we propose dynamic Bloom filters to represent dynamic sets, as well as static sets and design necessary item insertion, membership query, item deletion, and filter union algorithms. The dynamic Bloom filter can control the false positive probability at a low level by expanding its capacity as the set cardinality increases. Through comprehensive mathematical analysis, we show that the dynamic Bloom filter uses less expected memory than the Bloom filter when representing dynamic sets with an upper bound on set cardinality, and also that the dynamic Bloom filter is more stable than the Bloom filter due to infrequent reconstruction when addressing dynamic sets without an upper bound on set cardinality. Moreover, the analysis results hold in stand-alone applications, as well as distributed applications.

An efficient data structure for network anomaly detection

AbstractDespite the rapid advance in networking technologies, detection of network anomalies at high‐speed switches/routers is still far from maturity. To push the frontier, two major technologies need to be addressed. The first one is efficient feature‐extraction algorithms/hardware that can match a line rate in the order of Gb/second; the second one is fast and effective anomaly detection schemes. In this paper, we focus on design of efficient data structure and algorithms for feature extraction. Specifically, we propose a novel data structure that extracts the so‐called two‐directional (2D) matching features, which are shown to be effective indicators of network anomalies. Our key idea is to use a Bloom filter array (BFA) to trade‐off a small amount of accuracy in feature extraction, for much less space and time complexity, so that our data structure can catch up with a line rate in the order of Gb/second. Different from the existing work, our data structure has the following properties: (1) it dynamic Bloom filter, (2) combination of a it sliding window with Bloom filter, and (3) using an insertion–removal pair to enhance Bloom filter with a removal operation. Our analysis and simulation demonstrate that the proposed data structure has a better space/time trade‐off than conventional algorithms. For example, for a fixed time complexity, the conventional algorithm (i.e., hash table [1—8]) requires a memory of 1.01 Gbits while our data structure requires a memory of only 62.9 Mbits, at the cost of losing 1% accuracy in feature extraction. Copyright © 2008 John Wiley & Sons, Ltd.