Hash Table Lookup Research Articles

Improved Parallel Genetic Algorithm Based on Resetting Strategy and Hash Table

In the process of parallel genetic algorithm (PGA) convergence, the probability of generating repeated individuals in a new generation is gradually increasing, which leads to repetitive computations of fitness values. In order to improve the time-efficiency and quality of solutions, an improved algorithm based on resetting strategy and hash table (RHPGA) is proposed. On the one hand, RHPGA takes advantage of the low time complexity of hash table lookup to reduce fitness values’ calculation of repeated individuals. On the other hand, RHPGA uses a resetting strategy to improve the optimal solution searching ability. By solving a set covering problem, PGA and RHPGA are compared. The experiment proves that RHPGA can almost increase the time-efficiency by 300% and quality of solutions by 15%.

SETJoin: a novel top-k similarity join algorithm

As an important operation in data cleaning, near duplicate Web pages detection and data mining, similarity joins have received much attention recently. Existing similarity joins fall into two broad categories—the similarity-threshold-based similarity join and top-k similarity join (TopkJoin). Compared with the traditional one, TopkJoin is more suitable for cases where the similarity threshold is unknown before hand. In this paper, we focus on the performance optimization problem of TopkJoin. Particularly, we observed that the state-of-the-art TopkJoin algorithm has three serious performance issues, i.e., the inappropriate application of hash table, inefficient use of suffix filtering and unnecessary evaluation of excessive unqualified candidates. To resolve these problems, we proposed a novel algorithm, SETJoin, by combining the existing event-driven framework with three simple yet efficient optimization techniques, viz., (1) reducing the cost in hashing by rearranging the orders of the candidate filtering and hash table lookup operations; (2) maximizing the pruning capability of suffix filtering by judiciously choosing the (near) optimal recursion depth; and (3) terminating join operations earlier by setting a much tighter stop condition for iteration. The experimental results show that SETJoin achieves up to 1.26x–3.49x speedup over the state-of-the-art algorithm on several real datasets.

Retrieval performance analysis of multibiometric database using optimized multidimensional spectral hashing based indexing

Demand for efficient retrieval of data from biometric databases is increasing due to its widespread authentication applications that range from e-passport to attendance system. Major research contribution in this area is by using tree based indexing methods and data independent random hashing methods. In this work, the data dependent hashing technique using optimized multidimensional spectral hashing that uses hash table lookup is employed. Palmprint, Iris and Face biometric features are generated using GIST, optimized features are fused based on bio-inspired cuckoo search algorithm and then converted into binary hash code. The compact binary codes representing the fused features form the multibiometric database on which the retrieval performance is analyzed. Simulation results obtained indicate that the hit rate and penetration rate have improved considerably for the desired recognition accuracy.

Intelligent probing for locality sensitive hashing

The past decade has been marked by the (continued) explosion of diverse data content and the fast development of intelligent data analytics techniques. One problem we identified in the mid-2000s was similarity search of feature-rich data. The challenge here was achieving both high accuracy and high efficiency in high-dimensional spaces. Locality sensitive hashing (LSH), which uses certain random space partitions and hash table lookups to find approximate nearest neighbors, was a promising approach with theoretical guarantees. But LSH alone was insufficient since a large number of hash tables were required to achieve good search quality. Building on an idea of Panigrahy, our multi-probe LSH method introduced the idea of intelligent probing. Given a query object, we strategically probe its neighboring hash buckets (in a query-dependent fashion) by calculating the statistical probabilities of similar objects falling into each bucket. Such intelligent probing can significantly reduce the number of hash tables while achieving high quality. In this paper, we revisit the problem motivation, the challenges, the key design considerations of multi-probe LSH, as well as discuss recent developments in this space and some questions for further research.

HASKER: An efficient algorithm for string kernels. Application to polarity classification in various languages

String kernels have successfully been used for various NLP tasks, ranging from text categorization by topic to native language identification. In this paper, we present a simple and efficient algorithm for computing various spectrum string kernels. When comparing two strings, we store the p-grams in the first string into a hash table, and then we apply a hash table lookup for the p-grams that occur in the second string. In terms of time, we show that our algorithm can outperform a state-of-the-art tool for computing string similarity. In terms of accuracy, we show that our approach can reach state-of-the-art performance for polarity classification in various languages. Our efficient implementation is provided online for free at http://string-kernels.herokuapp.com.

Background Subtraction Using Background Sets With Image- and Color-Space Reduction

Background subtraction is a basic step for a variety of multimedia applications such as live video, traffic monitoring, communication system, interactive learning space, etc. Many approaches have been proposed for this problem, but the need for lower cost approaches still exists. In this paper, a relatively low-cost background-subtraction method is proposed, using background sets with image- and color-space reduction. The background sets are used to detect objects from dynamic backgrounds, which contain waves, trees, and fountains. The image space is reduced to deal with jittered and unsteady frames, e.g., the input from handheld mobile devices. The color space is reduced to compensate for color noise, e.g., the scattered RGB values of a digital camera. To reduce the cost, a combination of color-space reduction and hash-table look-up operations are used. The results compared with other methods show the feasibility of our method; moreover, it can be useful in mobile or embedded environments.

Gatling

In this article, we propose Gatling, a framework that automatically finds performance attacks caused by insider attackers in large-scale message-passing distributed systems. In performance attacks, malicious nodes deviate from the protocol when sending or creating messages, with the goal of degrading system performance. We identify a representative set of basic malicious message delivery and lying actions and design a greedy search algorithm that finds effective attacks consisting of a subset of these actions. Although lying malicious actions are protocol dependent, requiring the format and meaning of messages, Gatling captures them without needing to modify the target system by using a type-aware compiler. We have implemented and used Gatling on nine systems, a virtual coordinate system, a distributed hash table lookup service and application, two multicast systems and one file sharing application, and three secure systems designed specifically to tolerate insiders, two based on virtual coordinates and one using Outlier Detection, one invariant derived from physical laws, and the last one a Byzantine resilient replication system. We found a total of 48 attacks, with the time needed to find each attack ranging from a few minutes to a few hours.

FMPC: A Fast Multi-Dimensional Packet Classification Algorithm

With the rapid development of the Internet, the number of firewall rules is increasing. The enormous quantity of rules challenges the performance of the packet classification that has already become a bottleneck in firewalls. This dissertation proposes a rapid and multi-dimensional algorithm for packet classification based on BSOL(Binary Search On Leaves), which is named FMPC(FastMulti-dimensional Packet Classification). Different from BSOL, FMPC cuts all dimensions at the same time to decompose rule spaces and stores leaf spaces into hash tables; FMPC constructs a Bloom Filter for every hash table and stores them into embedded SRAM. When classifying a packet, FMPC performs parallel queries on Bloom Filters and determines how to visit hash tables according to the results. Algorithm analysis and the result of simulations show: the average number of hash-table lookups of FMPC is 1 when classifying a packet, which is much smaller than that of BSOL; inthe worst case, the number of hash-table lookups of FMPCisO(logwmax+1⁡), which is also smaller than that of BSOL in multi-dimensional environment, where wmax is the length, in bits, of the dimension whose length is the longest..

High-Throughput and Memory-Efficient Multimatch Packet Classification Based on Distributed and Pipelined Hash Tables

The emergence of new network applications, such as the network intrusion detection system and packet-level accounting, requires packet classification to report all matched rules instead of only the best matched rule. Although several schemes have been proposed recently to address the multimatch packet classification problem, most of them require either huge memory or expensive ternary content addressable memory (TCAM) to store the intermediate data structure, or they suffer from steep performance degradation under certain types of classifiers. In this paper, we decompose the operation of multimatch packet classification from the complicated multidimensional search to several single-dimensional searches, and present an asynchronous pipeline architecture based on a signature tree structure to combine the intermediate results returned from single-dimensional searches. By spreading edges of the signature tree across multiple hash tables at different stages, the pipeline can achieve a high throughput via the interstage parallel access to hash tables. To exploit further intrastage parallelism, two edge-grouping algorithms are designed to evenly divide the edges associated with each stage into multiple work-conserving hash tables. To avoid collisions involved in hash table lookup, a hybrid perfect hash table construction scheme is proposed. Extensive simulation using realistic classifiers and traffic traces shows that the proposed pipeline architecture outperforms HyperCuts and B2PC schemes in classification speed by at least one order of magnitude, while having a similar storage requirement. Particularly, with different types of classifiers of 4K rules, the proposed pipeline architecture is able to achieve a throughput between 26.8 and 93.1 Gb/s using perfect hash tables.

Fast and deterministic hash table lookup using discriminative bloom filters

Hash tables are widely used in network applications, as they can achieve O(1) query, insert, and delete operations at moderate loads. However, at high loads, collisions are prevalent in the table, which increases the access time and induces non-deterministic performance. Slow rates and non-determinism can considerably hurt the performance and scalability of hash tables in the multi-threaded parallel systems such as ASIC/FPGA and multi-core. So it is critical to keep the hash operations faster and more deterministic.This paper presents a novel fast collision-free hashing scheme using Discriminative Bloom Filters (DBFs) to achieve fast and deterministic hash table lookup. DBF is a compact summary stored in on-chip memory. It is composed of an array of parallel Bloom filters organized by the discriminator. Each element lookup performs parallel membership checks on the on-chip DBF to produce a possible discriminator value. Then, the element plus the discriminator value is hashed to a possible bucket in an off-chip hash table for validating the match. This DBF-based scheme requires one off-chip memory access per lookup as well as less off-chip memory usage. Experiments show that our scheme achieves up to 8.5-fold reduction in the number of off-chip memory accesses per lookup than previous schemes.

Sub-fingerprint masking for a robust audio fingerprinting system in a real-noise environment for portable consumer devices

The robustness of audio fingerprinting system in a noisy environment is a principal challenge in the area of content-based music retrieval, especially for use in portable consumer devices. Our new audio fingerprint method using sub-fingerprint masking based on the predominant pitch extraction dramatically increases the accuracy of the audio fingerprinting system in a noisy environment, while requiring much less computing power for matching, compared to the expanded hash table lookup method, where the searching complexity increases by the factor of 33 times the degree of expansion.

AVOIDING ROTATED BITBOARDS WITH DIRECT LOOKUP

This paper describes an approach for obtaining direct access to the attacked squares of sliding pieces without resorting to rotated bitboards. The technique involves creating four hash tables using the built in hash arrays from an interpreted, high level language. The rank, file, and diagonal occupancy are first isolated by masking the desired portion of the board. The attacked squares are then directly retrieved from the hash tables. Maintaining incrementally updated rotated bitboards becomes unnecessary as does all the updating, mapping and shifting required to access the attacked squares. Finally, rotated bitboard move generation speed is compared with that of the direct hash table lookup method.

$O(\log W)$ Multidimensional Packet Classification

We use a collection of hash tables to represent a multidimensional packet classification table. These hash tables are derived from a trie-representation of the multidimensional classifier. The height of this trie is O(W), where W is the sum of the maximum possible length, in bits, of each of the fields of a filter. The leaves at level i of the trie together with markers for some of the leaves at levels j such that j>i are stored in a hash table H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> . The placement of markers is such that a binary search of the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sub> 's successfully locates the highest-priority filter that matches any given packet. The number of hash tables equals the trie height, O(W). Hence, a packet may be classified by performing O(logW) hash-table lookups. So the expected lookup-complexity of our data structure for multidimensional packet classification is O(logW). Our proposed scheme affords a memory advantage over the O(logW) 1-D scheme of Waldvogel For multidimensional packet classification, our proposed scheme provides both a time and memory advantage over the extended grid-of-tries scheme of Baboescu et al

Fast hash table lookup using extended bloom filter

Hash tables are fundamental components of several network processing algorithms and applications, including route lookup, packet classification, per-flow state management and network monitoring. These applications, which typically occur in the data-path of high-speed routers, must process and forward packets with little or no buffer, making it important to maintain wire-speed throughout. A poorly designed hash table can critically affect the worst-case throughput of an application, since the number of memory accesses required for each lookup can vary. Hence, high throughput applications require hash tables with more predictable worst-case lookup performance. While published papers often assume that hash table lookups take constant time, there is significant variation in the number of items that must be accessed in a typical hash table search, leading to search times that vary by a factor of four or more.We present a novel hash table data structure and lookup algorithm which improves the performance over a naive hash table by reducing the number of memory accesses needed for the most time-consuming lookups. This allows designers to achieve higher lookup performance for a given memory bandwidth, without requiring large amounts of buffering in front of the lookup engine. Our algorithm extends the multiple-hashing Bloom Filter data structure to support exact matches and exploits recent advances in embedded memory technology. Through a combination of analysis and simulations we show that our algorithm is significantly faster than a naive hash table using the same amount of memory, hence it can support better throughput for router applications that use hash tables.

Random Walk Based Heuristic Algorithms for Distributed Memory Model Checking

We explore the use of distributed processing to enhance the performance of explicit state enumeration based safety model-checking. State enumeration based model-checkers employ a hash-table to cut off search when a state is revisited. Distributed model-checkers distribute this table across the processing nodes, employing inter-node messages to perform state lookups. This approach incurs the following penalties: hashing states, looking up hash-tables, and possibly exchanging messages. In this paper, we study how to avoid these penalties in the context of safety model-checking, assuming that completeness can be sacrificed (acceptable for quick error detection). We employ the basic strategy of distributed random walk - a process of multiple processors randomly, and in an uncoordinated fashion, moving through the state-space looking for safety violations, without recording visited states. This process has the potential of maximizing CPU utilization, and consequently greatly increase the rate of state generation, as the pressure on the memory system as well as communication network are minimal. Moreover, the probability that a randomwalk repeats the same sequence of moves can decrease exponentially with the length of the sequence; thus, the work wasted by occasionally repeating short sequences of searches may be more than offset by the increased state generation rate. Our choices are ideal for distributed systems that have low amounts of memory per node, and are interconnected by low bandwidth networks. We also explore techniques that backoff slightly from our extremal choices, by exploring heuristic combinations of breadth-first search (BFS) and random-walk (RW) that require a modest amount of hash-table lookup and message exchanges. These search methods are natural to combine, since BFS requires higher amounts of memory to maintain queues, but guarantees to find the shortest path to a state, while RW has the opposite characteristics. In this paper, we first study these heuristic methods on synthetic benchmarks to gain sharper (more quantifiable) insights. We then conduct studies on some realistic examples as well. We employ up to 10 single-processor CPUs that happen to be connected via 100BASE-T Ethernets. Our code was easily ported to other platforms, thanks to our use of the popular MPI distributed programming library.