Perfect Hash Research Articles

Tight Bounds for Monotone Minimal Perfect Hashing

Tight Bounds for Monotone Minimal Perfect Hashing

Detecting arrays for effects of multiple interacting factors

Detecting arrays provide test suites for complex engineered systems in which many factors interact. The determination of which interactions have a significant impact on system behaviour requires not only that each interaction appear in a test, but also that its effect can be distinguished from those of other significant interactions. In this paper, compact representations of detecting arrays using vectors over the finite field are developed. Covering strong separating hash families exploit linear independence over the field, while the weaker elongated covering perfect hash families permit some linear dependence. For both, probabilistic analyses are employed to establish effective upper bounds on the number of tests needed in a detecting array for a wide variety of parameters. The analyses underlie efficient algorithms for the explicit construction of detecting arrays.

Parallel and External-Memory Construction of Minimal Perfect Hash Functions With PTHash

Parallel and External-Memory Construction of Minimal Perfect Hash Functions With PTHash

ChainedFilter: Combining Membership Filters by Chain Rule

Membership (membership query/membership testing) is a fundamental problem across databases, networks and security. However, previous research has primarily focused on either approximate solutions, such as Bloom Filters, or exact methods, like perfect hashing and dictionaries, without attempting to develop an integral theory. In this paper, we propose a unified and complete theory, namely chain rule, for general membership problems, which encompasses both approximate and exact membership as extreme cases. Building upon the chain rule, we introduce a straightforward yet versatile algorithm framework, namely ChainedFilter, to combine different elementary filters without losing information. Our evaluation results demonstrate that ChainedFilter improves performance of many applications including static dictionary, lossless data compression, Cuckoo Hashing, LSM-Tree and Learned Filters.

String Matching Algorithm Using Multi-Characters Inverted Lists

This research article introduces a new string-matching algorithm that utilizes the multi-character inverted lists structure, implemented using a perfect hashing method. The new solution processes the pattern input string in a single pass. In the searching phase, the algorithm takes a linear time complexity. This approach demonstrates efficient and rapid performance in practical experiments when a binary character set is used. It also handles fewer characters and shorter lengths, even when the pattern length is increased. In some cases, this algorithm exhibits superior text search performance compared to other algorithms. It has been observed that this superiority is particularly evident when searching for smaller amounts of text. This algorithm can be explored more quickly, especially when the search factor is between 0.25 to 0.75 times the length of the pattern, approaching the performance of the traditional algorithm.

Two-layer partitioned and deletable deep bloom filter for large-scale membership query

The recently proposed Learned Bloom Filter (LBF) provides a new perspective on large-scale membership queries by using machine learning to replace the traditional bloom filter. However, reducing the false positive rate (FPR) of the learned model with small memory usage, and supporting deletion efficiently become the new issues. In this paper, we propose a novel Two-layer Partitioned and Deletable Deep Bloom Filter (PDDBF) for large-scale membership query, which can reduce the FPR with small memory usage and support deletion efficiently. The proposed PDDBF consists of three main parts: (1) Data partition. To improve the classification accuracy of the learned model, the K-means cluster with the elbow method is used for the data partition. (2) Deep Bloom Filter. To reduce the FPR, deep learning models are used to construct multiple independent learning mechanisms, which correspond to the clusters obtained by part1. (3) Partitioned backup filter. To support deletion under the premise of ensuring low FPR and reducing query time consumption, combine the perfect hash (PH) table and counting bloom filters (CBFs) on the basis of the partition bloom filter. Experiments show that the proposed PDDBF reduces the FPR 87.13% with the same memory usage compared with the state-of-the-art PLBF on real-world URLs data set. Moreover, the PDDBF reduces the FPR 99.68% with the same memory usage and reduces the query time consumption to 2.61x that of the PLBF after data deletion, respectively.

Locality-preserving minimal perfect hashing of k-mers.

Minimal perfect hashing is the problem of mapping a static set of n distinct keys into the address space {1,…,n} bijectively. It is well-known that n log 2(e) bits are necessary to specify a minimal perfect hash function (MPHF) f, when no additional knowledge of the input keys is to be used. However, it is often the case in practice that the input keys have intrinsic relationships that we can exploit to lower the bit complexity of f. For example, consider a string and the set of all its distinct k-mers as input keys: since two consecutive k-mers share an overlap of k-1 symbols, it seems possible to beat the classic log 2(e) bits/key barrier in this case. Moreover, we would like f to map consecutive k-mers to consecutive addresses, as to also preserve as much as possible their relationship in the codomain. This is a useful feature in practice as it guarantees a certain degree of locality of reference for f, resulting in a better evaluation time when querying consecutive k-mers. Motivated by these premises, we initiate the study of a new type of locality-preserving MPHF designed for k-mers extracted consecutively from a collection of strings. We design a construction whose space usage decreases for growing k and discuss experiments with a practical implementation of the method: in practice, the functions built with our method can be several times smaller and even faster to query than the most efficient MPHFs in the literature.

Fingerprinting-based Minimal Perfect Hashing Revisited

In this paper we study a fingerprint-based minimal perfect hash function ( FMPH for short). While FMPH is not as space-efficient as some other minimal perfect hash functions (for example RecSplit, CHD, or PTHash), it has a number of practical advantages that make it worthy of consideration. FMPH is simple and quite fast to evaluate. Its construction requires very little auxiliary memory, takes a short time and, in addition, can be parallelized or carried out without holding keys in memory. In this paper, we propose an effective method (called FMPHGO ) that reduces the size of FMPH, as well as a number of implementation improvements. In addition, we experimentally study FMPHGO performance and find the best values for its parameters. Our benchmarks show that with our method and an efficient structure to support the rank queries on a bit vector, the FMPH size can be reduced to about 2.1 bits/key, which is close to the size achieved by state-of-the-art methods and noticeably larger only compared to RecSplit. FMPHGO preserves most of the FMPH advantages mentioned above, but significantly reduces its construction speed. However, FMPHGO’s construction speed is still competitive with methods of similar space efficiency (like CHD or PTHash), and seems to be good enough for practical applications.

Beating the Probabilistic Lower Bound on q-Perfect Hashing

In this paper, we investigate the achievable rate of perfect hash code. For an integer $$q\ge 2$$ , a perfect q-hash code C is a block code over $$[q]:=\{1,\ldots ,q\}$$ of length n in which every subset $$\{{\textbf{c}}_1,{\textbf{c}}_2,\dots ,{\textbf{c}}_q\}$$ of q elements is separated, i.e., there exists $$i\in [n]$$ such that $$\{\textrm{proj}_i({\textbf{c}}_1),\dots ,\textrm{proj}_i({\textbf{c}}_q)\}=[q]$$ , where $$\textrm{proj}_i({\textbf{c}}_j)$$ denotes the ith position of $${\textbf{c}}_j$$ . Finding the maximum size M(n, q) of perfect q-hash codes of length n, for given q and n, is a fundamental problem in combinatorics, information theory, and computer science. In this paper, we are interested in asymptotic behavior of this problem. Precisely speaking, we will focus on the quantity $$R_q:=\limsup _{n\rightarrow \infty }\frac{\log _2 M(n,q)}{n}$$ . A well-known probabilistic argument shows an existence lower bound on $$R_q$$ , namely $$R_q\ge \frac{1}{q-1}\log _2\left( \frac{1}{1-q!/q^q}\right) $$ (Fredman and Komlós, SIAM J Algebr Discrete Methods 5(1):61–68, 1984; Körner, SIAM J Algebr Discrete Methods 7(4):560–570, 1986). This is still the best-known lower bound till now except for the case $$q=3$$ (Körner and Marton, Eur J Comb 9(6):523–530, 1988). The improved lower bound of $$R_3$$ was discovered in 1988 and there has been no progress on the lower bound of $$R_q$$ for more than 30 years. In this paper we show that this probabilistic lower bound can be improved for q from 4 to 15 and all odd integers between 17 and 25, and all sufficiently large q.

The gerrymander sequence, or A348456

The gerrymander sequence, gL, given as A348456 in the OEIS, counts the number of ways to bisect a 2L×2L chessboard into two polyominoes of equal area. Recently Kauers, Koutschan and Spahn announced a significant increase in the length of this sequence from 3 to 7 terms. We give a further extension to 11 terms, but more significantly prove that the coefficients grow as λ4L2, where λ≈1.7445498, and is equal to the corresponding quantity for self-avoiding walks crossing a square (WCAS), or self-avoiding polygons crossing a square (PCAS). These are, respectively, OEIS sequences A007764 and A333323. Thus we have established a close connection between these previously separate problems.We have also related the sub-dominant behaviour to that of WCAS and PCAS, allowing us to conjecture that the coefficients of the gerrymander sequence grow as gL∼λ4L2+2dL+e⋅(2L)h, where d=−4.04354±0.0001, e≈8 and h=0.75±0.01, with h almost certainly 3/4 exactly.We have also generated 26 terms of the related OEIS sequence A068416, which counts the number of ways to partition a L×L square into two connected components (not necessarily of equal area). We have thus been able to predict the asymptotic behaviour of this sequence with a satisfying degree of precision. Indeed, it behaves exactly as L times the corresponding coefficient of the generalised gerrymander sequence (defined below).The improved algorithm we give for counting these sequences is a variation of that which we recently developed for extending a number of sequences for SAWs and SAPs crossing a domain of the square or hexagonal lattices. It makes use of a minimal perfect hash function and in-place memory updating of the arrays for the counts of the number of paths.

Perfect Hash-Based Routing Lookup for LEO Constellation Backbone Network

The performance of packet forwarding mainly relies on the routing lookup (RL) in high-speed routers and switches. Due to the network topology change and the limited resource on aerospace devices, the RL strategy for LEO constellation backbone networks should be specifically designed. Targeting lower resource consumption, we first propose the perfect hash-based RL, including a Perfect Hash Function (PHF) for metarule space compression plus random access for the output interface. A Satellite Perfect Hash Function (SPHF) algorithm is proposed to generate efficient hash functions with limited storage. For 10000 randomized keys, the SPHF algorithm constructs 50-Kb PHFs in 3.05 milliseconds with 538 Kb storage. Then we design the Routing-oriented Hash Function (RHF) algorithm by fusing the perfect hash and RL, and optimize RHF towards the hardware implementation. The RHF algorithm is robust to different RL scenarios. In the FPGA (xc7k70t-fbv900-1) experiment with 10000 32-bit metarules and 4 output interfaces, a typical RHF algorithm (RHF1) costs only 13.2 ns and 1429 equivalent LUTs to lookup each packet under 312.5 MHz clock frequency. Compared with most advanced Ternary Content Addressable Memory (TCAM) based implementations, RHF1 reduces the storage and logic resources by at least an order of magnitude while keeping comparable lookup latency in the same FPGA Chip. This paper provides a design mentality for the RL of LEO constellation backbone networks.

Algorithms and Data Structures for Hyperedge Queries

We consider the problem of querying the existence of hyperedges in hypergraphs. More formally, given a hypergraph, we need to answer queries of the form: “Does the following set of vertices form a hyperedge in the given hypergraph?” Our aim is to set up data structures based on hashing to answer these queries as fast as possible. We propose an adaptation of a well-known perfect hashing approach for the problem at hand. We analyze the space and runtime complexity of the proposed approach and experimentally compare it with the state-of-the-art hashing-based solutions. Experiments demonstrate the efficiency of the proposed approach with respect to the state-of-the-art.

Self-avoiding walks and polygons crossing a domain on the square and hexagonal lattices

We have analysed the recently extended series for the number of self-avoiding walks (SAWs) that cross an L × L square between diagonally opposed corners. The number of such walks is known to grow as We have made more precise the estimate of based on additional series coefficients provided by several authors, and refined analysis techniques. We estimate that We have also studied the subdominant behaviour, and conjecture that CL(1)∼λSL2+bL+c⋅Lg, where and We implemented a very efficient algorithm for enumerating paths on the square and hexagonal lattices making use of a minimal perfect hash function and in-place memory updating of the arrays for the counts of the number of paths. Using this algorithm we extended and then analysed series for SAWs spanning the square lattice and self-avoiding polygons (SAPs) crossing the square lattice. These are known to also grow as The sub-dominant term λ b is found to be the same as for SAWs crossing the square, while the exponent for spanning SAWs and for SAPs. We have also studied the analogous problems on the hexagonal lattice, and generated series for a number of geometries. In particular, we study SAWs and SAPs crossing rhomboidal, triangular and square domains on the hexagonal lattice, as well as SAWs spanning a rhombus. We estimate that the analogous growth constant so an even more precise estimate than found for the square lattice. We also give estimates of the sub-dominant terms.

Sparse and skew hashing of K-mers.

MotivationA dictionary of k-mers is a data structure that stores a set of n distinct k-mers and supports membership queries. This data structure is at the hearth of many important tasks in computational biology. High-throughput sequencing of DNA can produce very large k-mer sets, in the size of billions of strings—in such cases, the memory consumption and query efficiency of the data structure is a concrete challenge.ResultsTo tackle this problem, we describe a compressed and associative dictionary for k-mers, that is: a data structure where strings are represented in compact form and each of them is associated to a unique integer identifier in the range . We show that some statistical properties of k-mer minimizers can be exploited by minimal perfect hashing to substantially improve the space/time trade-off of the dictionary compared to the best-known solutions.Availability and implementation https://github.com/jermp/sshash.Supplementary information Supplementary data are available at Bioinformatics online.

SSLC: A Search Algorithm Based on Linear Collisions and Poisson Probability Distribution

This article proposes an algorithm, sequential search based on linear collisions ( SSLC ), based on Poisson probability distribution. SSLC works on large static volumes of data, whose keys are ordered and uniformly distributed. We proved that it takes a constant execution time with high probability. Experimental validation focuses on two metrics—processing time and memory resources. We compared SSLC with perfect hashing over various datasets. The experimentation reveals that our algorithm is three times faster with the use of three times fewer memory resources than perfect hashing. Its main contribution is to demonstrate that simple routines can have more powerful computational results.

In-Parameter-Order strategies for covering perfect hash families

Combinatorial testing makes it possible to test large systems effectively while maintaining certain coverage guarantees. At the same time, the construction of optimized covering arrays (CAs) with a large number of columns is a challenging task. Heuristic and Metaheuristic approaches often become inefficient when applied to large instances, as the computation of the quality for new moves or solutions during the search becomes too slow. Recently, the generation of covering perfect hash families (CPHFs) has led to vast improvements to the state of the art for many different instances of covering arrays. CPHFs can be considered a compact form of a specific family of covering arrays. Their compact representation makes it possible to apply heuristic methods for instances with a much larger number of columns. In this work, we adapt the ideas of the well-known In-Parameter-Order (IPO) strategy for covering array generation to efficiently construct CPHFs, and therefore implicitly covering arrays. We design a way to realize the concept of vertical extension steps in the context of CPHFs and discuss how a horizontal extension can be implemented in an efficient manner. Further, we develop a horizontal extension strategy for CPHFs with subspace restrictions that identifies candidate columns greedily based on conditional expectation. Then using a local optimization strategy, a candidate may be adjoined to the solution or may replace one of the existing columns. An extensive set of computational results yields many significant improvements on the sizes of the smallest known covering arrays.

Set-Min Sketch: A Probabilistic Map for Power-Law Distributions with Application to k-Mer Annotation

k-mer counts are important features used by many bioinformatics pipelines. Existing k-mer counting methods focus on optimizing either time or memory usage, producing in output very large count tables explicitly representing k-mers together with their counts. Storing k-mers is not needed if the set of k-mers is known, making it possible to only keep counters and their association to k-mers. Solutions avoiding explicit representation of k-mers include Minimal Perfect Hash Functions (MPHFs) and Count-Min sketches. We introduce Set-Min sketch-a sketching technique for representing associative maps inspired from Count-Min-and apply it to the problem of representing k-mer count tables. Set-Min is provably more accurate than both Count-Min and Max-Min-an improved variant of Count-Min for static datasets that we define here. We show that Set-Min sketch provides a very low error rate, in terms of both the probability and the size of errors, at the expense of a very moderate memory increase. On the other hand, Set-Min sketches are shown to take up to an order of magnitude less space than MPHF-based solutions, for fully assembled genomes and large k. Space-efficiency of Set-Min in this case takes advantage of the power-law distribution of k-mer counts in genomic datasets.

Beating Fredman-Komlós for perfect k-hashing

We say a subset C⊆{1,2,…,k}n is a k-hash code (also called k-separated) if for every subset of k codewords from C, there exists a coordinate where all these codewords have distinct values. Understanding the largest possible rate (in bits), defined as (log2⁡|C|)/n, of a k-hash code is a classical problem. It arises in two equivalent contexts: (i) the smallest size possible for a perfect hash family that maps a universe of N elements into {1,2,…,k}, and (ii) the zero-error capacity for decoding with lists of size less than k for a certain combinatorial channel.A general upper bound of k!/kk−1 on the rate of a k-hash code (in the limit of large n) was obtained by Fredman and Komlós in 1984 for any k≥4. While better bounds have been obtained for k=4, their original bound has remained the best known for each k≥5. In this work, we present a method to obtain the first improvement to the Fredman-Komlós bound for every k≥5.

A Dynamic Repository Approach for Small File Management With Fast Access Time on Hadoop Cluster: Hash Based Extended Hadoop Archive

Small file processing in Hadoop is one of the challenging task. The performance of the Hadoop is quite good when dealing with large files because they require lesser metadata and consume less memory. But while dealing with enormous amount of small files, metadata grows linearly and Name Node memory gets overloaded hence overall performance of the Hadoop degrades. This paper presents a dual merge technique HB-EHA (Hash Based-Extended Hadoop Archive), that will resolve the small file issue of Hadoop and provide an excellent solution for massive small files that are generated in the health care management applications. The proposed technique merges the small files using two-level compaction, therefore, the size of metadata at the name node gets reduced and less memory will be used. The indexing will be carried out over the archives and files can be accessed after merging in real-time. Index files in the proposed approach can read partially that improves the name node memory usage and also offers the file appending capability in the existing archive. The proposed technique first creates Hadoop archive from the small files and then uses two special hash functions i.e. SSHF (Scalable-Splittable Hash Function) and HT-MMPHF (Hollow Trie Monotone Minimal Perfect Hash Function), SSHF is used to dynamically distribute the archives meta-data to the associated slave index files, and these slave index files will be further written to the final index files, the order of the meta-data in final index file will be preserved by the HT-MMPHF. The evaluation outcome exhibit that the proposed technique is 13% & 17% faster than HDFS with caching enabled and disabled respectively, and 38% & 47% faster than the HAR with caching and without caching, respectively. While comparing with the map file, the proposed technique is 28 & 35 times faster with caching and without caching, respectively. HB-EHA is a maximum of 40% & 28% faster than the HBAF with and without caching, respectively.

Q‐frame hash comparison based exact string matching algorithms for DNA sequences

AbstractThe importance of string matching is due to its applications in many fields, such as medicine and bioinformatics. Various string matching algorithms are developed to speed up the search. Especially, hash‐based exact string matching algorithms are among the most time‐efficient ones. The efficiency of hash‐based approaches depends on the hash function. Hence, perfect hashing plays an essential role in hash‐based string matching. In this study, two q‐frame hash comparison‐based exact string matching algorithms, Hq‐QF and HqBM‐QF, are proposed. We have used a collision‐free perfect hash function for DNA sequences in the proposed algorithms. In the first approach, after hash values match for the last qcharacters, the character comparisons in the Hash‐q algorithm are replaced with q‐frame hash comparison. In the second approach, we improved the first approach by utilizing the shift size indicated at the th entry in the good suffix shift table. Since the number of character comparisons is minimized, the worst‐case time complexity of the proposed algorithms is . In both approaches, q‐frame hash comparisons replace most character comparisons as a trade‐off. The results show that the proposed approaches are more efficient than the Hash‐q algorithm in terms of runtime efficiency and the number of character comparisons.