Hash Collisions Research Articles

LearningTuple: A packet classification scheme with high classification and high update

Packet classification is widely used in network infrastructures and is the key technique that supports security and other functions. The real-time nature of network services naturally demands high classification speed, while the emerging SDN makes rule changes more flexible, thus placing higher demands on the performance of rule update in classification schemes. In this paper, Learning Tuple(LT) is proposed to achieve high classification performance for packets while maintaining the high update characteristics of tuple space-based schemes. Specifically, to solve the issue of excessive tuples and rule overlap due to merging tuples, LT iteratively divides the space by using rule overlap and hash collisions as negative feedback and applies a reinforcement learning algorithm, SARSA, at each level to ensure its reasonableness. Efficient space partitioning guides the construction of tuples, and an excellent rule mapping method called PLR is designed, which improves classification performance. Experimental results demonstrate that compared with classic and advanced classification schemes TSS, TupleMerge, MultilayerTuple, PartitionSort, HybridTSS, and TupleTree, LT achieves average classification performance improvements of 9.23x, 1.74x, 1.45x, 2.85x, 1.37x and 1.25x, as well as average update performance improvements of 1.83x, 6.75x, 1.22x, 6.16x, 1.21x, 10.66x, respectively.

ActiveGuardian: An accurate and efficient algorithm for identifying active elephant flows in network traffic

Active elephant flows, which indicate the real-time data transmission status, are of primary interest in network management and various applications. However, existing network measurement efforts mainly focus on finding elephant flows, and limited works on identifying active elephant flows suffer from low accuracy and heavy overheads. To address this issue, this paper proposes ActiveGuardian to identify active elephant flows with high accuracy, low memory, and high throughput. The key idea is to intelligently separate potential elephant flows from mice flows, and guard and report the information of active elephant flows in every time window. To obtain high accuracy, we devise a filtering module that adaptively filters unnecessary flows with low arrival rates, by applying an Adaptive Counter Update strategy. To achieve high memory utilization, we design a Leapfrog Hashing algorithm for the guarding module to effectively solve hash collisions. Lastly, we perform theoretical derivation on the false positive and the error bound of ActiveGuardian, and experimental evaluations on its performance with real network traffic traces. The experimental results show that ActiveGuardian achieves higher accuracy (99.65%) with identical memory sizes, and higher throughput (26.53Mps) than the state-of-the-art solutions.

CHAT: Accurate Network Latency Measurement for 5G E2E Networks

As numerous latency-sensitive applications have emerged with the popularization of 5G networking, accurate and rapid end-to-end latency measurement has come to play an essential role in network fault diagnosis and optimization. Although the Bloom hash-based timestamp aggregation method has been reported to be scalable and efficient, two shortcomings that reduce its accuracy have yet to be fully considered: frequent hash collisions and its fixed measurement interval. To address these challenges, we construct an end-to-end network latency measurement framework named Cuckoo Hash Adjustive Table exchange (CHAT). By employing an improved cuckoo filter, we decrease the number of hash collisions to assess the latency more accurately. Moreover, CHAT adjusts the receiver-side measurement interval dynamically based on a gain indicator, maximizing the total number of valid packets used for latency estimation. Additionally, the proposed measurement framework minimizes the number of packets transferred over links to avoid interfering with the end-to-end latency measurement in an actual network. Finally, extensive experiments on simulations and a practical real-world environment show the effectiveness and applicability of CHAT.

A One-Pass Clustering Based Sketch Method for Network Monitoring

Network monitoring solutions need to cope with increasing network traffic volumes, as a result, sketch-based monitoring methods have been extensively studied to trade accuracy for memory scalability and storage reduction. However, sketches are sensitive to skewness in network flow distributions due to hash collisions, and need complicated performance optimization to adapt to line-rate packet streams. We provide Jellyfish, an efficient sketch method that performs one-pass clustering over the network stream. One-pass clustering is realized by adapting the monitoring granularity from the whole network flow to fragments called subflows, which not only reduces the ingestion rate but also provides an efficient intermediate representation for the input to the sketch. Jellyfish provides the network-flow level query interface by reconstructing the network-flow level counters by merging subflow records from the same network flow. We provide probabilistic analysis of the expected accuracy of both existing sketch methods and Jellyfish. Real-world trace-driven experiments show that Jellyfish reduces the average estimation errors by up to six orders of magnitude for per-flow queries, by six orders of magnitude for entropy queries, and up to ten times for heavy-hitter queries.

An Improved Hashing Approach for Biological Sequence to Solve Exact Pattern Matching Problems

Pattern matching algorithms have gained a lot of importance in computer science, primarily because they are used in various domains such as computational biology, video retrieval, intrusion detection systems, and fraud detection. Finding one or more patterns in a given text is known as pattern matching. Two important things that are used to judge how well exact pattern matching algorithms work are the total number of attempts and the character comparisons that are made during the matching process. The primary focus of our proposed method is reducing the size of both components wherever possible. Despite sprinting, hash-based pattern matching algorithms may have hash collisions. The Efficient Hashing Method (EHM) algorithm is improved in this research. Despite the EHM algorithm’s effectiveness, it takes a lot of time in the preprocessing phase, and some hash collisions are generated. A novel hashing method has been proposed, which has reduced the preprocessing time and hash collision of the EHM algorithm. We devised the Hashing Approach for Pattern Matching (HAPM) algorithm by taking the best parts of the EHM and Quick Search (QS) algorithms and adding a way to avoid hash collisions. The preprocessing step of this algorithm combines the bad character table from the QS algorithm, the hashing strategy from the EHM algorithm, and the collision-reducing mechanism. To analyze the performance of our HAPM algorithm, we have used three types of datasets: E. coli, DNA sequences, and protein sequences. We looked at six algorithms discussed in the literature and compared our proposed method. The Hash-q with Unique FNG (HqUF) algorithm was only compared with E. coli and DNA datasets because it creates unique bits for DNA sequences. Our proposed HAPM algorithm also overcomes the problems of the HqUF algorithm. The new method beats older ones regarding average runtime, number of attempts, and character comparisons for long and short text patterns, though it did worse on some short patterns.

Private Set Intersection Based on Lightweight Oblivious Key-Value Storage Structure

At this stage, the application of Private Set Intersection (PSI) protocols is essential for smart homes. Oblivious Key-Value Stores (OKVS) can be used to design efficient PSI protocols. Constructing OKVS with a cuckoo hashing graph is a common approach. It increases the number of hash functions while reducing the possibility of collisions into rings. However, the existing OKVS construction scheme requires a high time overhead, and such an OKVS applied to PSI protocols would also have a high communication overhead. In this paper, we propose a method called 3-Hash Garbled Cuckoo Graph (3H-GCG) for constructing cuckoo hash graphs. Specifically, this method handles hash collisions between different keys more efficiently than existing methods, and it can also be used to construct an OKVS structure with less storage space. Based on the 3H-GCG, we design a PSI protocol using the Vector Oblivious Linear Evaluation (VOLE) and OKVS paradigm, which achieves semi-honest security and malicious security. Extensive experiments demonstrate the effectiveness of our method. When the set size is 218–220, our PSI protocol is less computationally intensive than other existing protocols. The experiments also show an increase in the ratio of raw to constructed data of about 7.5%. With the semi-honest security setting, our protocol achieves the fastest runtime with the set size of 218. With malicious security settings, our protocol has about 10% improvement in communication compared with other existing protocols.

Redactable consortium blockchain based on verifiable distributed chameleon hash functions

With the evolving application demands, the inherent immutability of consortium blockchains hinders their widespread adoption. For example, expired data stored on the chain cannot be deleted, and erroneous data cannot be redacted, seriously limiting the flexibility of consortium blockchains. However, existing redactable blockchain solutions need to be improved in aspects of decentralization, efficiency, and fault tolerance. This paper develops a new verifiable distributed chameleon hash (VDCH) function to solve the above problems. With VDCH, nodes share chameleon keys with a secure multi-party computation protocol based on a verifiable key-sharing scheme, and the collision shares can be verified with a Schnorr non-interactive zero-knowledge proof protocol, which enhances the fault tolerance of the consortium chain while maintaining its decentralized nature. Then, this paper proposes a consensus protocol called CVTSS based on verifiable threshold signatures, which provides protocol support for collaborative hash collision computation by multiple nodes using VDCH, thus avoiding the dependence on Nakamoto consensus and improving the redaction efficiency. Meanwhile, CVTSS uses threshold signatures to prevent malicious nodes from tampering with data using one-time chameleon keys. Finally, this paper constructs an efficient, practical, and secure redactable consortium chain scheme based on VDCH and CVTSS. Theoretical analysis and experimental results show that the proposed scheme can operate safely in the presence of malicious nodes with an acceptable time cost.

Robust Authentication Analysis of Copyright Images through Deep Hashing Models with Self-supervision

The increased usage of the internet and ICT has posed a significant challenge to protect copyrighted content due to advanced image forgery techniques that make image authentication extremely difficult. The aim of this paper is to establish a binary classification method for determining copyright images from copyright-free ones. A deep hashing model is introduced for an image authentication system, which uses deep learning-based perceptual hashing. Hash codes from a deep hashing model trained with a copyright image dataset are used to identify images. The deep learning model is able to learn features that represent the implicit meaning or structural information of an image. The copyright dataset, which lacks class labels, is trained with deep hashing models with self-supervision. The proposed model is based on an autoencoder or variational autoencoder model and is improved by including convolutional filters, residual blocks, and vision transformer blocks. Experimental results show that the proposed model performs a one-to-one mapping with most stored images and can retrieve related images using image features in hash collisions. The model can find the query image among the top 5 images with comparable hash codes. The results indicate that the proposed deep hashing approach is robust and applicable.

Speeding at the Edge: An Efficient and Secure Redactable Blockchain for IoT-Based Smart Grid Systems

As a promising approach to extending cloud resources and services, blockchain-enabled internet of things (IoT)-based smart grid edge computing has attracted much attention. However, the edge node’s resource-constraint nature makes it difficult to store the entire chain as the sensing IoT data volume increases. To address this issue, we propose an FS scheme, a fast and secure multi-threshold trapdoor chameleon hash scheme which serves as the basis for block substitution at the edge nodes to solve the storage limitation problem. FS scheme is used to achieve a consensus-based block substitution, which allows t-out-of-n edge nodes to compute a hash collision collaboratively to reliably substitute a historical block without leaking the randomness R. Also, inspired by the rationale of fast polynomial interpolation, we optimize the FS scheme to FS-I to reduce the time complexity from O(nt) to O(tlog2t). In addition, we further optimize FS-I to FS-II by using a fast Fourier transform (FFT) to dramatically improve the computational efficiency of Lagrange interpolation, which leads to a significant improvement in terms of block substitution performance. Finally, We provide security analysis and evaluate the performance through comprehensive experiments and the results show that FS can achieve up to several magnitudes better than DTTCH. The results also demonstrate that the FS scheme can provide high service quality for large-scale IoT-based smart grid systems.

Seeding with minimized subsequence.

Modern methods for computation-intensive tasks in sequence analysis (e.g. read mapping, sequence alignment, genome assembly, etc.) often first transform each sequence into a list of short, regular-length seeds so that compact data structures and efficient algorithms can be employed to handle the ever-growing large-scale data. Seeding methods using kmers (substrings of length k) have gained tremendous success in processing sequencing data with low mutation/error rates. However, they are much less effective for sequencing data with high error rates as kmers cannot tolerate errors. We propose SubseqHash, a strategy that uses subsequences, rather than substrings, as seeds. Formally, SubseqHash maps a string of length n to its smallest subsequence of length k, k < n, according to a given order overall length-k strings. Finding the smallest subsequence of a string by enumeration is impractical as the number of subsequences grows exponentially. To overcome this barrier, we propose a novel algorithmic framework that consists of a specifically designed order (termed ABC order) and an algorithm that computes the minimized subsequence under an ABC order in polynomial time. We first show that the ABC order exhibits the desired property and the probability of hash collision using the ABC order is close to the Jaccard index. We then show that SubseqHash overwhelmingly outperforms the substring-based seeding methods in producing high-quality seed-matches for three critical applications: read mapping, sequence alignment, and overlap detection. SubseqHash presents a major algorithmic breakthrough for tackling the high error rates and we expect it to be widely adapted for long-reads analysis. SubseqHash is freely available at https://github.com/Shao-Group/subseqhash.

A better hash method for high-fidelity Monte Carlo simulations on nuclear reactors

With the increasing demand for high-fidelity nuclear reactor simulations, the acceleration of Monte Carlo particle transport codes is becoming a core problem. One of the bottlenecks is locating millions or even billions of cells and fetching their associated parameters in the repeated geometry structure. Typically, Monte Carlo codes utilize a hash function to accelerate the cell locating and parameter indexing process. Specifically, they use the “cell vector → hash → parameter” method to accelerate the direct “cell vector → parameter” method. In this work, we propose a better hash method based on the Cyclic Redundancy Check (CRC) mechanism, which has been mathematically proven to be efficient and produce fewer hash collisions. Experimentally, this new hash method has been compared with some other hash functions and showed its superiority in terms of the calculation speed and collision probabilities. This hash method has been integrated into the Reactor Monte Carlo code RMC and worked well in practical applications.

When Tree Meets Hash: Reducing Random Reads for Index Structures on Persistent Memories

Indexing structures are widely used in modern data-processing applications to support high-performance queries, and there are a variety of recent designs specifically optimized for the newly available persistent memory (PM). The primary focus of previous PM indexes is on reducing the expensive PM writes for persisting data. However, we find that in tree-based PM indexes, because of the smaller performance gap between writes and random reads on real PM devices, the read-intensive tree traversal phase dominates the overall latency. This observation calls for further optimizations on existing indexing structures for PM. In this paper, we propose Extendible Radix Tree (ERT), an efficient indexing structure for PM that significantly reduces tree heights to minimize random reads, while still maintaining fast in-node search speed. The key idea is to use extendible hashing for each node in a radix tree. This design allows us to have a relatively large fanout of the radix tree to keep the tree height small, and also to realize constant-time lookups within a node. Using extendible hashing also allows for incremental node modification without excessive writes during inserts and updates. Range queries are efficiently and robustly handled by enforcing partial ordering among the keys in the hash table of each node without introducing more hash collisions. Our experiments on both synthetic and real-world data sets demonstrate that ERT achieves up to 2.65×, 4.41×, and 2.43× speedups for search, insert, and range queries over the respectively state-of-the-art PM index.

Pontus: Finding Waves in Data Streams

The bumps and dips in data streams are valuable patterns for data mining and networking scenarios such as online advertising and botnet detection. In this paper, we define the wave, a data stream pattern with a serious deviation from the stable arrival rate for a period of time. We then propose Pontus, an efficient framework for wave detection and estimation. In Pontus, a lightweight data structure is utilized for the preliminary processing of incoming packets in the data plane to take advantage of its high processing speed; then, the powerful control plane carries out computationally intensive wave detection and estimation. In particular, we propose the Multi-Stage Progressive Tracking strategy which detects waves in stages and removes any disqualified items promptly to save memory. Hash collisions are addressed by a Stage Variance Maximization technique to reduce estimation error. Moreover, we prove the theoretical error bound and establish upper bounds of false positive and false negative. Experiment results show that the software version of Pontus can achieve around 97% F1-Score even under scarce memory when baselines fail. Furthermore, the implemented prototype of Pontus based on P4 achieves 842x higher throughput than the baseline strawman solution.

Search mechanism for data contents based on bloom filter and tree hybrid structure in system wide information management

AbstractThe system wide information management (SWIM) system infrastructure layer uses information‐centric networking (ICN) to implement the cache routing and sharing of air traffic information data. The SWIM network searches and forwards routes on the basis of content names. The variable length, hierarchical name structure, and routing updates caused by the frequent publication and deletion of content make the implementation of fast name routing lookup algorithms a significant but arduous task. To address this problem, this study designed a name matching mechanism on the basis of the hybrid structure of the Bloom filter and the Tree. Compared with the traditional Bloom filter search scheme, this search mechanism reduces the quantity of Bloom filter insertion entries and reduces the possibility of hash collisions. Compared with the traditional Tree search scheme, the proposed mechanism can effectively solve the problem of numerous memory accesses caused by extremely long name prefixes, improving the query efficiency. In addition, the proposed composite structure is compared with several methods, such as DIPIT and NPT, by analyzing the effects of layering, global delay and search speed. The experimental results show that the proposed structure has a favourable performance in terms of storage overhead and search speed.

Enhanced network load balancing technique for efficient performance in software defined network.

Hash collisions and redirection of loads are major limitations for recent Hash IP algorithms. To overcome this, we propose a new Hash IP algorithm dubbed HDW in network load balancing (NLB) to increase network's efficiency, availability and scalability. We achieve the new Hash IP load balancing algorithm via a constructive merger with weighted scheduler (WS) technique and dynamic switching of routing path (DSP). This helps to reduce delays, jitters and additionally assures some level of security owing to the hashing process. Our findings after comprehensive simulations and performance evaluation depicts that our proposed HDW algorithm is relatively efficient as against other related load balancing algorithms for software defined network.

Incorporating Prior Knowledge in Local Differentially Private Data Collection for Frequency Estimation

Local differential privacy (LDP) is a prevalent measure of privacy protection as it provides rigorous privacy guarantees and has been widely studied for statistical analysis, especially in frequency estimation. As a representative LDP-enabled frequency estimation algorithm, Google's <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Randomized Aggregation Privacy-Preserving Ordinal Response</i> (RAPPOR) has been put into practice. However, it achieves sub-optimal utility due to the following limitations. Firstly, the adoption of the MD5 hash function inevitably results in the hash collision. Secondly, the application of the randomized response technique leads to randomness. To improve the practical effectiveness of RAPPOR and the utility of frequency-based services, we propose an LDP-enabled frequency estimation method called PK-RAPPOR, in which we devise an effective re-encoding hash function (RE-HF) incorporating prior knowledge (PK) about the rough frequency ranking of items. RE-HF divides items into several cohorts based on the PK and generates a unique hash value set for each item. Compared with the original RAPPOR, the hash collision can be eliminated for items from different cohorts, and the effect of randomness can be decreased by the overlapping of items from the same cohorts. We validate our proposed method with theoretical analysis and demonstrate its effectiveness with experiments on both synthetic and real-world datasets.

SDOT: Secure Hash, Semantic Keyword Extraction, and Dynamic Operator Pattern-Based Three-Tier Forensic Classification Framework

Most traditional digital forensic techniques identify irrelevant files in a corpus using keyword search, frequent hashes, frequent paths, and frequent size methods. These methods are based on Message Digest and Secure Hash Algorithm-1, which result in a hash collision. The threshold criteria of files based on frequent sizes will lead to imprecise threshold values that result in an increased evaluation of irrelevant files. The blacklisted keywords used in forensic search are based on literal and non-lexical, thus resulting in increased false-positive search results and failure to disambiguate unstructured text. Due to this, many extraneous files are also being considered for further investigations, exacerbating the time lag. Moreover, the non-availability of standardized forensic labeled data results in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(O(2^{n}))$ </tex-math></inline-formula> time complexity during the file classification process. This research proposes a three-tier Keyword Metadata Pattern framework to overcome these significant concerns. Initially, Secure Hash algorithm-256 hash for the entire corpus is constructed along with custom regex and stop-words module to overcome hash collision, imprecise threshold values, and eliminate recurrent files. Then blacklisted keywords are constructed by identifying vectorized words that have proximity to overcome traditional keyword search’s drawbacks and to overcome false positive results. Dynamic forensic relevant patterns based on massive password datasets are designed to search for unique, relevant patterns to identify the significant files and overcome the time lag. Based on tier-2 results, files are preliminarily classified automatically in O(log n) complexity, and the system is trained with a machine learning model. Finally, when experimentally evaluated, the overall proposed system was found to be very effective, outperforming the existing two-tier model in terms of finding relevant files by automated labeling and classification in O(nlog n) complexity. Our proposed model could eliminate 223K irrelevant files and reduce the corpus by 4.1% in tier-1, identify 16.06% of sensitive files in tier-2, and classify files with 91% precision, 95% sensitivity, 91% accuracy, and 0.11% Hamming loss compared to the two-tier system.

Wolverine: A Scalable and Transaction-Consistent Redactable Permissionless Blockchain

The immutability of blockchains makes it a critical technology for cryptocurrencies, but an imperative need arises for the redaction of on-chain data due to privacy-protecting laws like GPDR. Recently, Ateniese et al. (EuroS&P 2017) proposed an elegant solution to this problem based on chameleon hash functions, followed by many subsequent works. While these works offered a solution to the permissioned blockchain, the approaches were not efficient enough for the permissionless setting, in terms of either security (which may cause inconsistent historical transactions) or performance (only up to a few hundred nodes). In this paper, we investigate this problem and present Wolverine, a redactable permissionless blockchain. First, we present a formal redactable blockchain model, carefully considering transaction consistency. Next, towards a practical scheme, we introduce the novel concept of non-interactive chameleon hash (NITCH). NITCHs dynamically distribute a trapdoor key among a group and each party in the group can compute its partial share without communicating with others. Anyone who possesses enough shares can then find a valid hash collision. To prevent the static group from being compromised after a sufficiently long time, we provide a generic transform from NITCHs to decentralized random beacons (DRBs) and design a committee evolution protocol based on DRBs that refresh the group after every fixed interval of time. Based on NITCH and the committee evolution protocol, we construct Wolverine which offers important features such as scalability, transaction consistency, and public accountability. Finally, we demonstrate the practicality of Wolverine by giving a proof-of-concept implementation based on Bitcoin in Golang.

ASA: A ccelerating S parse A ccumulation in Column-wise SpGEMM

Sparse linear algebra is an important kernel in many different applications. Among various sparse general matrix-matrix multiplication (SpGEMM) algorithms, Gustavson’s column-wise SpGEMM has good locality when reading input matrix and can be easily parallelized by distributing the computation of different columns of an output matrix to different processors. However, the sparse accumulation (SPA) step in column-wise SpGEMM, which merges partial sums from each of the multiplications by the row indices, is still a performance bottleneck. The state-of-the-art software implementation uses a hash table for partial sum search in the SPA, which makes SPA the largest contributor to the execution time of SpGEMM. There are three reasons that cause the SPA to become the bottleneck: (1) hash probing requires data-dependent branches that are difficult for a branch predictor to predict correctly; (2) the accumulation of partial sum is dependent on the results of the hash probing, which makes it difficult to hide the hash probing latency; and (3) hash collision requires time-consuming linear search and optimizations to reduce these collisions require an accurate estimation of the number of non-zeros in each column of the output matrix. This work proposes ASA architecture to accelerate the SPA. ASA overcomes the challenges of SPA by (1) executing the partial sum search and accumulate with a single instruction through ISA extension to eliminate data-dependent branches in hash probing, (2) using a dedicated on-chip cache to perform the search and accumulation in a pipelined fashion, (3) relying on the parallel search capability of a set-associative cache to reduce search latency, and (4) delaying the merging of overflowed entries. As a result, ASA achieves an average of 2.25× and 5.05× speedup as compared to the state-of-the-art software implementation of a Markov clustering application and its SpGEMM kernel, respectively. As compared to a state-of-the-art hashing accelerator design, ASA achieves an average of 1.95× speedup in the SpGEMM kernel.

MinSIB: Minimized static instrumentation for fuzzing binaries

Code instrumentation is widely used in software analysis. The coverage-guided fuzzers collect path information by instrumenting additional code into the executable to guide the mutator. Existing instrumentation tools try to instrument all basic blocks to achieve high code coverage. This causes additional run-time overhead to the target and slows down the fuzzing. Also, too many paths may lead to a hash collision, or distract the attention of the fuzzer to less important code. To pay more attention to the instrumented code and place the instrumentation points into the functional areas of interest or security-related basic blocks, this paper presents MinSIB (Minimized static instrumentation for fuzzing binaries), a static binary instrumentation toolkit for Windows binaries. It instruments additional code into the executable by inline mode to reduce the additional overhead caused by instrumentation. Meanwhile, three optimization approaches are adopted to minimize the number of instrumentation points. Based on this, MinSIB reduces the number of instrumentation points by 90% on average, and the run-time overhead is 79% less than that of full instrumentation. Also, MinSIB improves the efficiency of fuzzing by reproducing known vulnerabilities 4 to 24 times faster than the existing tools, and 14 unknown bugs are found from 6 real-world programs. The results indicate that MinSIB outperforms the state-of-the-art static binary instrumentation tool (PE-AFL) and dynamic instrumentation tool (DynamoRIO) for Windows PE files.