Locality Sensitive Hashing Research Articles

TLS fingerprint for encrypted malicious traffic detection with attributed graph kernel

Recently, more and more applications have adopted security protocols like Transport Layer Security (TLS) for data encryption. However, these privacy-enhancing approaches have also been abused by the attackers to deliver malicious payloads. Many existing encrypted network classification methods suffer from the imbalanced volume of normal and malicious traffic, which leads to bad model robustness. In this paper, we propose a novel TLS fingerprinting approach to capture the characteristics of encrypted network traffic. The fingerprints are attributed graphs obtained from TLS sessions, which can simultaneously take into consideration the sequential and statistical features of these sessions. As the communication patterns of different applications differ considerably, the graphs representing TLS connections could be used to characterize the network with the help of the graph kernel method, which results in a model with high accuracy in malicious TLS session detection and application discrimination. Moreover, we adopt Locality-Sensitive Hashing (LSH) and filtering techniques to reduce the time cost of our model. Model evaluation on real-world datasets shows that our model is more robust than existing methods presented in this work when the malicious traffic takes up an extremely small portion of the whole traffic.

MPSketch: Message Passing Networks via Randomized Hashing for Efficient Attributed Network Embedding.

Given a network, it is well recognized that attributed network embedding represents each node of the network in a low-dimensional space, and, thus, brings considerable benefits for numerous graph mining tasks. In practice, a diverse set of graph tasks can be processed efficiently via the compact representation that preserves content and structure information. The majority of attributed network embedding approaches, especially, the graph neural network (GNN) algorithms, are substantially costly in either time or space due to the expensive learning process, while the randomized hashing technique, locality-sensitive hashing (LSH), which does not need learning, can speedup the embedding process at the expense of losing some accuracy. In this article, we propose the MPSketch model, which bridges the performance gap between the GNN framework and the LSH framework by adopting the LSH technique to pass messages and capture high-order proximity in a larger aggregated information pool from the neighborhood. The extensive experimental results confirm that in node classification and link prediction, the proposed MPSketch algorithm enjoys performance comparable to the state-of-the-art learning-based algorithms and outperforms the existing LSH algorithms, while running faster than the GNN algorithms by 3-4 orders of magnitude. More precisely, MPSketch runs 2121, 1167, and 1155 times faster than GraphSAGE, GraphZoom, and FATNet on average, respectively.

Efficient Inference of Graph Neural Networks Using Local Sensitive Hash

Efficient Inference of Graph Neural Networks Using Local Sensitive Hash

DET-LSH: A Locality-Sensitive Hashing Scheme with Dynamic Encoding Tree for Approximate Nearest Neighbor Search

Locality-sensitive hashing (LSH) is a well-known solution for approximate nearest neighbor (ANN) search in high-dimensional spaces due to its robust theoretical guarantee on query accuracy. Traditional LSH-based methods mainly focus on improving the efficiency and accuracy of the query phase by designing different query strategies, but pay little attention to improving the efficiency of the indexing phase. They typically fine-tune existing data-oriented partitioning trees to index data points and support their query strategies. However, their strategy to directly partition the multi-dimensional space is time-consuming, and performance degrades as the space dimensionality increases. In this paper, we design an encoding-based tree called Dynamic Encoding Tree (DE-Tree) to improve the indexing efficiency and support efficient range queries based on Euclidean distance. Based on DE-Tree, we propose a novel LSH scheme called DET-LSH. DET-LSH adopts a novel query strategy, which performs range queries in multiple independent index DE-Trees to reduce the probability of missing exact NN points, thereby improving the query accuracy. Our theoretical studies show that DET-LSH enjoys probabilistic guarantees on query accuracy. Extensive experiments on real-world datasets demonstrate the superiority of DET-LSH over the state-of-the-art LSH-based methods on both efficiency and accuracy. While achieving better query accuracy than competitors, DET-LSH achieves up to 6x speedup in indexing time and 2x speedup in query time over the state-of-the-art LSH-based methods.

Data reduction for SVM training using density-based border identification.

Numerous classification and regression problems have extensively used Support Vector Machines (SVMs). However, the SVM approach is less practical for large datasets because of its processing cost. This is primarily due to the requirement of optimizing a quadratic programming problem to determine the decision boundary during training. As a result, methods for selecting data instances that have a better likelihood of being chosen as support vectors by the SVM algorithm have been developed to help minimize the bulk of training data. This paper presents a density-based method, called Density-based Border Identification (DBI), in addition to four different variations of the method, for the lessening of the SVM training data through the extraction of a layer of border instances. For higher-dimensional datasets, the extraction is performed on lower-dimensional embeddings obtained by Uniform Manifold Approximation and Projection (UMAP), and the resulting subset can be repetitively used for SVM training in higher dimensions. Experimental findings on different datasets, such as Banana, USPS, and Adult9a, have shown that the best-performing variations of the proposed method effectively reduced the size of the training data and achieved acceptable training and prediction speedups while maintaining an adequate classification accuracy compared to training on the original dataset. These results, as well as comparisons to a selection of related state-of-the-art methods from the literature, such as Border Point extraction based on Locality-Sensitive Hashing (BPLSH), Clustering-Based Convex Hull (CBCH), and Shell Extraction (SE), suggest that our proposed methods are effective and potentially useful.

SenseHash: Computing on Sensor Values Mystified at the Origin

We propose SenseHash, a novel design for the lightweight in-hardware mystification of the sensed data at the origin. The framework aims to ensure the privacy of sensitive sensor values while preserving their utility. The sensors are assumed to interface to various (potentially malicious) communication and computing components in the Internet-of-things (IoT) and other emerging pervasive computing scenarios. The primary security primitives of our work are Locality Sensitive Hashing (LSH) combined with Differential Privacy (DP) and secure construction of LSH. Our construction allows (i) sub-linear search in sensor readings while ensuring their security against triangulation attack, and (ii) differentially private statistics of the readings. SenseHash includes hardware architecture as well as accompanying protocols to efficiently utilize the secure readings in practical scenarios. Alongside these scenarios, we present an automated workflow to generalize the application of the mystified readings. Proof-of-concept FPGA implementation of the system demonstrates its practicability and low overhead in terms of hardware resources, energy consumption, and protocol execution time.

An algorithm for constructing spatial vector data storage based on KD-tree and density estimation

With the widespread application of spatial vector data in various fields, this paper proposes a novel algorithm for constructing spatial vector data storage. The algorithm is based on KD-tree and density estimation techniques, aiming to improve the storage efficiency and query performance of large-scale spatial vector data. The algorithm first efficiently organizes spatial vector data using KD-tree, and then performs density estimation based on the Locality Sensitive Hashing (LSH) algorithm. Algorithm optimizations for spatial partitioning are applied to the KD-tree construction algorithm, reducing the search range during queries and improving retrieval speed. By testing with Green Tide data, the algorithm based on KD-tree and density estimation demonstrates higher query efficiency and better scalability across multiple test datasets. Particularly, when dealing with large-scale datasets characterized by non-uniform data distributions, this algorithm significantly improves data retrieval speed while maintaining low storage overhead.

Time-aware outlier detection in health physique monitoring in edge-aided sport education decision-makings

The increasing popularity of various intelligent sensor and mobile communication technologies has enabled quick health physique sensing, monitoring, collection and analyses of students, which significantly promoted the development of sport education. Through collecting the students’ physiological signals and transmitted them to edge servers, we can precisely analyze and judge whether a student is in poor health (e.g., an outlier). However, with time elapsing, the accumulated physiological signals of students become massive, which places a heavy burden on the quick storage and in-time processing of physiological data of students. In this situation, it is becoming a necessity to develop a time-aware outlier detection technique for health physique evaluation of students in a time-efficient way. Considering this challenge, we propose a novel time-aware outlier detection method named TOD based on Locality-Sensitive Hashing. TOD condenses extensive physiological student data into a concise set of health indices. Leveraging these indices, we can efficiently identify potential student outliers from a large pool of candidates with precision and speed. Finally, we have designed a group of simulated experiments based on WS-DREAM dataset. Experiment results prove the feasibility and superiority of the TOD method compared with other existing methods.

Generator Assisted Mixture of Experts for Feature Acquisition in Batch

Given a set of observations, feature acquisition is about finding the subset of unobserved features which would enhance accuracy. Such problems has been explored in a sequential setting in prior work. Here, the model receives feedback from every new feature acquireed and chooses to explore more features or to predict. However, sequential acquisition is not feasible in some settings where time is of essence. We consider the problem of feature acquisition in batch, where the subset of features to be queried in batch is chosen based on the currently observed features, and then acquired as a batch, followed by prediction. We solve this problem using several technical innovations. First, we use a feature generator to draw a subset of the synthetic features for some examples, which reduces the cost of oracle queries. Second, to make the feature acquisition problem tractable for the large heterogeneous observed features, we partition the data into buckets, by borrowing tools from locality sensitive hashing and then train a mixture of experts model. Third, we design a tractable lower bound of the original objective. We use a greedy algorithm combined with model training to solve the underlying problem. Experiments with four datasets shows that our approach outperforms these methods in terms of trade off between accuracy and feature acquisition cost.

Bridging the Semantic Latent Space between Brain and Machine: Similarity Is All You Need

How our brain encodes complex concepts has been a longstanding mystery in neuroscience. The answer to this problem can lead to new understandings about how the brain retrieves information in large-scale data with high efficiency and robustness. Neuroscience studies suggest the brain represents concepts in a locality-sensitive hashing (LSH) strategy, i.e., similar concepts will be represented by similar responses. This finding has inspired the design of similarity-based algorithms, especially in contrastive learning. Here, we hypothesize that the brain and large neural network models, both using similarity-based learning rules, could contain a similar semantic embedding space. To verify that, this paper proposes a functional Magnetic Resonance Imaging (fMRI) semantic learning network named BrainSem, aimed at seeking a joint semantic latent space that bridges the brain and a Contrastive Language-Image Pre-training (CLIP) model. Given that our perception is inherently cross-modal, we introduce a fuzzy (one-to-many) matching loss function to encourage the models to extract high-level semantic components from neural signals. Our results claimed that using only a small set of fMRI recordings for semantic space alignment, we could obtain shared embedding valid for unseen categories out of the training set, which provided potential evidence for the semantic representation similarity between the brain and large neural networks. In a zero-shot classification task, our BrainSem achieves an 11.6% improvement over the state-of-the-art.

CONSULT-II: accurate taxonomic identification and profiling using locality-sensitive hashing.

Taxonomic classification of short reads and taxonomic profiling of metagenomic samples are well-studied yet challenging problems. The presence of species belonging to groups without close representation in a reference dataset is particularly challenging. While k-mer-based methods have performed well in terms of running time and accuracy, they tend to have reduced accuracy for such novel species. Thus, there is a growing need for methods that combine the scalability of k-mers with increased sensitivity. Here, we show that using locality-sensitive hashing (LSH) can increase the sensitivity of the k-mer-based search. Our method, which combines LSH with several heuristics techniques including soft lowest common ancestor labeling and voting, is more accurate than alternatives in both taxonomic classification of individual reads and abundance profiling. CONSULT-II is implemented in C++, and the software, together with reference libraries, is publicly available on GitHub https://github.com/bo1929/CONSULT-II.

DB-LSH 2.0: Locality-Sensitive Hashing With Query-Based Dynamic Bucketing

DB-LSH 2.0: Locality-Sensitive Hashing With Query-Based Dynamic Bucketing

Refining Codes for Locality Sensitive Hashing

Refining Codes for Locality Sensitive Hashing

A novel fault detection algorithm for high voltage DC networks

Faults in high voltage DC networks can lead to inefficient operation, power outage or even equipment damage. Therefore, several approaches have been proposed to monitor, detect, and handle such problems. Due to high currents during faults, run-time for detecting malfunctions is crucial. Thus, fast recognition of small deviations that might lead to an abnormal operation is quintessential in minimizing consequences. This work presents a novel fault-detection mechanism, which is based on locality-sensitive hashes and its run-time scales linearly with the number of analyzed samples. To detect a malfunction, the algorithm uses reference signals corresponding to the normal operating point. Yet, only 19 reference signals were sufficient and delivered results comparable to a system using 100. When having only 19 references, the detector’s runtime is below 0.15 ms for an analyzed signal of 400 samples. Apart from fault detection, the proposed mechanism can also be used to reduce the storage requirements of the monitoring system. For a later investigation of a faulty event, the monitoring system must store the samples containing signal degradation. Even some temporary small changes can yield valuable information regarding the cause of the fault. However, storing many samples inevitably leads to big storage units, which are costly. To address this issue, the proposed mechanism sends a criticality factor to the monitoring system, based on which the signal samples may be compressed or removed.

Efficient similar waveform search using short binary codes obtained through a deep hashing technique

Summary A similar waveform search plays a crucial role in seismology for detecting seismic events, such as small earthquakes and low-frequency events. However, the high computational costs associated with waveform cross-correlation calculations represent bottlenecks during the analysis of long, continuous records obtained from numerous stations. In this study, we developed a deep-learning network to obtain 64-bit hash codes containing information on seismic waveforms. Using this network, we performed a similar waveform search for ∼35 million moving windows developed for the 30 min waveforms recorded continuously at 10 MHz sampling rates using 16 acoustic emission transducers during a laboratory hydraulic fracturing experiment. The sampling points of each channel corresponded to those of the 5.8-year records obtained from typical seismic observations at 100 Hz sampling rates. Of the 35 million windows, we searched for windows with small average Hamming distances among the hash codes of 16 channel waveforms against template hash codes of 6057 events that were catalogued using conventional autoprocessing techniques. The calculation of average Hamming distances is 1430–1530 times faster than that of the corresponding network correlation. This hashing-based template matching enabled the detection of 23,462 additional events. We also demonstrated the feasibility of the hashing-based autocorrelation analysis, where similar event pairs were extracted without templates, by calculating the average Hamming distances for all possible pairs of the ∼35 million windows. This calculation required only 15.5 h under 120 thread parallelisation. This deep hashing approach significantly reduced the required memory compared with locality-sensitive hashing approaches based on random permutations, enabling similar waveform searching on a large-scale dataset.

Secure Discovery of Genetic Relatives across Large-Scale and Distributed Genomic Datasets.

Finding relatives within a study cohort is a necessary step in many genomic studies. However, when the cohort is distributed across multiple entities subject to data-sharing restrictions, performing this step often becomes infeasible. Developing a privacy-preserving solution for this task is challenging due to the significant burden of estimating kinship between all pairs of individuals across datasets. We introduce SF-Relate, a practical and secure federated algorithm for identifying genetic relatives across data silos. SF-Relate vastly reduces the number of individual pairs to compare while maintaining accurate detection through a novel locality-sensitive hashing approach. We assign individuals who are likely to be related together into buckets and then test relationships only between individuals in matching buckets across parties. To this end, we construct an effective hash function that captures identity-by-descent (IBD) segments in genetic sequences, which, along with a new bucketing strategy, enable accurate and practical private relative detection. To guarantee privacy, we introduce an efficient algorithm based on multiparty homomorphic encryption (MHE) to allow data holders to cooperatively compute the relatedness coefficients between individuals, and to further classify their degrees of relatedness, all without sharing any private data. We demonstrate the accuracy and practical runtimes of SF-Relate on the UK Biobank and All of Us datasets. On a dataset of 200K individuals split between two parties, SF-Relate detects 94.9% of third-degree relatives, and 99.9% of second-degree or closer relatives, within 15 hours of runtime. Our work enables secure identification of relatives across large-scale genomic datasets.

Accuracy-enhanced E-commerce recommendation based on deep learning and locality-sensitive hashing

Accuracy-enhanced E-commerce recommendation based on deep learning and locality-sensitive hashing

Identifying Anomaly in IoT Traffic Flow With Locality Sensitive Hashes

Identifying Anomaly in IoT Traffic Flow With Locality Sensitive Hashes

A Graph Classification Method Based on Support Vector Machines and Locality-Sensitive Hashing

A Graph Classification Method Based on Support Vector Machines and Locality-Sensitive Hashing

Context Recovery and Knowledge Retrieval: A Novel Two-Stream Framework for Video Anomaly Detection.

Video anomaly detection aims to find the events in a video that do not conform to the expected behavior. The prevalent methods mainly detect anomalies by snippet reconstruction or future frame prediction error. However, the error is highly dependent on the local context of the current snippet and lacks the understanding of normality. To address this issue, we propose to detect anomalous events not only by the local context, but also according to the consistency between the testing event and the knowledge about normality from the training data. Concretely, we propose a novel two-stream framework based on context recovery and knowledge retrieval, where the two streams can complement each other. For the context recovery stream, we propose a spatiotemporal U-Net which can fully utilize the motion information to predict the future frame. Furthermore, we propose a maximum local error mechanism to alleviate the problem of large recovery errors caused by complex foreground objects. For the knowledge retrieval stream, we propose an improved learnable locality-sensitive hashing, which optimizes hash functions via a Siamese network and a mutual difference loss. The knowledge about normality is encoded and stored in hash tables, and the distance between the testing event and the knowledge representation is used to reveal the probability of anomaly. Finally, we fuse the anomaly scores from the two streams to detect anomalies. Extensive experiments demonstrate the effectiveness and complementarity of the two streams, whereby the proposed two-stream framework achieves state-of-the-art performance on ShanghaiTech, Avenue and Corridor datasets among the methods without object detection. Even if compared with the methods using object detection, our method reaches competitive or better performance on the ShanghaiTech, Avenue, and Ped2 datasets.