Locality Sensitive Hashing Research Articles

FSPLO: a fast sensor placement location optimization method for cloud-aided inspection of smart buildings

With the awakening of health awareness, people are raising a series of health-related requirements for the buildings they live in, with a view to improving their living conditions. In this context, BIM (Building Information Modeling) makes full use of cutting-edge theories and technologies in many domains such as health, environment, and information technology to provide a new way for engineers to design and build various healthy and green buildings. Specifically, sensors are playing an important role in achieving smart building goals by monitoring the surroundings of buildings, objects and people with the help of cloud computing technology. In addition, it is necessary to quickly determine the optimal sensor placement to save energy and minimize the number of sensors for a building, which is a de-trial task for the cloud platform due to the limited number of sensors available and massive candidate locations for each sensor. In this paper, we propose a Fast Sensor Placement Location Optimization approach (FSPLO) to solve the BIM problem in cloud-aided smart buildings. In particular, we quickly filter out the repeated candidate locations of sensors in FSPLO using Locality Sensitive Hashing (LSH) techniques to maintain only a small number of optimized locations for deploying sensors around buildings. In this way, we can significantly reduce the number of sensors used for health and green buildings. Finally, a set of simulation experiments demonstrates the excellent performance of our proposed FSPLO method.

LSH-based missing value prediction for abnormal traffic sensors with privacy protection in edge computing

Traffic flow prediction is an important part of intelligent transportation systems (ITS). However, sensor failures or the transmission distortion often occur in the process of data acquisition, which will inevitably cause the loss or abnormality of traffic flow data transmitted to the edge server. In this situation, it is necessary to share traffic flow data among different platforms. However, existing traffic flow prediction methods are facing two challenges in the process of traffic flow data sharing. First, user privacy is often leaked in the process of sharing traffic data on various platforms. Moreover, with the continuous updating of data, the efficiency and scalability of data sharing between different platforms will become lower and lower. In view of the above challenges, in this paper, we propose a novel prediction method for the missing traffic flow data caused by abnormal sensors, named ASMVP_{distr-LSH} based on distributed locality-sensitive hashing (LSH) technique. At last, a case study is presented to illustrate the feasibility and effectiveness of our approach ASMVP_{distr-LSH}.

Double locality sensitive hashing Bloom filter for high-dimensional streaming anomaly detection

Most of the existing large-scale high-dimensional streaming anomaly detection methods suffer from extremely high time and space complexity. Moreover, these models are very sensitive to parameters,make their generalization ability very low, can also be merely applied to very few specific application scenarios. This paper proposes a three-layer structure high-dimensional streaming anomaly detection model, which is called the double locality sensitive hashing Bloom filter, namely dLSHBF. We first build the former two layers that is double locality sensitive hashing (dLSH), proving that the dLSH method reduces the hash coding length of the data, and it ensures that the projected data still has a favorable mapping distance-preserving property after projection. Second, we use a Bloom filter to build the third layer of dLSHBF model, which used to improve the efficiency of anomaly detection. Six large-scale high-dimensional data stream datasets in different IIoT anomaly detection domains were selected for comparison experiments. First, extensive experiments show that the distance-preserving performance of the former dLSH algorithm proposed in this paper is significantly better than the existing LSH algorithms. Second, we verify the dLSHBF model more efficient than the other existing advanced Bloom filter model (for example Robust Bloom Filter, Fly Bloom Filter, Sandwich Learned Bloom Filter, Adaptive Learned Bloom Filters). Compared with the state of the art, dLSHBF can perform with the detection rate (DR) and false alarm rate (FAR) of anomaly detection more than 97%, and less than 2.2% respectively. Its effectiveness and generalization ability outperform other existing streaming anomaly detection methods.

Learnable Locality-Sensitive Hashing for Video Anomaly Detection

Video anomaly detection (VAD) mainly refers to identifying anomalous events that have not occurred in the training set where only normal samples are available. Existing works usually formulate VAD as a reconstruction or prediction problem. However, the adaptability and scalability of these methods are limited. In this paper, we propose a novel distance-based VAD method to take advantage of all the available normal data efficiently and flexibly. In our method, the smaller the distance between a testing sample and normal samples, the higher the probability that the testing sample is normal. Specifically, we propose to use locality-sensitive hashing (LSH) to map the samples whose similarity exceeds a certain threshold into the same bucket in advance. To utilize multiple hashes and further alleviate the computation and memory usage, we propose to use the hash codes rather than the features as the representations of the samples. In this manner, the complexity of near neighbor search is cut down significantly. To make the samples that are semantically similar get closer and those not similar get further apart, we propose a novel learnable version of LSH that embeds LSH into a neural network and optimizes the hash functions with contrastive learning strategy. The proposed method is robust to data imbalance and can handle the large intra-class variations in normal data flexibly. Besides, it has a good ability of scalability. Extensive experiments demonstrate the superiority of our method, which achieves new state-of-the-art results on VAD benchmarks.

Neighbor Graph Based Tensor Recovery For Accurate Internet Anomaly Detection

Detecting anomalous traffic is a crucial task for network management. Although many anomaly detection algorithms have been proposed recently, constrained by their matrix-based traffic data model, existing algorithms often suffer from low detection accuracy. To fully utilize the multi-dimensional information hidden in the traffic data, this paper uses the tensor model for more accurate Internet anomaly detection. Only considering the low-rank linearity features hidden in the data, current tensor factorization techniques would result in low anomaly detection accuracy. We propose a novel Graph-based Tensor Recovery model (Graph-TR) to well explore both low-rank linearity features as well as the non-linear proximity information hidden in the traffic data for better anomaly detection. We encode the non-linear proximity information of the traffic data by constructing nearest neighbor graphs and incorporate this information into the tensor factorization using the graph Laplacian. Moreover, to facilitate the quick building of neighbor graph, we propose a nearest neighbor searching algorithm with the simple locality-sensitive hashing (LSH). Besides only detecting random anomalies, our algorithm can also effectively detect structured anomalies that appear as bursts. We have conducted extensive experiments using Internet traffic trace data Abilene and GÈANT. Compared with the state of art algorithms on matrix-based anomaly detection and tensor recovery approach, our Graph-TR can achieve higher Accuracy and Recall.

Hybrid Cloud-Based Privacy Preserving Clustering as Service for Enterprise Big Data

Clustering as service is being offered by many cloud service providers. It helps enterprises to learn hidden patterns and learn knowledge from large, big data generated by enterprises. Though it brings lot of value to enterprises, it also exposes the data to various security and privacy threats. Privacy preserving clustering is being proposed a solution to address this problem. But the privacy preserving clustering as outsourced service model involves too much overhead on querying user, lacks adaptivity to incremental data and involves frequent interaction between service provider and the querying user. There is also a lack of personalization to clustering by the querying user. This work “Locality Sensitive Hashing for Transformed Dataset (LSHTD)” proposes a hybrid cloud-based clustering as service model for streaming data that address the problems in the existing model such as privacy preserving k-means clustering outsourcing under multiple keys (PPCOM) and secure nearest neighbor clustering (SNNC) models, The solution combines hybrid cloud, LSHTD clustering algorithm as outsourced service model. Through experiments, the proposed solution is able is found to reduce the computation cost by 23% and communication cost by 6% and able to provide better clustering accuracy with ARI greater than 4.59% compared to existing works.

Maintaining Academic Integrity in Programming: Locality-Sensitive Hashing and Recommendations

Not many efficient similarity detectors are employed in practice to maintain academic integrity. Perhaps it is because they lack intuitive reports for investigation, they only have a command line interface, and/or they are not publicly accessible. This paper presents SSTRANGE, an efficient similarity detector with locality-sensitive hashing (MinHash and Super-Bit). The tool features intuitive reports for investigation and a graphical user interface. Further, it is accessible on GitHub. SSTRANGE was evaluated on the SOCO dataset under two performance metrics: f-score and processing time. The evaluation shows that both MinHash and Super-Bit are more efficient than their predecessors (Cosine and Jaccard with 60% less processing time) and a common similarity measurement (running Karp-Rabin greedy string tiling with 99% less processing time). Further, the effectiveness trade-off is still reasonable (no more than 24%). Higher effectiveness can be obtained by tuning the number of clusters and stages. To encourage the use of automated similarity detectors, we provide ten recommendations for instructors interested in employing such detectors for the first time. These include consideration of assessment design, irregular patterns of similarity, multiple similarity measurements, and effectiveness–efficiency trade-off. The recommendations are based on our 2.5-year experience employing similarity detectors (SSTRANGE’s predecessors) in 13 course offerings with various assessment designs.

FARGO: Fast Maximum Inner Product Search via Global Multi-Probing

Maximum inner product search (MIPS) in high-dimensional spaces has wide applications but is computationally expensive due to the curse of dimensionality. Existing studies employ asymmetric transformations that reduce the MIPS problem to a nearest neighbor search (NNS) problem, which can be solved using locality-sensitive hashing (LSH). However, these studies usually maintain multiple hash tables and locally examine them one by one, which may cause additional costs on probing unnecessary points. In addition, LSH is applied without taking into account the properties of the inner product. In this paper, we develop a fast search framework FARGO for MIPS on large-scale, high-dimensional data. We propose a global multi-probing (GMP) strategy that exploits the properties of the inner product to globally examine high quality candidates. In addition, we develop two optimization techniques. First, different with existing transformations that introduce either distortion errors or data distribution imbalances, we design a novel transformation, called random XBOX transformation, that avoids the negative effects of data distribution imbalances. Second, we propose a global adaptive early termination condition that finds results quickly and offers theoretical guarantees. We conduct extensive experiments with real-world data that offer evidence that FARGO is capable of outperforming existing proposals in terms of both accuracy and efficiency.

Image Retrieval Based on ConvNets and Hashing Algorithm

Image retrieval is a prominent subject of study in the fields of image processing and computer vision. With its application in various domains, such as logo search, product search or general image search in Google, Bing, etc., image retrieval has received significant attention for many years. In this work, we study and investigate a framework that leverages visual transferring fea-tures and hashing algorithms for the purpose of finding similar images in a dataset. The key idea of our solution is to find the answer to the following question: “How can we convert an image into binary code and search it more efficiently in a large-scale dataset?”. To achieve this pur-pose, we use pretrained CNN models from ImageNet for image representation and then convert them into binary code by using hashing algorithms. These images in the dataset are represented by binary codes, and the Hamming distance is used to find the images in the dataset that are in-dexed. To demonstrate the robustness of the system, we systematically tested the performance of the system based on speed with raw indexing and hashing indexing on 4 datasets: CIFAR-10, Caltech-101, Oxford-102-Flowers, and MS-COCO 2017. The experimental results show that lo-cal sensitive hashing (LSH) algorithms with 2,048 bits in binary code demonstrate the same or greater precision than raw indexing. Furthermore, the findings show that the MobileNet architec-ture consistently outperforms other architectures across these datasets, effectively balancing speed and precision.

Voting Classifier and Metaheuristic Optimization for Network Intrusion燚etection

Managing physical objects in the network’s periphery is made possible by the Internet of Things (IoT), revolutionizing human life. Open attacks and unauthorized access are possible with these IoT devices, which exchange data to enable remote access. These attacks are often detected using intrusion detection methodologies, although these systems’ effectiveness and accuracy are subpar. This paper proposes a new voting classifier composed of an ensemble of machine learning models trained and optimized using metaheuristic optimization. The employed metaheuristic optimizer is a new version of the whale optimization algorithm (WOA), which is guided by the dipper throated optimizer (DTO) to improve the exploration process ofthe traditional WOA optimizer. The proposed voting classifier categorizes the network intrusions robustly and efficiently. To assess the proposed approach, a dataset created from IoT devices is employed to record the efficiency of the proposed algorithm for binary attack categorization. The dataset records are balanced using the locality-sensitive hashing (LSH) and Synthetic Minority Oversampling Technique (SMOTE). The evaluation of the achieved results is performed in terms of statistical analysis and visual plots to prove the proposed approach’s effectiveness, stability, and significance. The achieved results confirmed the superiority of the proposed algorithm for the task of network intrusion detection.

Global Learnable Attention for Single Image Super-Resolution.

Self-similarity is valuable to the exploration of non-local textures in single image super-resolution (SISR). Researchers usually assume that the importance of non-local textures is positively related to their similarity scores. In this paper, we surprisingly found that when repairing severely damaged query textures, some non-local textures with low-similarity which are closer to the target can provide more accurate and richer details than the high-similarity ones. In these cases, low-similarity does not mean inferior but is usually caused by different scales or orientations. Utilizing this finding, we proposed a Global Learnable Attention (GLA) to adaptively modify similarity scores of non-local textures during training instead of only using a fixed similarity scoring function such as the dot product. The proposed GLA can explore non-local textures with low-similarity but more accurate details to repair severely damaged textures. Furthermore, we propose to adopt Super-Bit Locality-Sensitive Hashing (SB-LSH) as a preprocessing method for our GLA. With the SB-LSH, the computational complexity of our GLA is reduced from quadratic to asymptotic linear with respect to the image size. In addition, the proposed GLA can be integrated into existing deep SISR models as an efficient general building block. Based on the GLA, we constructed a Deep Learnable Similarity Network (DLSN), which achieves state-of-the-art performance for SISR tasks of different degradation types (e.g., blur and noise). Our code and a pre-trained DLSN have been uploaded to GitHub† for validation.

Hybrid Grey Wolf and Dipper Throated Optimization in Network Intrusion Detection Systems

The Internet of Things (IoT) is a modern approach that enables connection with a wide variety of devices remotely. Due to the resource constraints and open nature of IoT nodes, the routing protocol for low power and lossy (RPL) networks may be vulnerable to several routing attacks. That’s why a network intrusion detection system (NIDS) is needed to guard against routing assaults on RPL-based IoT networks. The imbalance between the false and valid attacks in the training set degrades the performance of machine learning employed to detect network attacks. Therefore, we propose in this paper a novel approach to balance the dataset classes based on metaheuristic optimization applied to locality-sensitive hashing and synthetic minority oversampling technique (LSH-SMOTE). The proposed optimization approach is based on a new hybrid between the grey wolf and dipper throated optimization algorithms. To prove the effectiveness of the proposed approach, a set of experiments were conducted to evaluate the performance of NIDS for three cases, namely, detection without dataset balancing, detection with SMOTE balancing, and detection with the proposed optimized LSH-SOMTE balancing. Experimental results showed that the proposed approach outperforms the other approaches and could boost the detection accuracy. In addition, a statistical analysis is performed to study the significance and stability of the proposed approach. The conducted experiments include seven different types of attack cases in the RPL-NIDS17 dataset. Based on the proposed approach, the achieved accuracy is (98.1%), sensitivity is (97.8%), and specificity is (98.8%).

High-Quality Hypergraph Partitioning

Hypergraphs are a generalization of graphs where edges (aka nets ) are allowed to connect more than two vertices. They have a similarly wide range of applications as graphs. This article considers the fundamental and intensively studied problem of balanced hypergraph partitioning (BHP) , which asks for partitioning the vertices into k disjoint blocks of bounded size while minimizing an objective function over the hyperedges. Here, we consider the two most commonly used objectives: the cut-net metric and the connectivity metric . We describe our open-source hypergraph partitioner KaHyPar which is based on the successful multi-level approach—driving it to the extreme of using one level for (almost) every vertex. Using carefully designed data structures and dynamic update techniques, this approach turns out to have a very good time–quality tradeoff. We present two preprocessing techniques— pin sparsification using locality-sensitive hashing (LSH) and community detection based on the Louvain algorithm . The community structure is used to guide the coarsening process that incrementally contracts vertices. Portfolio-based partitioning of the contracted hypergraph then already achieves a good initial solution. While reversing the contraction process, a combination of several refinement techniques achieves a good final partitioning. In particular, we support a highly-localized local search that can directly produce a k -way partitioning and complement this with flow-based techniques that take a more global view. Optionally, a memetic algorithm evolves a pool of solution candidates to an overall good solution. We evaluate KaHyPar for a large set of instances from a wide range of application domains. With respect to quality, KaHyPar outperforms all previously considered systems that can handle large hypergraphs such as hMETIS, PaToH, Mondriaan, or Zoltan. Somewhat surprisingly, to some extend, this even extends to graph partitioners such as KaHIP when considering the special case of graphs. KaHyPar is also faster than most of these systems except for PaToH which represents a different speed–quality tradeoff.

MQH

Point-to-hyperplane nearest neighbor search (P2HNNS) is a fundamental problem which has many applications in data mining and machine learning. In this paper, we propose a provable Locality-Sensitive-Hashing (LSH) scheme based on multi-level quantization errors to solve this problem. In the indexing phase, for each data point, we compute the hash values of its residual vectors generated by a stepwise quantization process. In the query phase, for each processed point, we first determine its suitable level for hashing and then determine the size of hash bucket based on its quantization error in that level. We theoretically show that this treatment not only yields a probability guarantee on query results, but also makes the generated hash functions much more efficient to prune those false points. Experimental results on five real datasets show that the proposed approach generally runs 2X-10X faster than the state-of-the-art LSH-based approaches.

Locality sensitive hashing-driven multifactorial evolutionary algorithms for multitask optimization

In recent years, evolutionary multitasking optimization (EMTO) has attracted considerable attention. Different from the traditional algorithms that solve one optimization problem at a time, EMTO aims at solving multiple problems simultaneously, through transferring useful information among individuals. The most well-known algorithm in this area is the multifactorial evolutionary algorithm (MFEA). However, when designing the information sharing mechanism, MFEA and its variants focus solely on the properties of individuals in the objective space, while neglecting the valuable information in the decision space. To make full use of the inherent knowledge of the evolving population, this paper presents a novel EMTO framework by using the locality sensitive hashing (LSH) method to analyze the population distribution in the decision space and then guide the information sharing. Specifically, LSH uses hash functions to map the individuals in the decision space into hash codes, ensuring that similar individuals have a higher probability to be mapped into the same code. Then, based on the matching relationships of the hash codes and skill factors between individuals, different reproduction and evaluation strategies are exerted to satisfy the specific requirements under different conditions. This way, the knowledge from both the decision space and the objective space are utilized for the purpose of better manipulating the individuals. Additionally, an opposition-based learning method is integrated in this algorithm to enhance exploration. The comprehensive experimental studies confirm the efficacy and efficiency of the proposed method on both single-objective multitask benchmarks and multi-objective multitask benchmarks. According to the statistic tests, the proposed single-objective and multi-objective algorithms significantly outperform their best competitors on 14/18 and 12/18 tasks, respectively.

Privacy-Aware Point-of-Interest Category Recommendation in Internet of Things

In location-based social networks (LBSNs), extensive user check-in data incorporating user preferences for location is collected through Internet of Things devices, including cell phones and other sensing devices. However, directly acquiring the preferences of spars users remains an open challenge. This article offers a point-of-interest (POI) category recommendation model based on group preferences (PPCM). This model is proposed for three reasons: 1) because data influence the training of a deep learning model, the group influence of users is taken into account. To protect the privacy of users’ check-in records and classify similar users into the same group, locality-sensitive hashing (LSH) is used; 2) a successive POI category recommendation model should capture the long- and short-term dependence ability. The attention mechanism and a temporal sliding window are paired with the long short-term memory (LSTM). This paradigm is useful for efficiently mining users’ long-term dependencies and interests; and 3) although the overall users’ check-in data are vast, check-in location options are also massive for a single user. There is a scarcity of data that may be utilized to mine user interests. Thus, instead of using POI, we leverage the POI category to better mine the user’s interests. On real check-in data sets from New York City and Tokyo, the PPCM is compared to other models. The comparison results indicate that the PPCM has improved recommendation performance.

Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search

Lifelong on-device learning is a key challenge for machine intelligence, and this requires learning from few, often single, samples. Memory-augmented neural networks have been proposed to achieve the goal, but the memory module must be stored in off-chip memory, heavily limiting the practical use. In this work, we experimentally validated that all different structures in the memory-augmented neural network can be implemented in a fully integrated memristive crossbar platform with an accuracy that closely matches digital hardware. The successful demonstration is supported by implementing new functions in crossbars, including the crossbar-based content-addressable memory and locality sensitive hashing exploiting the intrinsic stochasticity of memristor devices. Simulations show that such an implementation can be efficiently scaled up for one-shot learning on more complex tasks. The successful demonstration paves the way for practical on-device lifelong learning and opens possibilities for novel attention-based algorithms that were not possible in conventional hardware.

Weakly supervised deep metric learning on discrete metric spaces for privacy-preserved clustering

With an increase in the number of data instances, data processing operations (e.g. clustering) requires an increasing amount of computational resources, and it is often the case that for considerably large datasets such operations cannot be executed on a single workstation. This requires the use of a server computer for carrying out the operations. However, to ensure privacy of the shared data, a privacy preserving data processing workflow involves applying an encoding transformation on the set of data points prior to applying the computation. This encoding should ideally cater to two objectives—first, it should be difficult to reconstruct the data, second, the results of the operation executed on the encoded space should be as close as possible to the results of the same operation executed on the original data. While standard encoding mechanisms, such as locality sensitive hashing, caters to the first objective, the second objective may not always be adequately satisfied.In this paper, we specifically focus on ‘clustering’ as the data processing operation. We apply a deep metric learning approach to learn a parameterized encoding transformation function with an objective to maximize the alignment of the clusters in the encoded space to those in the original data. We conduct experimentation on four standard benchmark datasets, particularly MNIST, Fashion-MNIST (each dataset contains 70K grayscale images), CIFAR-10 consisting of 60K color images and 20-Newsgroups containing 18K news articles. Our experiments demonstrate that the proposed method yields better clusters in comparison to approaches where the encoding process is agnostic of the clustering objective.

Improving binary diffing speed and accuracy using community detection and locality-sensitive hashing: an empirical study

Binary diffing is a commonly used technique for detecting syntactic and semantic similarities and/or differences between two programs’ binary executables (not source code). Here we present REveal, a binary diffing application. REveal is based on the detection of Function Call Graph (FCG) approximate isomorphism and improves both speed and accuracy, mainly by the use of two techniques. First, we propose the use of hierarchical Community Detection (CD) in executables’ FCGs, for the purpose of detecting groups of densely connected functions, thus partitioning them in smaller groups. Moreover, we use Locality-Sensitive Hashing (LSH) for further grouping of similar functions in hash buckets. Both techniques are used in a divide-and-conquer fashion to simplify the diffing process of the programs being compared, practically reducing it to diffing of their FCG communities and LSH buckets.

StateAFL: Greybox fuzzing for stateful network servers

Fuzzing network servers is a technical challenge, since the behavior of the target server depends on its state over a sequence of multiple messages. Existing solutions are costly and difficult to use, as they rely on manually-customized artifacts such as protocol models, protocol parsers, and learning frameworks. The aim of this work is to develop a greybox fuzzer (StateAFL) for network servers that only relies on lightweight analysis of the target program, with no manual customization, in a similar way to what the AFL fuzzer achieved for stateless programs. The proposed fuzzer instruments the target server at compile-time, to insert probes on memory allocations and network I/O operations. At run-time, it infers the current protocol state of the target server by taking snapshots of long-lived memory areas, and by applying a fuzzy hashing algorithm (Locality-Sensitive Hashing) to map memory contents to a unique state identifier. The fuzzer incrementally builds a protocol state machine for guiding fuzzing. We implemented and released StateAFL as open-source software. As a basis for reproducible experimentation, we integrated StateAFL with a large set of network servers for popular protocols, with no manual customization to accomodate for the protocol. The experimental results show that the fuzzer can be applied with no manual customization on a large set of network servers for popular protocols, and that it can achieve comparable, or even better code coverage and bug detection than customized fuzzing. Moreover, our qualitative analysis shows that states inferred from memory better reflect the server behavior than only using response codes from messages.