Locality-sensitive Hash Functions Research Articles

Learning locality-sensitive bucketing functions.

Many tasks in sequence analysis ask to identify biologically related sequences in a large set. The edit distance, being a sensible model for both evolution and sequencing error, is widely used in these tasks as a measure. The resulting computational problem-to recognize all pairs of sequences within a small edit distance-turns out to be exceedingly difficult, since the edit distance is known to be notoriously expensive to compute and that all-versus-all comparison is simply not acceptable with millions or billions of sequences. Among many attempts, we recently proposed the locality-sensitive bucketing (LSB) functions to meet this challenge. Formally, a (d1,d2)-LSB function sends sequences into multiple buckets with the guarantee that pairs of sequences of edit distance at most d1 can be found within a same bucket while those of edit distance at least d2 do not share any. LSB functions generalize the locality-sensitive hashing (LSH) functions and admit favorable properties, with a notable highlight being that optimal LSB functions for certain (d1,d2) exist. LSB functions hold the potential of solving above problems optimally, but the existence of LSB functions for more general (d1,d2) remains unclear, let alone constructing them for practical use. In this work, we aim to utilize machine learning techniques to train LSB functions. With the development of a novel loss function and insights in the neural network structures that can potentially extend beyond this specific task, we obtained LSB functions that exhibit nearly perfect accuracy for certain (d1,d2), matching our theoretical results, and high accuracy for many others. Comparing to the state-of-the-art LSH method Order Min Hash, the trained LSB functions achieve a 2- to 5-fold improvement on the sensitivity of recognizing similar sequences. An experiment on analyzing erroneous cell barcode data is also included to demonstrate the application of the trained LSB functions. The code for the training process and the structure of trained models are freely available at https://github.com/Shao-Group/lsb-learn.

Cryptographically Secure Private Record Linkage using Locality-Sensitive Hashing

Private record linkage (PRL) is the problem of identifying pairs of records that approximately match across datasets in a secure, privacy-preserving manner. Two-party PRL specifically allows each of the parties to obtain records from the other party, only given that each record matches with one of their own. The privacy goal is that no other information about the datasets should be released than the matching records. A fundamental challenge is not to leak information while at the same time not comparing all pairs of records. In plaintext record linkage this is done using a blocking strategy, e.g., locality-sensitive hashing. One recent approach proposed by He et al. (ACM CCS 2017) uses locality-sensitive hashing and then releases a provably differential private representation of the hash bins. However, differential privacy still leaks some, although provable bounded information and does not protect against attacks, such as property inference attacks. Another recent approach by Khurram and Kerschbaum (IEEE ICDE 2020) uses locality-preserving hashing and provides cryptographic security, i.e., it releases no information except the output. However, locality-preserving hash functions are much harder to construct than locality-sensitive hash functions and hence accuracy of this approach is limited, particularly on larger datasets. In this paper, we address the open problem of providing cryptographic security of PRL while using locality-sensitive hash functions. Using recent results in oblivious algorithms, we design a new cryptographically secure PRL with locality-sensitive hash functions. Our prototypical implementation can match 40000 records in the British National Library/Toronto Public Library and the North Carolina Voter Registry datasets with 99.3% and 99.9% accuracy, respectively, in less than an hour which is more than an order of magnitude faster than Khurram and Kerschbaum's work at a higher accuracy.

Locality-sensitive bucketing functions for the edit distance

BackgroundMany bioinformatics applications involve bucketing a set of sequences where each sequence is allowed to be assigned into multiple buckets. To achieve both high sensitivity and precision, bucketing methods are desired to assign similar sequences into the same bucket while assigning dissimilar sequences into distinct buckets. Existing k-mer-based bucketing methods have been efficient in processing sequencing data with low error rates, but encounter much reduced sensitivity on data with high error rates. Locality-sensitive hashing (LSH) schemes are able to mitigate this issue through tolerating the edits in similar sequences, but state-of-the-art methods still have large gaps.ResultsIn this paper, we generalize the LSH function by allowing it to hash one sequence into multiple buckets. Formally, a bucketing function, which maps a sequence (of fixed length) into a subset of buckets, is defined to be (d_1, d_2)-sensitive if any two sequences within an edit distance of d_1 are mapped into at least one shared bucket, and any two sequences with distance at least d_2 are mapped into disjoint subsets of buckets. We construct locality-sensitive bucketing (LSB) functions with a variety of values of (d_1,d_2) and analyze their efficiency with respect to the total number of buckets needed as well as the number of buckets that a specific sequence is mapped to. We also prove lower bounds of these two parameters in different settings and show that some of our constructed LSB functions are optimal.ConclusionThese results lay the theoretical foundations for their practical use in analyzing sequences with high error rates while also providing insights for the hardness of designing ungapped LSH functions.

Locality sensitive hashing via mechanical behavior

From healing wounds to maintaining homeostasis in cyclically loaded tissue, living systems have a phenomenal ability to sense, store, and respond to mechanical stimuli. Broadly speaking, there is significant interest in designing engineered systems to recapitulate this incredible functionality. In engineered systems, we have seen significant recent computationally driven advances in sensing and control. And, there has been a growing interest – inspired in part by the incredible distributed and emergent functionality observed in the natural world – in exploring the ability of engineered systems to perform computation through mechanisms that are fundamentally driven by physical laws. In this work, we focus on a small segment of this broad and evolving field: locality sensitive hashing via mechanical behavior. Specifically, we will address the question: can mechanical information (i.e., loads) be transformed by mechanical systems (i.e., converted into sensor readouts) such that the mechanical system meets the requirements for a locality sensitive hash function? Overall, we not only find that mechanical systems are able to perform this function, but also that different mechanical systems vary widely in their efficacy at this task. Looking forward, we view this work as a starting point for significant future investigation into the design and optimization of mechanical systems for conveying mechanical information for downstream computing.

A Binary-Search-Based Locality-Sensitive Hashing Method for Cross-Site User Identification

In recent years, we have witnessed increasing attention paid to the problem of cross-site user identification (CSUI) with the bloom of social media. Despite noticeable progress in this field, the problem of enormous computation posed by the full-scale pairwise comparison still remains unsolved, which constrains the real application of the state-of-the-art approaches, especially when the user number reaches tens of millions. To address this issue, we propose a novel trajectory-oriented method, which aims at incorporating the locality-sensitive hashing (LSH) technology into the strategy of approximate nearest neighbors searching. Specifically, we design an LSH function based on binary search (BLSH) for processing user trajectory easily, and by searching the BLSH buckets, we cluster similar users to avoid the onerous full-scale pairwise comparison. Moreover, a hierarchy of discrete attention is devised to further improve the effectiveness, which develops our method into the mature one, hierarchically attentioned binary-search-based LSH (HA-BLSH). Extensive experiments on ground-truth datasets demonstrate that our HA-BLSH outperforms the cutting-edge approaches on efficiency and effectiveness. The further discussion indicates that our method is also quite flexible and able to accommodate more scenarios.

Rapid Relevance Feedback Strategy Based on Distributed CBIR System

This article describes the capability of online data storage which has been enhanced by the emergence of cloud datacenter development. Distributed Hash Table (DHT) based image retrieval system using locality sensitive hash (LSH) has provided an efficient way to set up distributed Content Based Image Retrieval (CBIR) frameworks. However, with the fixed LSH function adopted, LSH and other codebook-based distributed retrieval systems are facing the problem of flexibility, and also are difficult to satisfy the user's demand. In this article, LRFMIR is proposed to introduce semantic search into DHT based CBIR system. LRFMIR is established on a DHT based network, where a flexible result truncating strategy is employed to fuse provided results by using multiple features measurements. Experiments show that LRFMIR provides a higher accuracy and recall rate than single feature employed retrieval systems, and possesses good load balancing and query efficiency performance.

Using Locality-Sensitive Hashing for SVM Classification of Large Data Sets

We propose a novel method using Locality-Sensitive Hashing (LSH) for solving the optimization problem that arises in the training stage of support vector machines for large data sets, possibly in high dimensions. LSH was introduced as an efficient way to look for neighbors in high dimensional spaces. Random projections-based LSH functions create bins so that when great probability points belonging to the same bin are close, the points that are far will not be in the same bin. Based on these bins, it is not necessary to consider the whole original set but representatives in each one of them, thus reducing the effective size of the data set. A key of our proposal is that we work with the feature space and use only the projections to search for closeness in this space. Moreover, instead of choosing the projection directions at random, we sample a small subset and solve the associated SVM problem. Projections in this direction allows for a more precise sample in many cases and an approximation of the solution of the large problem is found in a fraction of the running time with small degradation of the classification error. We present two algorithms, theoretical support, and numerical experiments showing their performances on real life problems taken from the LIBSVM data base.

A Three-stage Filtering Approach for Face Recognition

Face recognition has become a crucial topic in recent decades, which offers important opportunities for applications in security surveillance, human-computer interaction, and forensics. However, it poses challenges, including uncontrolled environments, large datasets, and insufficiency of training data. In this paper, a face recognition system is proposed to iron out the above problems with a new framework based on a hashing function in a three-stage filtering approach. At the first stage, candidate subjects are chosen using the Locality-Sensitive Hashing (LSH) function. We employ a voting system to select candidates via disregarding a large number of dissimilar identities considering their local features. At the second stage, a robust image hashing based on Discrete Cosine Transform (DCT) coefficients is used to further refine the candidate images in terms of global visual information. Finally, the test image is recognized among selected identities using other visual information, resulting in further accuracy gains. Extensive experiments on FERET, AR, and ORL datasets show that the proposed method outperforms with a significant improvement in accuracy over the state-of-the-art methods.

Secure Multi-keyword Similarity Search Over Encrypted Data With Security Improvement

Searchable encryption (SE) is an interesting tool that enables clients to outsource their encrypted data into external cloud servers with unlimited storage and computing power and gives them the ability to search their data without decryption. The current solutions of SE support single-keyword search making them impractical in real-world scenarios. In this paper, we design and implement a multi-keyword similarity search scheme over encrypted data by using locality-sensitive hashing functions and Bloom filter. The proposed scheme can recover common spelling mistakes and enjoys enhanced security properties such as hiding the access and search patterns but with costly latency. To support similarity search, we utilize an efficient bi-gram-based method for keyword transformation. Such a method improves the search results accuracy. Our scheme employs two non-colluding servers to break the correlation between search queries and search results. Experiments using real-world data illustrate that our scheme is practically efficient, secure, and retains high accuracy.

Semantic-based Compound Keyword Search over Encrypted Cloud Data

Keyword search over encrypted data is essential for accessing outsourced sensitive data in cloud computing. In some circumstances, the keywords that the user searches on are only semantically related to the data rather than via an exact or fuzzy match. Hence, semantic-based keyword search over encrypted cloud data becomes of paramount importance. However, existing schemes usually depend upon a global dictionary, which not only affects the accuracy of search results but also causes inefficiency in data updating. Additionally, although compound keyword search is common in practice, the existing approaches only process them as single words, which split the original semantics and achieve low accuracy. To address these limitations, we initially propose a compound concept semantic similarity (CCSS) calculation method to measure the semantic similarity between compound concepts. Next, by integrating CCSS with Locality-Sensitive Hashing function and the secure kk<; inline-graphic xlink:href="wang-ieq1-2847318.gif"/>-Nearest Neighbor scheme, a semantic-based compound keyword search (SCKS) scheme is proposed. SCKS achieves not only semantic-based search but also multi-keyword search and ranked keyword search. Additionally, SCKS also eliminates the predefined global library and can efficiently support data update. The experimental results on real-world dataset indicate that SCKS introduces low overhead on computation and the search accuracy outperforms the existing schemes.

Site2Vec: a reference frame invariant algorithm for vector embedding of protein–ligand binding sites

Protein–ligand interactions are one of the fundamental types of molecular interactions in living systems. Ligands are small molecules that interact with protein molecules at specific regions on their surfaces called binding sites. Binding sites would also determine ADMET properties of a drug molecule. Tasks such as assessment of protein functional similarity and detection of side effects of drugs need identification of similar binding sites of disparate proteins across diverse pathways. To this end, methods for computing similarities between binding sites are still evolving and is an active area of research even today. Machine learning methods for similarity assessment require feature descriptors of binding sites. Traditional methods based on hand engineered motifs and atomic configurations are not scalable across several thousands of sites. In this regard, deep neural network algorithms are now deployed which can capture very complex input feature space. However, one fundamental challenge in applying deep learning to structures of binding sites is the input representation and the reference frame. We report here a novel algorithm, Site2Vec, that derives reference frame invariant vector embedding of a protein–ligand binding site. The method is based on pairwise distances between representative points and chemical compositions in terms of constituent amino acids of a site. The vector embedding serves as a locality sensitive hash function for proximity queries and determining similar sites. The method has been the top performer with more than 95% quality scores in extensive benchmarking studies carried over 10 data sets and against 23 other site comparison methods in the field. The algorithm serves for high throughput processing and has been evaluated for stability with respect to reference frame shifts, coordinate perturbations and residue mutations. We also provide the method as a standalone executable and a web service hosted at (http://services.iittp.ac.in/bioinfo/home).

Efficient locality-sensitive hashing over high-dimensional streaming data

Approximate nearest neighbor (ANN) search in high-dimensional spaces is fundamental in many applications. Locality-sensitive hashing (LSH) is a well-known methodology to solve the ANN problem. Existing LSH-based ANN solutions typically employ a large number of individual indexes optimized for searching efficiency. Updating such indexes might be impractical when processing high-dimensional streaming data. In this paper, we present a novel disk-based LSH index that offers efficient support for both searches and updates. The contributions of our work are threefold. First, we use the write-friendly LSM-trees to store the LSH projections to facilitate efficient updates. Second, we develop a novel estimation scheme to estimate the number of required LSH functions, with which the disk storage and access costs are effectively reduced. Third, we exploit both the collision number and the projection distance to improve the efficiency of candidate selection, improving the search performance with theoretical guarantees on the result quality. Experiments on four real-world datasets show that our proposal outperforms the state-of-the-art schemes.

A Fast and Memory-Efficient Spectral Library Search Algorithm Using Locality-Sensitive Hashing.

With the accumulation of MS/MS spectra collected in spectral libraries, the spectral library searching approach emerges as an important approach for peptide identification in proteomics, complementary to the commonly used protein database searching approach, in particular for the proteomic analyses of well-studied model organisms, such as human. Existing spectral library searching algorithms compare a query MS/MS spectrum with each spectrum in the library with matched precursor mass and charge state, which may become computationally intensive with the rapidly growing library size. Here, the software msSLASH, which implements a fast spectral library searching algorithm based on the Locality-Sensitive Hashing (LSH) technique, is presented. The algorithm first converts the library and query spectra into bit-strings using LSH functions, and then computes the similarity between the spectra with highly similar bit-string. Using the spectral library searching of large real-world MS/MS spectra datasets, it is demonstrated that the algorithm significantly reduced the number of spectral comparisons, and as a result, achieved 2-9X speedup in comparison with existing spectral library searching algorithm SpectraST. The spectral searching algorithm is implemented in C/C++, and is ready to be used in proteomic data analyses.

Accurate and Fast Recovery of Network Monitoring Data: A GPU Accelerated Matrix Completion

Gaining a full knowledge of end-to-end network performance is important for some advanced network management and services. Although it becomes increasingly critical, end-to-end network monitoring usually needs active probing of the path and the overhead will increase quadratically with the number of network nodes. To reduce the measurement overhead, matrix completion is proposed recently to predict the end-to-end network performance among all node pairs by only measuring a small set of paths. Despite its potential, applying matrix completion to recover the missing data suffers from low recovery accuracy and long recovery time. To address the issues, we propose MC-GPU to exploit Graphics Processing Units (GPUs) to enable parallel matrix factorization for high-speed and highly accurate Matrix Completion. To well exploit the special architecture features of GPUs for both task independent and data-independent parallel task execution, we propose several novel techniques: similar OD (origin and destination) pairs reordering taking advantage of the locality-sensitive hash (LSH) functions, balanced matrix partition, and parallel matrix completion. We implement the proposed MC-GPU on the GPU platform and evaluate the performance using real trace data. We compare the proposed MC-GPU with the state of the art matrix completion algorithms, and our results demonstrate that MC-GPU can achieve significantly faster speed with high data recovery accuracy.

Research on power-law distribution of long-tail data and its application to tourism recommendation

PurposeOne challenge for tourism recommendation systems (TRSs) is the long-tail phenomenon of ratings or popularity among tourist products. This paper aims to improve the diversity and efficiency of TRSs utilizing the power-law distribution of long-tail data.Design/methodology/approachUsing Sina Weibo check-in data for example, this paper demonstrates that the long-tail phenomenon exists in user travel behaviors and fits the long-tail travel data with power-law distribution. To solve data sparsity in the long-tail part and increase recommendation diversity of TRSs, the paper proposes a collaborative filtering (CF) recommendation algorithm combining with power-law distribution. Furthermore, by combining power-law distribution with locality sensitive hashing (LSH), the paper optimizes user similarity calculation to improve the calculation efficiency of TRSs.FindingsThe comparison experiments show that the proposed algorithm greatly improves the recommendation diversity and calculation efficiency while maintaining high precision and recall of recommendation, providing basis for further dynamic recommendation.Originality/valueTRSs provide a better solution to the problem of information overload in the tourism field. However, based on the historical travel data over the whole population, most current TRSs tend to recommend hot and similar spots to users, lacking in diversity and failing to provide personalized recommendations. Meanwhile, the large high-dimensional sparse data in online social networks (OSNs) brings huge computational cost when calculating user similarity with traditional CF algorithms. In this paper, by integrating the power-law distribution of travel data and tourism recommendation technology, the authors’ work solves the problem existing in traditional TRSs that recommendation results are overly narrow and lack in serendipity, and provides users with a wider range of choices and hence improves user experience in TRSs. Meanwhile, utilizing locality sensitive hash functions, the authors’ work hashes users from high-dimensional vectors to one-dimensional integers and maps similar users into the same buckets, which realizes fast nearest neighbors search in high-dimensional space and solves the extreme sparsity problem of high dimensional travel data. Furthermore, applying the hashing results to user similarity calculation, the paper greatly reduces computational complexity and improves calculation efficiency of TRSs, which reduces the system load and enables TRSs to provide effective and timely recommendations for users.

A Cryptanalysis of Two Cancelable Biometric Schemes Based on Index-of-Max Hashing

Cancelable biometric schemes generate secure biometric templates by combining user specific tokens and biometric data. The main objective is to create irreversible, unlinkable, and revocable templates, with high accuracy of comparison. In this paper, we cryptanalyze two recent cancelable biometric schemes based on a particular locality sensitive hashing function, index-of-max (IoM): Gaussian Random Projection-IoM (GRP-IoM) and Uniformly Random Permutation-IoM (URP-IoM). As originally proposed, these schemes were claimed to be resistant against reversibility, authentication, and linkability attacks under the stolen token scenario. We propose several attacks against GRP-IoM and URP-IoM, and argue that both schemes are severely vulnerable against authentication and linkability attacks. We also propose better, but not yet practical, reversibility attacks against GRP-IoM. The correctness and practical impact of our attacks are verified over the same dataset provided by the authors of these two schemes.

Privacy-Preserving Image Retrieval and Sharing in Social Multimedia Applications

Every day social multimedia applications generate millions of images. To handle such huge amount of images, an optimal solution is using the public cloud, since it has powerful storage capability. Images usually contain a wealth of sensitive information, therefore social service providers need not only to provide services such as retrieval and sharing but also to protect the privacies of the images. In this paper, we propose a privacy-preserving scheme for content-based image retrieval and sharing in social multimedia applications. First, the users extract visual features from the images, and perform locality-sensitive hashing functions on visual features to generate image profile vectors. We then model the retrieval on the images as the equality search on the image profile vectors. To enable accurate and efficient retrieval, we design the secure index structure based on cuckoo hashing, which has constant lookup time. To meet the requirements of dynamic image updating, we enrich our service with image insertion and deletion. In order to reduce the key management overhead and the access control overhead in social applications, we process keys using secret sharing techniques to enable the users holding similar images to query and decrypt images independently. Finally we implement the prototype of the proposed scheme, and perform experiments over encrypted image databases.

Symmetric keyring encryption scheme for biometric cryptosystem

In this paper, we propose a novel biometric cryptosystem for vectorial biometrics called symmetric keyring encryption (SKE), inspired by Rivest's keyring model (2016). Unlike conventional biometric secret-binding primitives, such as fuzzy commitment and fuzzy vault approaches, the proposed scheme reframes the biometric secret-binding problem as a fuzzy symmetric encryption problem using a concept called a resilient vector pair. In this study, this pair resembles the encryption–decryption key pair in symmetric key cryptosystems. This scheme is realized using an index of maximum hashed vectors, a special instance of the ranking-based locality-sensitive hashing function. With a simple filtering mechanism and an [m, k] Shamir's secret-sharing scheme, we show that SKE, both in theory and in an empirical evaluation, can retrieve the exact secret with overwhelming probability for a genuine input yet negligible probability for an imposter input. Although SKE can be applied to any vectorial biometrics, we adopt fingerprint and face vectors in this work. Experiments were performed using the Fingerprint Verification Competition (FVC) and Labeled Face in the Wild (LFW) datasets. We formalize and analyze the threat model for SKE, which involves several major security attacks.

Hamming Metric Multi-Granularity Locality-Sensitive Bloom Filter

A Bloom filter is a type of space-efficient data structure that supports membership tests in numerous network applications. Recently, emerging applications require an approximate membership test (AMT) rather than conventional (exact-matching) membership test. Some AMT problems can be effectively solved by using a locality-sensitive hashing (LSH) based Bloom filter. However, existing work cannot handle changing Hamming distances. In this paper, we present a new Hamming metric locality-sensitive Bloom filter (HLBF) to tackle the challenge. Each object of the data set is hashed by bit sampling LSH functions and encoded into a standard Bloom filter in the HLBF structure. To support AMTs with different given Hamming distances, we propose a multi-granularity test algorithm (called the M-HLBF) based on the HLBF and virtual objects which are created from the given test object. Theoretical analyses show that false positive rates and false negative rates can be controlled within low levels. To further accelerate the processing of an AMT, we also illustrate a hardware implementation. Extensive experimental results demonstrate that our method is quite promising in achieving high efficiency and flexibility for processing AMTs with different granularities/distances.

CoveringLSH

We consider a new construction of locality-sensitive hash functions for Hamming space that is covering in the sense that is it guaranteed to produce a collision for every pair of vectors within a given radius r . The construction is efficient in the sense that the expected number of hash collisions between vectors at distance cr , for a given c &gt;1, comes close to that of the best possible data independent LSH without the covering guarantee, namely, the seminal LSH construction of Indyk and Motwani (STOC’98). The efficiency of the new construction essentially matches their bound when the search radius is not too large—e.g., when cr = o (log ( n )/ log log n ), where n is the number of points in the dataset, and when cr = log ( n )/ k , where k is an integer constant. In general, it differs by at most a factor ln (4) in the exponent of the time bounds. As a consequence, LSH-based similarity search in Hamming space can avoid the problem of false negatives at little or no cost in efficiency.