Approximate K-nearest Neighbor Research Articles

Daniel Huici - Bringing Approximate K-Nearest Neighbor Search to Similarity Digests

Daniel Huici - Bringing Approximate K-Nearest Neighbor Search to Similarity Digests

High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operations

Approximate K nearest neighbor (AKNN) search in the high-dimensional Euclidean vector space is a fundamental and challenging problem. We observe that in high-dimensional space, the time consumption of nearly all AKNN algorithms is dominated by that of the distance comparison operations (DCOs). For each operation, it scans full dimensions of an object and thus, runs in linear time wrt the dimensionality. To speed it up, we propose a randomized algorithm named ADSampling which runs in logarithmic time wrt the dimensionality for the majority of DCOs and succeeds with high probability. In addition, based on ADSampling we develop one generic and two algorithm-specific techniques as plugins to enhance existing AKNN algorithms. Both theoretical and empirical studies confirm that: (1) our techniques introduce nearly no accuracy loss and (2) they consistently improve the efficiency.

HCTree+: A workload-guided index for approximate [formula omitted]NN search

The problem of approximate k-nearest neighbor search in high-dimensional space is a fundamental problem in many applications. We have observed that most existing approaches are unable to leverage the query results to improve the search performance. To address this limitation, we present a new index, called HCTree+, that aims to improve the query performance based on incoming queries and their results. First, we adopt a simple yet effective index to support efficient search. Second, incoming queries and their results are used to optimize the index dynamically for future queries. The experimental study shows that HCTree+ outperforms the state-of-the-art algorithms in terms of accuracy while achieving the desired efficiency.

Robust and efficient single-cell Hi-C clustering with approximate k-nearest neighbor graphs.

MotivationHi-C technology provides insights into the 3D organization of the chromatin, and the single-cell Hi-C method enables researchers to gain knowledge about the chromatin state in individual cell levels. Single-cell Hi-C interaction matrices are high dimensional and very sparse. To cluster thousands of single-cell Hi-C interaction matrices, they are flattened and compiled into one matrix. Depending on the resolution, this matrix can have a few million or even billions of features; therefore, computations can be memory intensive. We present a single-cell Hi-C clustering approach using an approximate nearest neighbors method based on locality-sensitive hashing to reduce the dimensions and the computational resources.ResultsThe presented method can process a 10 kb single-cell Hi-C dataset with 2600 cells and needs 40 GB of memory, while competitive approaches are not computable even with 1 TB of memory. It can be shown that the differentiation of the cells by their chromatin folding properties and, therefore, the quality of the clustering of single-cell Hi-C data is advantageous compared to competitive algorithms.Availability and implementationThe presented clustering algorithm is part of the scHiCExplorer, is available on Github https://github.com/joachimwolff/scHiCExplorer, and as a conda package via the bioconda channel. The approximate nearest neighbors implementation is available via https://github.com/joachimwolff/sparse-neighbors-search and as a conda package via the bioconda channel.Supplementary information Supplementary data are available at Bioinformatics online.

Approximate k-Nearest Neighbor Query of High Dimensional Data Based on Dimension Grouping and Reducing

Approximate k-Nearest Neighbor Query of High Dimensional Data Based on Dimension Grouping and Reducing

On the Merge of k-NN Graph

k-nearest neighbor graph is a fundamental data structure in many disciplines such as information retrieval, data-mining, pattern recognition, and machine learning, etc. In the literature, considerable research has been focusing on how to efficiently build an approximate k-nearest neighbor graph (k-NN graph) for a fixed dataset. Unfortunately, a closely related issue of how to merge two existing k-NN graphs has been overlooked. In this paper, we address the issue of k-NN graph merging in two different scenarios. In the first scenario, a symmetric merge algorithm is proposed to combine two approximate k-NN graphs. The algorithm facilitates large-scale processing by the efficient merging of k-NN graphs that are produced in parallel. In the second scenario, a joint merge algorithm is proposed to expand an existing k-NN graph with a raw dataset. The algorithm enables the incremental construction of a hierarchical approximate k-NN graph. Superior performance is attained when leveraging the hierarchy for NN search of various data types, dimensionality, and distance measures.

Fast and general density peaks clustering

Density peaks is a popular clustering algorithm, used for many different applications, especially for non-spherical data. Although powerful, its use is limited by quadratic time complexity, which makes it slow for large datasets. In this work, we propose a fast density peaks algorithm that solves the time complexity problem. The proposed algorithm uses a fast and generic construction of approximate k-nearest neighbor graph both for density and for delta calculation. This approach maintains the generality of density peaks, which allows using it for all types of data, as long as a distance function is provided. For a dataset of size 100,000, our approach achieves a 91:1 speedup factor. The algorithm scales up for datasets up to 1 million in size, which could not be solved by the original algorithm at all. With the proposed method, time complexity is no longer a limiting factor of the density peaks clustering.

A supervised learning to index model for approximate nearest neighbor image retrieval

Image indexing plays a significant role in approximate K-nearest neighbor image retrieval. In the past decades, the most commonly used methods in image indexing are unsupervised-based. We recognize that unsupervised-based methods cannot benefit from supervised information. In this paper, supervised learning to index method is proposed to realize approximate K-nearest neighbor image retrieval. For designing the supervised learning to index algorithm, we first need to establish the supervision information of an image that should be quantified to the codeword in the codebook. Thus, we propose an image relabeling algorithm to relabel the database images so that the supervised codebook learning can be realized. Second, compared with only distance-based unsupervised image indexing method, a better idea of supervised learning to index is incorporating the distance-based similarities with the classification probabilities of samples at the image indexing stage. In order to achieve this goal, a classifier is also trained based on the relabeled images, and we finally build a constraint optimization objective function to learn the codebook and the classifier simultaneously. The proposed method is verified on the MNIST, the CIFAR-10, and the ImageNet data sets. Experimental results demonstrate the effectiveness of the proposed method.

On approximate k-nearest neighbor searches based on the earth mover’s distance for efficient content-based multimedia information retrieval

The Earth Mover's Distance (EMD) is one of the most-widely used distance functions to measure the similarity between two multimedia objects. While providing good search results, the EMD is too much time consuming to be used in large multimedia databases. To solve the problem, we propose an approximate k-nearest neighbor (k-NN) search method based on the EMD. In the proposed method, the overhead for both disk accesses and EMD computations is reduced significantly, thanks to the approximation. First, the proposed method builds an index using the M-tree, a distance-based multi-dimensional index structure, to reduce the disk access overhead. When building the index, we reduce the number of features in the multimedia objects through dimensionalityreduction. When performing the k-NN search on the M-tree, we find a small set of candidates from the disk using the index and then perform the post-processing on them. Second, the proposed method uses the approximate EMD for index retrieval and post-processing to reduce the computational overhead of the EMD. To compensate the errors due to the approximation, the method provides a way of accuracy improvement of the approximate EMD. We performed extensive experiments to show the efficiency of the proposed method. As a result, the method achieves significant improvement in performance with only small errors: the proposed method outperforms the previous method by up to 67.3% with only 3.5% error.

On approximate k-nearest neighbor searches based on the earth mover’s distance for efficient content-based multimedia information retrieval

On approximate k-nearest neighbor searches based on the earth mover’s distance for efficient content-based multimedia information retrieval

Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

We present a new approach for the approximate K-nearest neighbor search based on navigable small world graphs with controllable hierarchy (Hierarchical NSW, HNSW). The proposed solution is fully graph-based, without any need for additional search structures (typically used at the coarse search stage of the most proximity graph techniques). Hierarchical NSW incrementally builds a multi-layer structure consisting of a hierarchical set of proximity graphs (layers) for nested subsets of the stored elements. The maximum layer in which an element is present is selected randomly with an exponentially decaying probability distribution. This allows producing graphs similar to the previously studied Navigable Small World (NSW) structures while additionally having the links separated by their characteristic distance scales. Starting the search from the upper layer together with utilizing the scale separation boosts the performance compared to NSW and allows a logarithmic complexity scaling. Additional employment of a heuristic for selecting proximity graph neighbors significantly increases performance at high recall and in case of highly clustered data. Performance evaluation has demonstrated that the proposed general metric space search index is able to strongly outperform previous opensource state-of-the-art vector-only approaches. Similarity of the algorithm to the skip list structure allows straightforward balanced distributed implementation.

분산 처리 환경에서 동적 분할 그리드 색인 기반 근사 k-최근접 질의 처리

분산 처리 환경에서 동적 분할 그리드 색인 기반 근사 k-최근접 질의 처리

PANENE: A Progressive Algorithm for Indexing and Querying Approximate k-Nearest Neighbors.

We present PANENE, a progressive algorithm for approximate nearest neighbor indexing and querying. Although the use of k-nearest neighbor (KNN) libraries is common in many data analysis methods, most KNN algorithms can only be queried when the whole dataset has been indexed, i.e., they are not online. Even the few online implementations are not progressive in the sense that the time to index incoming data is not bounded and cannot satisfy the latency requirements of progressive systems. This long latency has significantly limited the use of many machine learning methods, such as t-SNE, in interactive visual analytics. PANENE is a novel algorithm for Progressive Approximate k-NEarest NEighbors, enabling fast KNN queries while continuously indexing new batches of data. Following the progressive computation paradigm, PANENE operations can be bounded in time, allowing analysts to access running results within an interactive latency. PANENE can also incrementally build and maintain a cache data structure, a KNN lookup table, to enable constant-time lookups for KNN queries. Finally, we present three progressive applications of PANENE, such as regression, density estimation, and responsive t-SNE, opening up new opportunities to use complex algorithms in interactive systems.

Distance Encoded Product Quantization for Approximate K-Nearest Neighbor Search in High-Dimensional Space.

Approximate K-nearest neighbor search is a fundamental problem in computer science. The problem is especially important for high-dimensional and large-scale data. Recently, many techniques encoding high-dimensional data to compact codes have been proposed. The product quantization and its variations that encode the cluster index in each subspace have been shown to provide impressive accuracy. In this paper, we explore a simple question: is it best to use all the bit-budget for encoding a cluster index? We have found that as data points are located farther away from the cluster centers, the error of estimated distance becomes larger. To address this issue, we propose a novel compact code representation that encodes both the cluster index and quantized distance between a point and its cluster center in each subspace by distributing the bit-budget. We also propose two distance estimators tailored to our representation. We further extend our method to encode global residual distances in the original space. We have evaluated our proposed methods on benchmarks consisting of GIST, VLAD, and CNN features. Our extensive experiments show that the proposed methods significantly and consistently improve the search accuracy over other tested techniques. This result is achieved mainly because our methods accurately estimate distances.

Privacy-preserving mobile crowd sensing in ad hoc networks

The presence of a rich set of embedded sensors on mobile devices has been propelling various sensing applications regarding individual activities and their surrounding environment, and these persuasive sensing-capable mobile devices are pushing the new paradigm of Mobile Crowd Sensing (MCS) from sketch to reality. MCS aims to outsource sensing data collection to Mobile Device Owner (MDO) and it could revolutionize the conventional ways of sensing data collection and processing. Nonetheless, the widespread deployment of MCS gives rise to the privacy concerns from both the MDOs and the Sensing Service Consumers (SSC), especially in the case where MCS relies on untrustworthy third-party infrastructures. This paper proposes three protocols to address the privacy issues of MCS in ad hoc network without depending on any third-parties. It first presents Privacy-Preserving Summation (PPS) protocol to protect the privacy of the SSCs. Next, it puts forward Privacy-Preserving Difference Rank Computation (PPDRC) protocol to ensure the privacy of the MDOs. Finally, it proposes Approximate K-Nearest Neighbor with Privacy Preservation(AKN2P2) to approximately identify the k-nearest neighbors without privacy leaks of both the MDOs and the SSCs. The performance evaluations demonstrate the computation overhead in different settings.

Towards Secure Approximate <inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math> </inline-formula>-Nearest Neighbor Query Over Encrypted High-Dimensional Data

As many services have been subcontracted to the cloud, data in such services (i.e., management applications, signal processing, and so on) are stored on cloud in encrypted form for protecting user private information. In many emerging data and signal processing applications, approximate k-nearest neighbor (ANN) query is a prerequisite component for massive high-dimensional data (such as time series, biometric, signal, multimedia data, and so on). Unfortunately, existing secure ANN (SANN) methods encounter the limitation of dimensionality. To further resolve ANN query over high-dimensionally encrypted data, in this paper, we propose an effective SANN model in Euclidean space. In overview, a secure greedy partition method is carefully designed by applying locality sensitive hashing coding and an optimized linear order. Based on that, a novel partition-based SANN ( $\text {SANN}_{\text {P}}$ ) and a multi-division version $\text {mSANN}_{\text {P}}$ resolve SANN query by sequentially scanning a candidate set, which is produced by matching a cloak query with a map index. Our proposed solutions guarantee security and accuracy simultaneously, and reduce communication cost significantly. Meanwhile, the greedy partition method is proved to be indistinguishable secure under chosen-plaintext attack , which is the foundation of security for the proposed solutions. Through extensive experimental studies on four data sets, the proposed mechanisms outperform the state-of-the-art approaches and provide effective and tradeoff between result accuracy and communication cost.

Minmax Circular Sector Arc for External Plagiarism’s Heuristic Retrieval stage

Heuristic Retrieval (HR) task aims to retrieve a set of documents from which the External Plagiarism detection identifies plagiarized pieces of text. In this context, we present Minmax Circular Sector Arcs (MinmaxCSA) algorithms that treats HR task as an approximate k-nearest neighbor search problem. Moreover, MinmaxCSA algorithms aim to retrieve the set of documents with greater amounts of plagiarized fragments, while reducing the amount of time to accomplish the HR task. Our theoretical framework is based on two aspects: (i) a triangular property to encode a range of sketches on a unique value; and (ii) a Circular Sector Arc property which enables (i) to be more accurate. Both properties were proposed for handling high-dimensional spaces, hashing them to a lower number of hash values. Our two MinmaxCSA methods, Minmax Circular Sector Arcs Lower Bound (CSAL) and Minmax Circular Sector Arcs Full Bound (CSA), achieved Recall levels slightly more imprecise than Minmaxwise hashing in exchange for a better Speedup in document indexing and query extraction and retrieval time in high-dimensional plagiarism-related datasets.

A reusable and single-interactive model for secure approximate k-nearest neighbor query in cloud

In sensitive database applications (e.g., time series, scientific databases, and biometric), where database is encrypted and outsourced to a public cloud, secure approximate k-nearest neighbor (SANN) query is a fundamental research topic, aiming at retrieving high-dimensional objects that are similar to a given query from encrypted database. To process such queries without ever decrypting the data in cloud is still a challenging task. Existing works encounter various inherent limitations, such as query distinguishability, low-level efficiency and non-recoverability. All of them lead to either fragile security or low accuracy. Hence, the majority of existing works in this field are impractical for industrial applications.In this work, we present a novel model to remove the above limitations. Specifically, a reusable and single-interactive SANN paradigm is proposed in Euclidean high-dimensional space. Firstly, we present a secure variation of B+-tree (i.e., Bc-tree) to quickly locate high-dimensional candidates in cloud by leveraging on comparable encryption. Based on that, an arbitrary query requestor acquires approximate k-nearest neighbors by linearly scanning over candidates. Meanwhile, two refinements, multi-index strategy and boosting refinement strategy, are proposed to further improve the accuracy of search result and overcome the high-dependency of bandwidth, respectively. In the end, through extensive evaluations on four data sets, the proposed mechanisms are demonstrated to be superior in the tradeoff between accuracy and security.

A novel algorithm for scalable k-nearest neighbour graph construction

Finding the k-nearest neighbours of every node in a dataset is one of the most important data operations with wide application in various areas such as recommendation and information retrieval. However, a major challenge is that the execution time of existing approaches grows rapidly as the number of nodes or dimensions increases. In this paper, we present greedy filtering, an efficient and scalable algorithm for finding an approximate k-nearest neighbour graph. It selects a fixed number of nodes as candidates for every node by filtering out node pairs that do not have any matching dimensions with large values. Greedy filtering achieves consistent approximation accuracy across nodes in linear execution time. We also present a faster version of greedy filtering that uses inverted indices on the node prefixes. Through theoretical analysis, we show that greedy filtering is effective for datasets whose features have Zipfian distribution, a characteristic observed in majority of large datasets. We also conduct extensive comparative experiments against (a) three state-of-the-art algorithms, and (b) three algorithms in related research domains. Our experimental results show that greedy filtering consistently outperforms other algorithms in various types of high-dimensional datasets.

Nonlinear Dimensionality Reduction via the ENH-LTSA Method for Hyperspectral Image Classification

The problems of neglecting spatial features in hyperspectral imagery (HSI) and the high complexity of Local Tangent Space Alignment (LTSA) still exist in the nonlinear dimensionality reduction with LTSA for classification. Therefore, this paper proposes an innovative ENH-LTSA (Enhanced-Local Tangent Space Alignment) method to solve the two problems. First, random projection is introduced to preliminarily reduce the dimension of HSI data. It aims to improve the speed of neighbor searching and the local tangent space construction. Then, the new method presents the similarity measure via the adaptive weighted summation kernel (AWSK) distance. The AWSK distance considers both spectral and spatial features in HSI data, and attempts to ameliorate the k-nearest neighbors (KNNs) of each pixel. Furthermore, the adaptive spatial window is proposed to automatically estimate the proper window size for the description of spatial features. After that, fast approximate KNNs graph construction via Recursive Lanczos Bisection is incorporated into the new method to reduce the complexity of KNNs searching. When finishing constructing each local tangent space, the new method uses a fast low-rank approximate singular value decomposition to speed up eigenvalue decomposition of the global alignment matrix that is constituted with local manifold coordinates. Five groups of experiments with two different HSI datasets are designed to completely analyze and testify the ENH-LTSA method. Experimental results show that ENH-LTSA outperforms LTSA, both in classification results and in computational speed.