Approximate Nearest Neighbor Research Articles

Refining Cellular Data for Accurate Trip Chain Identification: A Novel Approach for Urban Travel Analysis

Cellular data play a crucial role in supporting travel demand assessment and urban traffic planning because of their cost-effectiveness and extensive coverage. However, inherent inaccuracies, such as imprecise positioning, data duplication, and abnormal communication frequencies, hinder their ability to depict travelers’ trips accurately. In this paper, we present a novel approach to mitigate these issues by employing base station mapping to refine positioning data, eliminating duplicates and abnormal frequency records. This refinement enables more effective utilization of cellular data for trip chain identification. We address the ping-pong handover effect by employing a finite automaton machine and an approximate nearest neighbor searching method with carefully selected seeds to identify activities. Our method’s accuracy is validated through ground truth value analysis, focusing on permanent resident population estimation and metro passenger flow estimation. Furthermore, we demonstrate the practical effectiveness and value of our proposed method through a series of representative applications in a real-world case study in the city of Nanning, Guangxi Province, China.

An efficient infinity norm minimization algorithm for under-determined inverse problems

The problem of solving under-determined systems of linear equations with minimum peak magnitude (ℓ∞ norm) has numerous applications in signal processing. These include Peak-to-Average Power Ratio (PAPR) reduction in MIMO-OFDM systems, vector quantization, approximate nearest neighbor search, optimal control in robotics, and power grid optimization. Several methods have been proposed to address this problem, but they often face limitations in computational speed or representation accuracy. Some methods also impose constraints on the frame matrix, such as restrictions on the type of its entries or its aspect ratio. In this paper, we present the Fast Iterative Peak Shrinkage Algorithm (FIPSA), which iterates over feasible solutions to consistently reduce peak magnitude and provably converge to near-optimal solutions. Our experimental results, conducted across various frame matrix types and aspect ratios, demonstrate that FIPSA consistently achieves near-minimal ℓ∞ norm values. In addition, it operates 1.3 to 7.3 times faster than previous methods, while maintaining an average representation error of 10−15. Notably, these advancements are achieved without imposing any constraints on the frame matrix.

In-memory search with learning to hash based on resistive memory for recommendation acceleration

Similarity search is essential in current artificial intelligence applications and widely utilized in various fields, such as recommender systems. However, the exponential growth of data poses significant challenges in search time and energy consumption on traditional digital hardware. Here, we propose a software-hardware co-optimization to address these challenges. On the software side, we employ a learning-to-hash method for vector encoding and achieve an approximate nearest neighbor search by calculating Hamming distance, thereby reducing computational complexity. On the hardware side, we leverage the resistance random-access memory crossbar array to implement the hash encoding process and the content-addressable memory with an in-memory computing paradigm to lower the energy consumption during searches. Simulations on the MovieLens dataset demonstrate that the implementation achieves comparable accuracy to software and reduces energy consumption by 30-fold compared to traditional digital systems. These results provide insight into the development of energy-efficient in-memory search systems for edge computing.

Binning Metagenomic Contigs Using Contig Embedding and Decomposed Tetranucleotide Frequency.

Metagenomic binning is a crucial step in metagenomic research. It can aggregate the genome sequences belonging to the same microbial species into independent bins. Most existing methods ignore the semantic information of contigs and lack effective processing of tetranucleotide frequency, resulting in insufficient and complex feature information extracted for binning and poor binning results. To address the above problems, we propose CedtBin, a metagenomic binning method based on contig embedding and decomposed tetranucleotide frequency. First, the improved BERT model is used to learn the contigs to obtain their embedding representation. Secondly, the tetranucleotide frequencies are decomposed using a non-negative matrix factorization (NMF) algorithm. After that, the two features are spliced and input into the clustering algorithm for binning. Considering the sensitivity of the DBSCAN clustering algorithm to input parameters, in order to solve the drawbacks of manual parameter input, we also propose an Annoy-DBSCAN algorithm that can adaptively determine the parameters of the DBSCAN algorithm. This algorithm uses Approximate Nearest Neighbors Oh Yeah (Annoy) and combines it with a grid search strategy to find the optimal parameters of the DBSCAN algorithm. On simulated and real datasets, CedtBin achieves better binning results than mainstream methods and can reconstruct more genomes, indicating that the proposed method is effective.

Spatial-Aware and Load Balancing Distributed Data Partitioning Strategies for Content-Based Multimedia Retrieval.

Content-Based Multimedia Retrieval (CBMR) has become very popular in several applications, driven by the growing routine use of multimedia data. Since the datasets used in real-world applications are very large and descriptor's dimensionality is high, querying is an expensive, albeit important functionality. Further, exact search is prohibitive in most cases, motivating the use of Approximate Nearest Neighbour Search (ANNS) algorithms, trading accuracy for performance. These have been mainly developed targeting a sequential execution in a single node. However, the large and increasing datasets used and the high query loads submitted to those systems typically surpass the memory and computing resources available in a single node. This motivated the development of parallel distributed memory ANNS solutions to meet the computing capabilities required by those applications. A common problem that must be handled when using distributed memory systems is data partitioning and its impact on load imbalance. Several data partitioning approaches have already been proposed, including elaborated spatial-aware strategies. However, little effort has been put into carefully analyzing the performance of those strategies at scale. Here, we evaluated the commonly used data partitioning strategies in ANNS and identified their limitations to propose a novel class of partitioning algorithms that can minimize load imbalance while improving data locality to attain high performance on the distributed memory search. Experimentally, we found that our proposed algorithms (SABBS and SABBSR) improved search performance by up to 1.64× compared to the best previous solution. In a distributed memory weak scaling evaluation, with up to 12 billion 128-dimensional descriptors and 60 compute nodes, the gains were maintained as the system scaled with our novel approaches. These results demonstrate the efficiency of our new algorithms for billion-scale ANNS and the importance of considering not only data locality but also data and load imbalance in the data partitioning.

Indonesian-English Textual Similarity Detection Using Universal Sentence Encoder (USE) and Facebook AI Similarity Search (FAISS)

The tremendous development in Natural Language Processing (NLP) has enabled the detection of bilingual and multilingual textual similarity. One of the main challenges of the Textual Similarity Detection (TSD) system lies in learning effective text representation. The research focuses on identifying similar texts between Indonesian and English across a broad range of semantic similarity spectrums. The primary challenge is generating English and Indonesian dense vector representation, a.k.a. embeddings that share a single vector space. Through trial and error, the research proposes using the Universal Sentence Encoder (USE) model to construct bilingual embeddings and FAISS to index the bilingual dataset. The comparison between query vectors and index vectors is done using two approaches: the heuristic comparison with Euclidian distance and a clustering algorithm, Approximate Nearest Neighbors (ANN). The system is tested with four different semantic granularities, two text granularities, and evaluation metrics with a cutoff value of k={2,10}. Four semantic granularities used are highly similar or near duplicate, Semantic Entailment (SE), Topically Related (TR), and Out of Topic (OOT), while the text granularities take on the sentence and paragraph levels. The experimental results demonstrate that the proposed system successfully ranks similar texts in different languages within the top ten. It has been proven by the highest F1@2 score of 0.96 for the near duplicate category on the sentence level. Unlike the near-duplicate category, the highest F1 scores of 0.77 and 0.89 are shown by the SE and TR categories, respectively. The experiment results also show a high correlation between text and semantic granularity.

Stitching method for panoramic nail fold images based on capillary contour enhancement.

Nail fold capillaroscopy is an important means of monitoring human health. Panoramic nail fold images improve the efficiency and accuracy of examinations. However, the acquisition of panoramic nail fold images is seldom studied and the problem manifests of few matching feature points when image stitching is used for such images. Therefore, this paper presents a method for panoramic nail fold image stitching based on vascular contour enhancement, which first solves the problem of few matching feature points by pre-processing the image with contrast-constrained adaptive histogram equalization (CLAHE), bilateral filtering (BF), and sharpening algorithms. The panoramic images of the nail fold blood vessels are then successfully stitched using the fast robust feature (SURF), fast library of approximate nearest neighbors (FLANN) and random sample agreement (RANSAC) algorithms. The experimental results show that the panoramic image stitched by this paper's algorithm has a field of view width of 7.43 mm, which improves the efficiency and accuracy of diagnosis.

Similarity search on social networks with incremental graph indexing based on probabilistic inference

PurposeThis paper aims to propose an incremental graph indexing method based on probabilistic inferences in Bayesian network (BN) for approximate nearest neighbor search (ANNS) that adds unindexed queries into the graph index incrementally.Design/methodology/approachThis paper first uses the attention mechanism based graph convolutional network to embed a social network into the low-dimensional vector space, which could improve the efficiency of graph index construction. To add the unindexed queries into the graph index incrementally, this study proposes to learn the rule-based BN from social interactions. Thus, the dependency relations of unindexed queries and their neighbors are represented, and the probabilistic inferences in BN are then performed.FindingsExperimental results demonstrate that the proposed method improves the search precision by at least 5% and search efficiency by 10% compared to the state-of-the-art methods.Originality/valueThis paper proposes a novel method to construct the incremental graph index based on probabilistic inferences in BN, such that both indexed and unindexed queries in ANNS could be addressed efficiently.

RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search

Searching for approximate nearest neighbors (ANN) in the high-dimensional Euclidean space is a pivotal problem. Recently, with the help of fast SIMD-based implementations, Product Quantization (PQ) and its variants can often efficiently and accurately estimate the distances between the vectors and have achieved great success in the in-memory ANN search. Despite their empirical success, we note that these methods do not have a theoretical error bound and are observed to fail disastrously on some real-world datasets. Motivated by this, we propose a new randomized quantization method named RaBitQ, which quantizes D-dimensional vectors into D-bit strings. RaBitQ guarantees a sharp theoretical error bound and provides good empirical accuracy at the same time. In addition, we introduce efficient implementations of RaBitQ, supporting to estimate the distances with bitwise operations or SIMD-based operations. Extensive experiments on real-world datasets confirm that (1) our method outperforms PQ and its variants in terms of accuracy-efficiency trade-off by a clear margin and (2) its empirical performance is well-aligned with our theoretical analysis.

ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data

Applications increasingly leverage mixed-modality data, and must jointly search over vector data, such as embedded images, text and video, as well as structured data, such as attributes and keywords. Proposed methods for this hybrid search setting either suffer from poor performance or support a severely restricted set of search predicates (e.g., only small sets of equality predicates), making them impractical for many applications. To address this, we present ACORN, an approach for performant and predicate-agnostic hybrid search. ACORN builds on Hierarchical Navigable Small Worlds (HNSW), a state-of-the-art graph-based approximate nearest neighbor index, and can be implemented efficiently by extending existing HNSW libraries. ACORN introduces the idea of predicate subgraph traversal to emulate a theoretically ideal, but impractical, hybrid search strategy. ACORN's predicate-agnostic construction algorithm is designed to enable this effective search strategy, while supporting a wide array of predicate sets and query semantics. We systematically evaluate ACORN on both prior benchmark datasets, with simple, low-cardinality predicate sets, and complex multi-modal datasets not supported by prior methods. We show that ACORN achieves state-of-the-art performance on all datasets, outperforming prior methods with 2--1,000× higher throughput at a fixed recall. Our code is available at: https://github.com/stanford-futuredata/ACORN.

DET-LSH: A Locality-Sensitive Hashing Scheme with Dynamic Encoding Tree for Approximate Nearest Neighbor Search

Locality-sensitive hashing (LSH) is a well-known solution for approximate nearest neighbor (ANN) search in high-dimensional spaces due to its robust theoretical guarantee on query accuracy. Traditional LSH-based methods mainly focus on improving the efficiency and accuracy of the query phase by designing different query strategies, but pay little attention to improving the efficiency of the indexing phase. They typically fine-tune existing data-oriented partitioning trees to index data points and support their query strategies. However, their strategy to directly partition the multi-dimensional space is time-consuming, and performance degrades as the space dimensionality increases. In this paper, we design an encoding-based tree called Dynamic Encoding Tree (DE-Tree) to improve the indexing efficiency and support efficient range queries based on Euclidean distance. Based on DE-Tree, we propose a novel LSH scheme called DET-LSH. DET-LSH adopts a novel query strategy, which performs range queries in multiple independent index DE-Trees to reduce the probability of missing exact NN points, thereby improving the query accuracy. Our theoretical studies show that DET-LSH enjoys probabilistic guarantees on query accuracy. Extensive experiments on real-world datasets demonstrate the superiority of DET-LSH over the state-of-the-art LSH-based methods on both efficiency and accuracy. While achieving better query accuracy than competitors, DET-LSH achieves up to 6x speedup in indexing time and 2x speedup in query time over the state-of-the-art LSH-based methods.

Toward Cost-Effective Adaptive Random Testing: An Approximate Nearest Neighbor Approach

Toward Cost-Effective Adaptive Random Testing: An Approximate Nearest Neighbor Approach

A Large-Crack Image-Stitching Method with Cracks as the Regions of Interest

While crack detection is crucial for maintaining concrete structures, existing methods often overlook the analysis of large cracks that span multiple images. Such analyses typically rely on image stitching to create a complete image of a crack. Current stitching methods are not only computationally demanding but also require manual adjustments; thus, a fast and reliable solution is still lacking. To address these challenges, we introduce a stitching method that leverages the advantages of crack image-segmentation models. This method first utilizes the Mask R-CNN model for the identification of crack regions as regions of interest (ROIs) within images. These regions are then used to calculate keypoints of the scale-invariant feature transform (SIFT), and descriptors for these keypoints are computed with the original images for image matching and stitching. Compared with traditional methods, our approach significantly reduces the computational time; by 98.6% in comparison to the Brute Force (BF) matcher, and by 58.7% with respect to the Fast Library for Approximate Nearest Neighbors (FLANN) matcher. Our stitching results on images with different degrees of overlap or changes in shooting posture show superior structural similarity index (SSIM) values, demonstrating excellent detail-matching performance. Moreover, the ability to measure complete crack images is indicated by the relative error of 7%, which is significantly better than that of traditional methods.

Continuous monitoring of reverse approximate nearest neighbour queries on road network

Reverse Approximate Nearest Neighbour (RANN) query relaxes the RkNN definition of influence, where a user u can be influenced by not only its closest facility but also by every other facility that is almost as close to u as its closest facility is. In this paper, we study the continuous monitoring of RANN queries on road network. Existing continuous RANN algorithms on Euclidean space cannot be extended to continuously monitor RANN queries on road network. We propose two different methods to efficiently monitor RANN queries. We conduct an extensive experiment on different real data sets and demonstrate that our both proposed algorithms are significantly better than the competitor

Terminal Embeddings in Sublinear Time

Recently (Elkin, Filtser, Neiman 2017) introduced the concept of a {\it terminal embedding} from one metric space $(X,d_X)$ to another $(Y,d_Y)$ with a set of designated terminals $T\subset X$. Such an embedding $f$ is said to have distortion $\rho\ge 1$ if $\rho$ is the smallest value such that there exists a constant $C>0$ satisfying \begin{equation*} \forall x\in T\ \forall q\in X,\ C d_X(x, q) \le d_Y(f(x), f(q)) \le C \rho d_X(x, q) . \end{equation*} When $X,Y$ are both Euclidean metrics with $Y$ being $m$-dimensional, recently (Narayanan, Nelson 2019), following work of (Mahabadi, Makarychev, Makarychev, Razenshteyn 2018), showed that distortion $1+\epsilon$ is achievable via such a terminal embedding with $m = O(\epsilon^{-2}\log n)$ for $n := |T|$. This generalizes the Johnson-Lindenstrauss lemma, which only preserves distances within $T$ and not to $T$ from the rest of space. The downside of prior work is that evaluating their embedding on some $q\in \mathbb{R}^d$ required solving a semidefinite program with $\Theta(n)$ constraints in~$m$ variables and thus required some superlinear $\mathrm{poly}(n)$ runtime. Our main contribution in this work is to give a new data structure for computing terminal embeddings. We show how to pre-process $T$ to obtain an almost linear-space data structure that supports computing the terminal embedding image of any $q\in\mathbb{R}^d$ in sublinear time $O^* (n^{1-\Theta(\epsilon^2)} + d)$. To accomplish this, we leverage tools developed in the context of approximate nearest neighbor search.

SeRF: Segment Graph for Range-Filtering Approximate Nearest Neighbor Search

Effective vector representation models, e.g., word2vec and node2vec, embed real-world objects such as images and documents in high dimensional vector space. In the meanwhile, the objects are often associated with attributes such as timestamps and prices. Many scenarios need to jointly query the vector representations of the objects together with their attributes. These queries can be formalized as range-filtering approximate nearest neighbor search (ANNS) queries. Specifically, given a collection of data vectors, each associated with an attribute value whose domain has a total order. The range-filtering ANNS consists of a query range and a query vector. It finds the approximate nearest neighbors of the query vector among all the data vectors whose attribute values fall in the query range. Existing approaches suffer from a rapidly degrading query performance when the query range width shifts. The query performance can be optimized by a solution that builds an ANNS index for every possible query range; however, the index time and index size become prohibitive -- the number of query ranges is quadratic to the number n of data vectors. To overcome these challenges, for the query range contains all attribute values smaller than a user-provided threshold, we design a structure called the segment graph whose index time and size are the same as a single ANNS index, yet can losslessly compress the n ANNS indexes, reducing the indexing cost by a factor of Ω(n). To handle general range queries, we propose a 2D segment graph with average-case index size O(n log n) to compress n segment graphs, breaking the quadratic barrier. Extensive experiments conducted on real-world datasets show that our proposed structures outperformed existing methods significantly; our index also exhibits superior scalability.

ANNProof: Building a verifiable and efficient outsourced approximate nearest neighbor search system on blockchain

Data-as-a-service is increasingly prevalent, with outsourced K-approximate nearest neighbors search (K-ANNS) gaining popularity in applications like similar image retrieval and anti-money laundering. However, malicious search service providers and dataset providers in current outsourced query systems cause incorrect user query results. To address this, we propose ANNProof, a novel framework supporting verifiable outsourced K-ANNS on the blockchain. ANNProof utilizes two innovative authenticated data structures (ADS), the Merkle HNSW node tree, and the Merkle vector identifier tree, for efficient K-ANNS query verification. Additionally, we employ the Merkle sharding tree as an ADS optimization technique, reducing the overhead of delivering verifiable queries. We implement the ADS construction protocol based on blockchain smart contracts to ensure tamper-evident datasets and enhance execution efficiency via a contract state consistency checking scheme. Extensive evaluations show that ANNProof reduces VO generation time, result verification time, and VO size by 160, 120, and 28×, respectively, compared to the state-of-the-art systems. Moreover, ADS construction using ANNProof takes at most 2% of the index construction time, resulting in a negligible overhead for implementing verifiable queries. Meanwhile, the sharding optimization accelerates ADS updates by 53×.

Bridging Software-Hardware for CXL Memory Disaggregation in Billion-Scale Nearest Neighbor Search

We propose CXL-ANNS , a software-hardware collaborative approach to enable scalable approximate nearest neighbor search (ANNS) services. To this end, we first disaggregate DRAM from the host via compute express link (CXL) and place all essential datasets into its memory pool. While this CXL memory pool allows ANNS to handle billion-point graphs without an accuracy loss, we observe that the search performance significantly degrades because of CXL’s far-memory-like characteristics. To address this, CXL-ANNS considers the node-level relationship and caches the neighbors in local memory, which are expected to visit most frequently. For the uncached nodes, CXL-ANNS prefetches a set of nodes most likely to visit soon by understanding the graph traversing behaviors of ANNS. CXL-ANNS is also aware of the architectural structures of the CXL interconnect network and lets different hardware components collaborate with each other for the search. Furthermore, it relaxes the execution dependency of neighbor search tasks and allows ANNS to utilize all hardware in the CXL network in parallel. Our evaluation shows that CXL-ANNS exhibits 93.3% lower query latency than state-of-the-art ANNS platforms that we tested. CXL-ANNS also outperforms an oracle ANNS system that has unlimited local DRAM capacity by 68.0%, in terms of latency.

COMPARING ROTATION-ROBUST MECHANISMS IN LOCAL FEATURE MATCHING: HAND-CRAFTED VS. DEEP LEARNING ALGORITHMS

The objective of this research is to conduct a performance comparison between hand-crafted feature matching algorithms and deep learning-based counterparts in the context of rotational variances. Hand-crafted algorithms underwent testing utilizing FLANN (Fast Library for Approximate Nearest Neighbors) as the matcher and RANSAC (Random sample consensus) for outlier detection and elimination, contributing to enhanced accuracy in the results. Surprisingly, experiments revealed that hand-crafted algorithms could yield comparable or superior results to deep learning-based algorithms when exposed to rotational variances. Notably, the application of horizontally flipped images showcased a distinct advantage for deep learning-based algorithms, demonstrating significantly improved results compared to their hand-crafted counterparts. While deep learning-based algorithms exhibit technological advancements, the study found that hand-crafted algorithms like AKAZE and AKAZE-SIFT could effectively compete with their deep learning counterparts, particularly in scenarios involving rotational variances. However, the same level of competitiveness was not observed in horizontally flipped cases, where hand-crafted algorithms exhibited suboptimal results. Conversely, deep learning algorithms such as DELF demonstrated superior results and accuracy in horizontally flipped scenarios. The research underscores that the choice between hand-crafted and deep learning-based algorithms depends on the specific use case. Hand-crafted algorithms exhibit competitiveness, especially in addressing rotational variances, while deep learning-based algorithms, exemplified by DELF, excel in scenarios involving horizontally flipped images, showcasing the unique advantages each approach holds in different contexts.

Learning Adaptive Hypersphere: Boosting Efficiency on Approximate Nearest Neighbor Search

Learning Adaptive Hypersphere: Boosting Efficiency on Approximate Nearest Neighbor Search