Navigable Small World Research Articles

GSearch: ultra-fast and scalable genome search by combining K-mer hashing with hierarchical navigable small world graphs.

Genome search and/or classification typically involves finding the best-match database (reference) genomes and has become increasingly challenging due to the growing number of available database genomes and the fact that traditional methods do not scale well with large databases. By combining k-mer hashing-based probabilistic data structures (i.e. ProbMinHash, SuperMinHash, Densified MinHash and SetSketch) to estimate genomic distance, with a graph based nearest neighbor search algorithm (Hierarchical Navigable Small World Graphs, or HNSW), we created a new data structure and developed an associated computer program, GSearch, that is orders of magnitude faster than alternative tools while maintaining high accuracy and low memory usage. For example, GSearch can search 8000 query genomes against all available microbial or viral genomes for their best matches (n= ∼318 000 or ∼3 000 000, respectively) within a few minutes on a personal laptop, using ∼6 GB of memory (2.5 GB via SetSketch). Notably, GSearch has an O(log(N)) time complexity and will scale well with billions of genomes based on a database splitting strategy. Further, GSearch implements a three-step search strategy depending on the degree of novelty of the query genomes to maximize specificity and sensitivity. Therefore, GSearch solves a major bottleneck of microbiome studies that require genome search and/or classification.

Novel Probabilistic Collision Detection for Manipulator Motion Planning Using HNSW

Collision detection is very important for robot motion planning. The existing accurate collision detection algorithms regard the evaluation of each node as a discrete event, ignoring the correlation between nodes, resulting in low efficiency. In this paper, we propose a novel approach that transforms collision detection into a binary classification problem. In particular, the proposed method searches the k-nearest neighbor (KNN) of the new node and estimates its collision probability by the prior node. We perform the hierarchical navigable small world (HNSW) method to query the nearest neighbor data and store the detected nodes to build the database incrementally. In addition, this research develops a KNN query technique tailored for linear data, incorporating threshold segmentation to facilitate collision detection along continuous paths. Moreover, it refines the distance function of the collision classifier to enhance the precision of probability estimations. Simulation results demonstrate the effectiveness of the proposed method.

From Text to Recommendations: How Vector Databases are Revolutionizing Personalized Content Delivery

Abstract: Effective similarity search and retrieval are now possible thanks to vector databases, which have become a potent tool for organizing and searching high-dimensional data. This article provides a thorough examination of vector databases, their underlying theories, and their applications across a range of industries. In handling complex data types, we address the significance of vector representation and emphasize the benefits of vector databases over traditional databases [1]. The article explores the process of creating a vector database, highlighting the critical function of indexing strategies such as IVF (Inverted File Indexing) and HNSW (Hierarchical Navigable Small World) in guaranteeing the effectiveness and precision of searches [2]. In addition, we discuss the problems caused by the curse of dimensionality and offer solutions to lessen its effects on nearest neighbor searches [3]

Decentralized Face Identification with Hierarchical Navigable Small World on Blockchain

This paper presents a novel method for decentralized storage in deep-learning-based face recognition systems using the Hierarchical Navigable Small World (HNSW) algorithm. The proposed solution utilizes Ethereum smart contracts, which acts as highly available data storage systems for storing identifiable data for authorized personnel. In addition, the solution is integrated with a centralized vector database that is in charge of vector indexing, searching and associating face embeddings to an identity on the Ethereum blockchain with anonymous hashes. Vector indexing and search processes involve different machine learning algorithms that enable computations to be carried out in a reasonable time with good matching accuracy. Specifically, we compared different approaches and selected the HNSW algorithm. Accordingly, we successfully implemented a prototype of a reliable and privacy-focused decentralized face identification system for areas under government surveillance, such as customs inspection sites. In our measurements, the system could handle 20,000 face vectors easily with high matching accuracy, and the performance could be further improved using more powerful hardware. Finally, we also propose additional methods to further scale up the system to handle millions of face vectors.

A novel extended rule-based system based on K-Nearest Neighbor graph

The Belief Rule-Based (BRB) system faces the rule combination explosion issue, making it challenging to construct the rule base efficiently. The Extended Belief Rule-Based (EBRB) system offers a solution to this problem by using data-driven methods. However, using EBRB system requires the traversal of the entire rule base, which can be time-consuming and result in the activation of many irrelevant rules, leading to an incorrect decision. Existing search optimization methods can somewhat solve this issue, but they have limitations. Moreover, the calculation of the rule activation weight only considers the similarity between input data and a single rule, ignoring the influence of the rule linkage. To address these problems, we propose a new EBRB system based on the K-Nearest Neighbor graph index (Graph-EBRB). We introduce the Hierarchical Navigable Small World (HNSW) algorithm to create the K-Nearest Neighbor graph index of the EBRB system. This index allows us to efficiently search and activate a set of key rules. We also propose a new activation weight calculation method based on the Graph Convolution Neural Network (GCN), and we optimize the system performance using a parameter learning strategy. We conduct a comprehensive experiment on 14 commonly used public data sets, and the results show that Graph-EBRB system significantly improves the reasoning efficiency and accuracy of the EBRB system. Finally, we apply the Graph-EBRB system to tree disease identification and achieve excellent classification performance, identifying over 90% of the diseased trees on the complete dataset.

Exploring deep learning-based content-based video retrieval with Hierarchical Navigable Small World index and ResNet-50 features for anomaly detection

Surveillance video can capture a variety of realistic scenarios for providing security in both private and public spaces. The prediction of anomalies promptly is essential. However, existing methods are time-consuming due to marking conflicting clips in training videos. Therefore, this paper proposes a CBVR (Content-Based Video Retrieval) framework for detecting anomalies in surveillance videos. The proposed method leverages deep learning techniques and the HNSW (Hierarchical Navigable Small World) index to achieve efficient and accurate video retrieval. To preprocess the video data, frames are resized and normalized, and the dataset is augmented with data transformations such as rotations, translations, and flips to improve model generalization. The best results are offered while evaluating with the UCSD Ped2 dataset which produces the MAP score of 94.25%, and recall, precision, and F1-scores of 96.15%, 98.03%, and 92.63% respectively. The results proved that maximizing the number of layers in the HNSW index enhances the MAP score, suggesting that deeper layers are more effective at retrieving relevant results. Overall, the proposed CBVR framework provides an efficient and accurate method for anomaly detection in surveillance videos. It has potential applications in various domains, including video summarization, recommendation, and surveillance, among others.

VIEW GRAPH CONSTRUCTION FOR LARGE-SCALE UAV IMAGES: AN EVALUATION OF STATE-OF-THE-ART METHODS

Abstract. Structure from Motion (SfM) is a 3D reconstruction framework that has achieved great success on large-scale Unmanned Aerial Vehicle (UAV) images. Due to the high time consumption of feature matching, a matching candidate subset is obtained by image retrieval to improve efficiency. Bag of Word (BoW) based image retrieval has been widely used in SfM systems, but the large number of local features and the high dimension of the BoW vector cause the retrieval method time-consuming. Vector of Locally Aggregated Descriptors (VLAD) and learning-based NetVLAD perform well in image retrieval, and these vector representation methods are evaluated in this study. After images are transformed into vectors, Nearest Neighbour (NN) searching methods like Brute-force and KD-Tree are used to find similar images. But as the number of images and the vector dimension increase, Approximate Nearest Neighbour (ANN) searching methods like Hierarchical Navigable Small World (HNSW) and Locality-Sensitive Hashing (LSH) are considered to replace NN searching to avoid efficiency degradation. These vector searching methods are also evaluated in this study. The test results demonstrate that the optimal method VLAD with HNSW can speed up about 100 times in finding matching candidate subset. A view graph that guides scene partition and sub-scene reconstruction in parallel SfM can be created by the optimal method. With this view graph construction method, the efficiency of SfM is significantly improved.

An Efficient Online Loop Closure Detection System With Local Spatial Co-Occurrence Information

As a vital component in simultaneous localization and mapping techniques, appearance-based loop closure detection (LCD) plays important roles in bounding the long-term drift errors. In this paper, an online LCD system based on the mutual co-occurrence information among visual features is proposed. First, a feature tracker module is designed to generate distinctive visual words, exploiting tracked words tool to improve efficiency. Then an incrementally built vocabulary is organized by a hierarchical navigable small world graph, where the visual words are indexed. To merge a homologous word into the existing one, the vocabulary applies an improved high dimensional online clustering method, which regards individual cluster as a normal distribution form. At the query phase, a list of candidate frames is located due to the co-occurrence constraint. Ultimately, the loop closure is specified by passing the temporal and similarity check, which avoids the memory consumption of historic image data. Validation tests based on public datasets and experimental sequence demonstrate the merit of low running time and memory cost, while the high-precision performance is retained in the system.

Loop Closure Detection With Bidirectional Manifold Representation Consensus

Loop closure detection (LCD) is an indispensable module in simultaneous localization and mapping. It is responsible to recognize pre-visited areas during the navigation of a robot, providing auxiliary information to revise pose estimation. Unlike most current methods which focus on seeking an appropriate representation of images, we propose a novel two-stage pipeline dominated by the estimation of spatial geometric relationship. Specifically, to avoid unnecessary memory costs, consecutive images are segmented into sequences as per the similarity of their global features. Then the sequence descriptor is incrementally inserted into hierarchical navigable small world for the construction of reference database, from which the most similar image for the query one is searched parallelly. To further identify whether the candidate pair is geometry-consistent, a feature matching method termed as bidirectional manifold representation consensus (BMRC) is proposed. It constructs local neighborhood structures of feature points via manifold representation, and formulates the matching problem into an optimization model, enabling linearithmic time complexity via a closed-form solution. Meanwhile, an accelerated version of it is introduced (BMRC*), which performs about 63% faster than BMRC in an image pair with 352 initial correspondences. Extensive experiments on nine publicly available datasets demonstrate that BMRC and BMRC* perform well in feature matching and the proposed pipeline has remarkable performance in the LCD task.

Semantics-Aware Dataset Discovery from Data Lakes with Contextualized Column-Based Representation Learning

Dataset discovery from data lakes is essential in many real application scenarios. In this paper, we propose Starmie, an end-to-end framework for dataset discovery from data lakes (with table union search as the main use case). Our proposed framework features a contrastive learning method to train column encoders from pre-trained language models in a fully unsupervised manner. The column encoder of Starmie captures the rich contextual semantic information within tables by leveraging a contrastive multi-column pre-training strategy. We utilize the cosine similarity between column embedding vectors as the column unionability score and propose a filter-and-verification framework that allows exploring a variety of design choices to compute the unionability score between two tables accordingly. Empirical results on real table benchmarks show that Starmie outperforms the best-known solutions in the effectiveness of table union search by 6.8 in MAP and recall. Moreover, Starmie is the first to employ the HNSW (Hierarchical Navigable Small World) index to accelerate query processing of table union search which provides a 3,000X performance gain over the linear scan baseline and a 400X performance gain over an LSH index (the state-of-the-art solution for data lake indexing).

Accelerating Large-Scale Graph-Based Nearest Neighbor Search on a Computational Storage Platform

K-nearest neighbor search is one of the fundamental tasks in various applications and the hierarchical navigable small world (HNSW) has recently drawn attention in large-scale cloud services, as it easily scales up the database while offering fast search. On the other hand, a computational storage device (CSD) that combines programmable logic and storage modules on a single board becomes popular to address the data bandwidth bottleneck of modern computing systems. In this paper, we propose a computational storage platform that can accelerate a large-scale graph-based nearest neighbor search algorithm based on SmartSSD CSD. To this end, we modify the algorithm more amenable on the hardware and implement two types of accelerators using HLS- and RTL-based methodology with various optimization methods. In addition, we scale up the proposed platform to have 4 SmartSSDs and apply graph parallelism to boost the system performance further. As a result, the proposed computational storage platform achieves 75.59 query per second throughput for the SIFT1B dataset at 258.66W power dissipation, which is 12.83x and 17.91x faster and 10.43x and 24.33x more energy efficient than the conventional CPU-based and GPU-based server platform, respectively. With multi-terabyte storage and custom acceleration capability, we believe that the proposed computational storage platform is a promising solution for cost-sensitive cloud datacenters.

Loop Closure Detection With Reweighting NetVLAD and Local Motion and Structure Consensus

Dear Editor, Loop closure detection (LCD) is an important module in simultaneous localization and mapping (SLAM). In this letter, we address the LCD task from the semantic aspect to the geometric one. To this end, a network termed as AttentionNetVLAD which can simultaneously extract global and local features is proposed. It leverages attentive selection for local features, coupling with reweighting the soft assignment in NetVLAD via the attention map for global features. Given a query image, candidate frames are first identified coarsely by retrieving similar global features in the database via hierarchical navigable small world (HNSW). As global features mainly summarize the semantic information of images and lead to compact representation, information about spatial arrangement of visual elements is lost. To provide fine results, we further propose a feature matching method termed as local motion and structure consensus (LMSC) to conduct geometric verification between candidate pairs. It constructs local neighborhood structures of local features through motion consistency and manifold representation, and formulates the matching problem into an optimization model, enabling linearithmic time complexity via a closed-form solution. Experiments on several public datasets demonstrate that LMSC performs well in feature matching, and the proposed LCD system can yield satisfying results.

GuideMaker: Software to design CRISPR-Cas guide RNA pools in non-model genomes.

CRISPR-Cas systems have expanded the possibilities for gene editing in bacteria and eukaryotes. There are many excellent tools for designing CRISPR-Cas guide RNAs (gRNAs) for model organisms with standard Cas enzymes. GuideMaker is intended as a fast and easy-to-use design tool for challenging projects with (i) non-standard Cas enzymes, (ii) non-model organisms, or (iii) projects that need to design a panel of gRNA for genome-wide screens. GuideMaker can rapidly design gRNAs for gene targets across the genome using a degenerate protospacer-adjacent motif (PAM) and a genome. The tool applies hierarchical navigable small world graphs to speed up the comparison of guide RNAs and optionally provides on-target and off-target scoring. This allows the user to design effective gRNAs targeting all genes in a typical bacterial genome in ∼1-2 minutes. GuideMaker enables the rapid design of genome-wide gRNA for any CRISPR-Cas enzyme in non-model organisms. While GuideMaker is designed with prokaryotic genomes in mind, it can efficiently process eukaryotic genomes as well. GuideMaker is available as command-line software, a stand-alone web application, and a tool in the CyCverse Discovery Environment. All versions are available under a Creative Commons CC0 1.0 Universal Public Domain Dedication.

LANNS

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks (HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying top-K (100 ≤ k ≤ 200) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of ~2.5k Queries Per Second (QPS) on a single node, on large (e.g., ~ 180M data points) high dimensional (50-2048 dimensional) datasets.

Using Proximity Graph Cut for Fast and Robust Instance-Based Classification in Large Datasets

K-nearest neighbours (kNN) is a very popular instance-based classifier due to its simplicity and good empirical performance. However, large-scale datasets are a big problem for building fast and compact neighbourhood-based classifiers. This work presents the design and implementation of a classification algorithm with index data structures, which would allow us to build fast and scalable solutions for large multidimensional datasets. We propose a novel approach that uses navigable small-world (NSW) proximity graph representation of large-scale datasets. Our approach shows 2–4 times classification speedup for both average and 99th percentile time with asymptotically close classification accuracy compared to the 1-NN method. We observe two orders of magnitude better classification time in cases when method uses swap memory. We show that NSW graph used in our method outperforms other proximity graphs in classification accuracy. Our results suggest that the algorithm can be used in large-scale applications for fast and robust classification, especially when the search index is already constructed for the data.

ESA-VLAD: A Lightweight Network Based on Second-Order Attention and NetVLAD for Loop Closure Detection

Loop closure detection (LCD) is an important portion of Simultaneous Localization and Mapping (SLAM) because of its ability to reduce accumulated position errors. In this letter, we propose a novel loop closure detection algorithm named ESA-VLAD. The crucial part of ESA-VLAD is a redesigned network with EfficientNetB0 as backbone for extracting global features, which integrates a second-order attention module in order to effectively learn the correlations between features within the feature map. A trainable Vector of Local Aggregated Descriptors (NetVLAD) is integrated in the last layer of the network to generate a compact and fixed-length global feature. Knowledge distillation strategy is adopted in training of the proposed network to accelerate the training process. For the global features, Hierarchical Navigable Small World (HNSW) is employed to retrieve the loop closure candidate images. In addition, an efficient geometrical consistency check based on local difference binary (LDB) descriptors is designed to verify loop closure matches. Experiments on several public datasets demonstrate that ESA-VLAD can obtain higher recall rates under 100% precision and less processing time per frame compared to other typical and state-of-the-art methods.

A Small World Graph Approach for an Efficient Indoor Positioning System.

The main goal of an Indoor Positioning System (IPS) is to estimate the position of mobile devices in indoor environments. For this purpose, the primary source of information is the signal strength of packets received by a set of routers. The fingerprint technique is one of the most used techniques for IPSs. By using supervised machine learning techniques, it trains a model with the received signal intensity information so it can be used to estimate the positions of the devices later in an online phase. Although the k-Nearest Neighbors (kNN) is one of the most widely used classification methods due to its accuracy, it has no scalability since a sample that needs to be classified must be compared to all other samples in the training database. In this work, we use a novel hierarchical navigable small world graph technique to build a search structure so the location of a sample can be efficiently found, allowing the IPSs to be used in large-scale scenarios or run on devices with limited resources. To carry out our performance evaluation, we proposed a synthetic IPS dataset generator as well as implemented a complete real-world, large-scale IPS testbed. We compared the performance of our graph-based solution with other known kNN variants, such as Kd-Tree and Ball-Tree. Our results clearly show the performance gains of the proposed solution at when compared to the classic kNN and at least when compared to tree-based approaches.

SWFC-ART: A cost-effective approach for Fixed-Size-Candidate-Set Adaptive Random Testing through small world graphs

Adaptive random testing (ART) improves the failure-detection effectiveness of random testing by leveraging properties of the clustering of failure-causing inputs of most faulty programs: ART uses a sampling mechanism that evenly spreads test cases within a software’s input domain. The widely-used Fixed-Sized-Candidate-Set ART (FSCS-ART) sampling strategy faces a quadratic time cost, which worsens as the dimensionality of the software input domain increases. In this paper, we propose an approach based on small world graphs that can enhance the computational efficiency of FSCS-ART: SWFC-ART. To efficiently perform nearest neighbor queries for candidate test cases, SWFC-ART incrementally constructs a hierarchical navigable small world graph for previously executed, non-failure-causing test cases. Moreover, SWFC-ART has shown consistency in programs with high dimensional input domains. Our simulation and empirical studies show that SWFC-ART reduces the computational overhead of FSCS-ART from quadratic to log-linear order while maintaining the failure-detection effectiveness of FSCS-ART, and remaining consistent in high dimensional input domains. We recommend using SWFC-ART in practical software testing scenarios, where real-life programs often have high dimensional input domains and low failure rates.

Efficient Retrieval of Music Recordings Using Graph-Based Index Structures

Flexible retrieval systems are required for conveniently browsing through large music collections. In a particular content-based music retrieval scenario, the user provides a query audio snippet, and the retrieval system returns music recordings from the collection that are similar to the query. In this scenario, a fast response from the system is essential for a positive user experience. For realizing low response times, one requires index structures that facilitate efficient search operations. One such index structure is the K-d tree, which has already been used in music retrieval systems. As an alternative, we propose to use a modern graph-based index, denoted as Hierarchical Navigable Small World (HNSW) graph. As our main contribution, we explore its potential in the context of a cross-version music retrieval application. In particular, we report on systematic experiments comparing graph- and tree-based index structures in terms of the retrieval quality, disk space requirements, and runtimes. Despite the fact that the HNSW index provides only an approximate solution to the nearest neighbor search problem, we demonstrate that it has almost no negative impact on the retrieval quality in our application. As our main result, we show that the HNSW-based retrieval is several orders of magnitude faster. Furthermore, the graph structure also works well with high-dimensional index items, unlike the tree-based structure. Given these merits, we highlight the practical relevance of the HNSW graph for music information retrieval (MIR) applications.

Multiattribute approximate nearest neighbor search based on navigable small world graph

SummaryApproximate nearest neighbor search (ANNS) is widely employed to find the most similar data efficiently from a large‐scale dataset. The key of ANNS is to construct an effective index to prune the search space and retrieve the approximate data rather than the exact one in a very short time. Most of the existing ANNS methods only adopt the distance metric to build an index, which cannot support multiattribute ANNS. In this article, we present a novel approach for multiattribute ANNS based on navigable small world (NSW) graph, called MA‐NSW. Given a dataset, MA‐NSW (i) builds a proximity subgraph overlay for each multiattribute combination, and integrates all overlays to a hierarchical index, (ii) adopts a navigation tree to access related subgraph overlay and prune unrelated overlays, and (iii) gets nearest neighbor results from the related overlays by greedy search. MA‐NSW guarantees efficiency and it is defined in terms of arbitrary metric spaces (eg, Euclidean distance and cosine similarity). Performance evaluation has demonstrated that the proposed approach shows superior performance in multiattribute ANNS.