Algorithm For Nearest Neighbor Search Research Articles

Intercomparisons of Tracker v1.1 and four other ocean particle-tracking software packages in the Regional Ocean Modeling System

Abstract. Particle tracking is widely utilized to study transport features in a range of physical, chemical, and biological processes in oceanography. In this study, a new offline particle-tracking package, Tracker v1.1, is introduced, and its performance is evaluated in comparison to an online Eulerian dye, one online particle-tracking software package, and three offline particle-tracking software packages in a small, high-resolution model domain and a large coarser model domain. It was found that both particle and dye approaches give similar results across different model resolutions and domains when they were tracking the same water mass, as indicated by similar mean advection pathways and spatial distributions of dye and particles. The flexibility of offline particle tracking and its similarity against online dye and online particle tracking make it a useful tool to complement existing ocean circulation models. The new Tracker was shown to be a reliable particle-tracking package to complement the Regional Ocean Modeling System (ROMS) with the advantages of platform independence and speed improvements, especially in large model domains achieved by the nearest-neighbor search algorithm. Lastly, trade-offs of computational efficiency, modifiability, and ease of use that can influence the choice of which package to use are explored. The main value of the present study is that the different particle and dye tracking codes were all run on the same model output or within the model that generated the output. This allows some measure of intercomparison between the different tracking schemes, and we conclude that all choices that make each tracking package unique do not necessarily lead to very different results.

Data-driven enhanced FDEM for simulating the rock mechanical behavior

In this paper, a data-driven enhanced combined finite-discrete element method (DDFDEM) is proposed to simulate the rock mechanical behavior by directly assigning the rock experimental data into mechanical simulations, thus bypassing the empirical constitutive model of rock. To achieve this, a novel data-driven iterative solver with transient dynamic system equilibrium constraint is first proposed and embedded into the FDEM computing framework. Then, the bonding force update algorithm for crack elements is improved to mitigate the error accumulation during the iteration process. After that, a recommended formation for the proper size of the experimental data set is proposed to balance the computational accuracy and efficiency of DDFDEM, and an efficient nearest neighbor search algorithm is introduced to significantly improve the computational efficiency of DDFDEM. After the above improvements, the computational accuracy and efficiency of DDFDEM in simulating the rock deformation and complex failure process are validated by conducting several examples, including the thin plate tension test, uniaxial compression test, and Brazilian disk test. Finally, the advantage of DDFDEM over conventional FDEM in comprehensively presenting the information hidden behind the experimental data sets is validated by reproducing the nonlinear mechanical behaviors of rock samples based on a data set following the nonlinear elastic material law.

Indoor Clutter Object Removal Method for an As-Built Building Information Model Using a Two-Dimensional Projection Approach

Point cloud data are used to create an as-built building information model (as-built BIM) that reflects the actual status of any building, whether being constructed or already completed. However, indoor clutter objects in the point cloud data, such as people, tools, and materials, should be effectively eliminated to create the as-built BIM. In this study, the authors proposed a novel method to automatically remove indoor clutter objects based on the Manhattan World assumption and object characteristics. Our method adopts a two-dimensional (2D) projection of a 3D point cloud approach and utilizes different properties of indoor clutter objects and structural elements in the point cloud. Voxel-grid downsampling, density-based spatial clustering (DBSCAN), the statistical outlier removal (SOR) filter, and the unsupervised radius-based nearest neighbor search algorithm were applied to our method. Based on the evaluation of our proposed method using six actual scan datasets, we found that our method achieved a higher mean accuracy (0.94), precision (0.97), recall (0.90), and F1 core (0.93) than the commercial point cloud processing software. Our method shows better results than commercial point cloud processing software in classifying and removing indoor clutter objects in complex indoor environments acquired from construction sites. As a result, assumptions about the different properties of indoor clutter objects and structural elements are being used to identify indoor clutter objects. Additionally, it is confirmed that the parameters used in the proposed method could be determined by the voxel size once it is decided during the downsampling process.

DeepJoin: Joinable Table Discovery with Pre-Trained Language Models

Due to the usefulness in data enrichment for data analysis tasks, joinable table discovery has become an important operation in data lake management. Existing approaches target equi-joins, the most common way of combining tables for creating a unified view, or semantic joins, which tolerate misspellings and different formats to deliver more join results. They are either exact solutions whose running time is linear in the sizes of query column and target table repository, or approximate solutions lacking precision. In this paper, we propose DeepJoin, a deep learning model for accurate and efficient joinable table discovery. Our solution is an embedding-based retrieval, which employs a pre-trained language model (PLM) and is designed as one framework serving both equi- and semantic (with a similarity condition on word embeddings) joins for textual attributes with fairly small cardinalities. We propose a set of contextualization options to transform column contents to a text sequence. The PLM reads the sequence and is fine-tuned to embed columns to vectors such that columns are expected to be joinable if they are close to each other in the vector space. Since the output of the PLM is fixed in length, the subsequent search procedure becomes independent of the column size. With a state-of-the-art approximate nearest neighbor search algorithm, the search time is sublinear in the repository size. To train the model, we devise the techniques for preparing training data as well as data augmentation. The experiments on real datasets demonstrate that by training on a small subset of a corpus, DeepJoin generalizes to large datasets and its precision consistently outperforms other approximate solutions'. DeepJoin is even more accurate than an exact solution to semantic joins when evaluated with labels from experts. Moreover, when equipped with a GPU, DeepJoin is up to two orders of magnitude faster than existing solutions.

Voyage optimization using dynamic programming with initial quadtree based route

Abstract This research proposes an integrated voyage optimization algorithm that combines quadtree graph generation, visibility graph simplification, Dijkstra’s algorithm, and a 3D dynamic programming (3DDP) method. This approach enables the determination of a minimum distance initial reference route and the creation of a 2D navigational graph for efficient route optimization. We effectively store and process complex terrain information by transforming the GEBCO uniform grid into a quadtree structure. By utilizing a nearest neighbour search algorithm, edges are connected between adjacent ocean nodes, facilitating the generation of a quadtree graph. Applying Dijkstra’s algorithm to the quadtree graph, we derive the shortest initial route and construct a visibility graph based on the waypoints. This results in a simplified reference route with reduced search distance, allowing for more efficient navigation. For each waypoint along the reference route, a boundary is defined angled at 90 degrees to the left and right, based on the waypoint’s reference bearing. A line segment formed by the waypoint and both boundaries is defined as a navigational stage. A navigational graph is defined by connecting adjacent stages. Employing a 3DDP method on the navigational graph, and incorporating weather forecasting data, including wind, wave, and currents, we search for a route that minimizes fuel oil consumption with estimated time of arrival restrictions. Our approach is tested on several shipping routes, demonstrating a fuel consumption reduction compared to other voyage optimization routes. This integrated algorithm offers a potential solution for tackling complex voyage optimization problems in marine environments while considering various weather factors.

Accelerating Large-Scale Graph-Based Nearest Neighbor Search on a Computational Storage Platform

K-nearest neighbor search is one of the fundamental tasks in various applications and the hierarchical navigable small world (HNSW) has recently drawn attention in large-scale cloud services, as it easily scales up the database while offering fast search. On the other hand, a computational storage device (CSD) that combines programmable logic and storage modules on a single board becomes popular to address the data bandwidth bottleneck of modern computing systems. In this paper, we propose a computational storage platform that can accelerate a large-scale graph-based nearest neighbor search algorithm based on SmartSSD CSD. To this end, we modify the algorithm more amenable on the hardware and implement two types of accelerators using HLS- and RTL-based methodology with various optimization methods. In addition, we scale up the proposed platform to have 4 SmartSSDs and apply graph parallelism to boost the system performance further. As a result, the proposed computational storage platform achieves 75.59 query per second throughput for the SIFT1B dataset at 258.66W power dissipation, which is 12.83x and 17.91x faster and 10.43x and 24.33x more energy efficient than the conventional CPU-based and GPU-based server platform, respectively. With multi-terabyte storage and custom acceleration capability, we believe that the proposed computational storage platform is a promising solution for cost-sensitive cloud datacenters.

Cloud service selection based on weighted KD tree nearest neighbor search

The military organization cloud cooperation is a new type of military organization that encapsulates combat resources into services and makes them available as a service. Because of the heterogeneity of the combat resources, the cloud services are different not only in functional properties, but also in non-functional properties. Hence, the idea of cloud service selection, which aims at providing service requester with cloud services that meet their needs for non-functional attributes, has emerged. To this end, this paper proposes a cloud service selection method based on the weighted KD tree nearest neighbor search (WKDTNNS) algorithm for the personalized search of cloud services with different weights in each dimension. Firstly, for the atomic cloud service selection, the KD tree of all atomic cloud services with the quality of service (QoS) as the key is constructed, and WKDTNNS is used to match cloud services for service requests from users. Then, for combined cloud service selection, the KD tree is constructed based on pre-established combined cloud services to select several similar services as seeds, and the WKDTNNS algorithm is applied to optimize the seeds and obtain the final results. Finally, a case study is presented to show the feasibility and effectiveness of the proposed method, the results show that the proposed method could achieve better results than other algorithms, with 26%–56% less computation time.

Noncontact Sleeping Heartrate Monitoring Method Using Continuous-Wave Doppler Radar Based on the Difference Quadratic Sum Demodulation and Search Algorithm.

Continuous-wave doppler radar, which has the advantages of simple structure, low cost, and low power consumption, has attracted extensive attention in the detection of human vital signs. However, while respiration and heartbeat signals are mixed in the echo phase, the amplitude difference between the two signals is so large that it becomes difficult to measure the heartrate (HR) from the interference of respiration stably and accurately. In this paper, the difference quadratic sum demodulation method is proposed. According to the mixed characteristics of respiration and heartbeat after demodulation, the heartbeat features can be extracted with the help of the easy-to-detect breathing signal; combined with the constrained nearest neighbor search algorithm, it can realize sleeping HR monitoring overnight without body movements restraint. Considering the differences in vital-sign characteristics of different individuals and the irregularity of sleep movements, 54 h of sleep data for nine nights were collected from three subjects, and then compared with ECG-based HR reference equipment. After excluding the periods of body turning over, the HR error was within 10% for more than 70% of the time. Experiments confirmed that this method, as a tool for long-term HR monitoring, can play an important role in sleeping monitoring, smart elderly care, and smart homes.

Stochastic Geometry-Based Low Latency Routing in Massive LEO Satellite Networks

In this article, the routing in massive low earth orbit (LEO) satellite networks is studied. When the satellite-to-satellite communication distance is limited, we choose different relay satellites to minimize the latency in a constellation at a constant altitude. First, the global optimum solution is obtained in the ideal scenario when there are available satellites at all the ideal locations. Next, we propose a nearest neighbor search algorithm for realistic (nonideal) scenarios with a limited number of satellites. The proposed algorithm can approach the global optimum solution under an ideal scenario through a finite number of iterations and a tiny range of searches. Compared with other routing strategies, the proposed algorithm shows significant advantages in terms of latency. Furthermore, we provide two approximation techniques that can give tight lower and upper bounds for the latency of the proposed algorithm, respectively. Finally, the relationships between latency and constellation height, satellites' number, and communication distance are investigated.

Multidepot Drone Path Planning With Collision Avoidance

Intersections of flight paths in multidrone missions are indications of a high likelihood of in-flight drone collisions. This likelihood can be proactively minimized during path planning. This article proposes two offline collision-avoidance multidrone path-planning algorithms: 1) DETACH and 2) STEER. Large drone tasks can be divided into smaller ones and carried out by multiple drones. Each drone follows a planned flight path that is optimized to efficiently perform the task. The path planning of the set of drones can then be optimized to complete the task in a short time, with minimum energy expenditure, or with maximum waypoint coverage. Here, we focus on maximizing waypoint coverage. Different from existing schemes, our proposed offline path-planning algorithms detect and remove possible in-flight collisions. They are based on a constrained nearest-neighbor search algorithm that aims to cover a large number of waypoints per flight path. DETACH and STEER perform vector intersection check for flight path analysis, but each at different stages of path planning. We evaluate the waypoint coverage of the proposed algorithms through a novel profit model and compare their performance on a work area with different waypoint densities. Our results show that STEER covers 40% more waypoints and generates 20% more profit than DETACH in high-density waypoint scenarios.

Improved RANSAC Point Cloud Spherical Target Detection and Parameter Estimation Method Based on Principal Curvature Constraint.

Spherical targets are widely used in coordinate unification of large-scale combined measurements. Through its central coordinates, scanned point cloud data from different locations can be converted into a unified coordinate reference system. However, point cloud sphere detection has the disadvantages of errors and slow detection time. For this reason, a novel method of spherical object detection and parameter estimation based on an improved random sample consensus (RANSAC) algorithm is proposed. The method is based on the RANSAC algorithm. Firstly, the principal curvature of point cloud data is calculated. Combined with the k-d nearest neighbor search algorithm, the principal curvature constraint of random sampling points is implemented to improve the quality of sample points selected by RANSAC and increase the detection speed. Secondly, the RANSAC method is combined with the total least squares method. The total least squares method is used to estimate the inner point set of spherical objects obtained by the RANSAC algorithm. The experimental results demonstrate that the method outperforms the conventional RANSAC algorithm in terms of accuracy and detection speed in estimating sphere parameters.

A coverage path planning approach for autonomous radiation mapping with a mobile robot

In nuclear and radiation-related industries, it is crucial to ensure that the radiation dose exposure to the radiation worker is maintained below the permissible dose limit. A radiation map is a useful tool for visualizing the radiation distribution across the work area and for coordinating activities involving the hotspots (high radiation areas). The goal of this work was to design and implement a coverage path planning approach for autonomous radiation mapping carried out by a mobile robot. Given a 2D occupancy map, a method to generate uniformly distributed sampling points was proposed. The geometry of the region of interest, the radiation detector module, and the radiation measurement parameters were considered in formulating the sampling positions. Next, the coverage path planning planner integrates the nearest neighbor and depth-first search algorithms to create a continuous path that enables the robot to visit all the sampling points. The K-means clustering algorithm is added for systematic coverage of a large number of sampling points. The clustering provides options to partition the region of interest into smaller spaces, where the robot would perform the mapping cluster by cluster. Finally, the method of building the radiation map from the acquired data was also presented. The approach was implemented in ROS using a commercial mobile robot equipped with a Geiger–Muller detector. The performance and reliability of the proposed approach were evaluated with a series of simulations and real-world experiments. The results showed that the robot is able to perform autonomous radiation mapping at various target areas. The accuracy of the generated radiation map and the hotspots classifications were also compared and evaluated with conventional manual measurements. Overall, the theoretical frameworks and experiments have provided convincing results in the automation of hazardous work and subsequently toward improving the occupational safety of radiation workers.

A new fast inverted file-based algorithm for approximate nearest neighbor search without accuracy reduction

The inverted file (IVF) is one of the most efficient non-exhaustive search algorithms for approximate nearest neighbor (ANN) search. In this paper, we propose a new three-step fast search algorithm in the framework of IVF named fast IVF (fIVF) to accelerate the ANN search. Different from the traditional IVF algorithm that conducts a full search at the coarse quantization stage, we introduce (1) cell-level rejection, (2) vector-level rejection, and (3) subspace-based pyramid model with multi-resolution to speed up this high time-consuming stage. Since our proposed fIVF concentrates on finding the exact w nearest coarse codewords at the coarse quantization stage and follows the same product quantization stage, it can achieve the same search accuracy as the traditional IVF. Experimental results on different datasets demonstrate that our proposed fIVF can significantly speed up the search process while keeping the same search accuracy as the traditional IVF.

Efficient exact k-flexible aggregate nearest neighbor search in road networks using the M-tree

This study proposes an efficient exact k-flexible aggregate nearest neighbor (k-FANN) search algorithm in road networks using the M-tree. The state-of-the-art IER-kNN algorithm used the R-tree and pruned off unnecessary nodes based on the Euclidean coordinates of objects in road networks. However, IER-kNN made many unnecessary accesses to index nodes since the Euclidean distances between objects are significantly different from the actual shortest-path distances between them. In contrast, our algorithm proposed in this study can greatly reduce unnecessary accesses to index nodes compared with IER-kNN since the M-tree is constructed based on the actual shortest-path distances between objects. To the best of our knowledge, our algorithm is the first exact FANN algorithm that uses the M-tree. We prove that our algorithm does not cause any false drop. In conducting a series of experiments using various real road network datasets, our algorithm consistently outperformed IER-kNN by up to 6.92 times.

Applications of Nearest Neighbor Search Algorithm Toward Efficient Rubber-Based Solid Waste Management in Concrete

Indeed, natural processes of discarding rubber waste have many disadvantages for the environment. As a result, multiple researchers suggested addressing this problem by recycling rubber as an aggregate in concrete mixtures. Previously, numerous studies have been undertaken experimentally to investigate the properties of rubberized concrete. Furthermore, investigations were carried out to develop estimating techniques to precisely specify the generated concrete's characteristics, making its use in real-life applications easier. However, there is still a gap in the conducted studies on the performance of the k-nearest neighbor algorithm. Hence, this research explores the accuracy of using the k-nearest neighbor's algorithm in predicting the compressive and tensile strength and the modulus of elasticity of rubberized concrete. It will be done by developing an optimized machine learning model using the aforementioned method and then benchmarking its results to the outcomes of multiple linear regression and artificial neural networks. The study's findings have shown that the k-nearest neighbor's algorithm provides significantly higher accuracy than other methods. This kind of study needs to be discussed in the literature so that people can better deal with rubber waste in concrete. Doi: 10.28991/CEJ-2022-08-04-06 Full Text: PDF

LANNS

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks (HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying top-K (100 ≤ k ≤ 200) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of ~2.5k Queries Per Second (QPS) on a single node, on large (e.g., ~ 180M data points) high dimensional (50-2048 dimensional) datasets.

Feature Matching Method Based on SURF and Fast Library for Approximate Nearest Neighbor Search

Aiming at dimensional feature vector matching problem of low accuracy, a kind of image matching algorithm is based on SURF and fast library for approximate nearest neighbor search. Fast-Hessian was used to detect the feature points, and the SURF feature description vector proposed was generated; through fast library for approximate nearest neighbor search algorithm, pre-matching point was got. With the introduction of Random Sample Consensus (RANSAC) algorithm points false-matching, based on the SIFT algorithm, the result of SURF algorithm and optimization algorithm was proposed and mismatching points were eliminated after the experiment simulation. Experimental results show that at the same time the matching algorithm improve matching accuracy rate, and the algorithm of the real-time performance is also improved.

A Revisit of Hashing Algorithms for Approximate Nearest Neighbor Search

Approximate Nearest Neighbor Search (ANNS) is a fundamental problem in many areas of machine learning and data mining. During the past decade, numerous hashing algorithms are proposed to solve this problem. Every proposed algorithm claims to outperform Locality Sensitive Hashing (LSH), which is the most popular hashing method. However, the evaluation of these hashing article was not thorough enough, and the claim should be re-examined. If implemented correctly, almost all the hashing methods will have their performance improved as the code length increases. However, many existing hashing article only report the performance with the code length shorter than 128. In this article, we carefully revisit the problem of search-with-a-hash-index and analyze the pros and cons of two popular hash index search procedures. Then we proposed a simple but effective novel hash index search approach and made a thorough comparison of eleven popular hashing algorithms. Surprisingly, the random-projection-based Locality Sensitive Hashing ranked the first, which is in contradiction to the claims in all the other 10 hashing article. Despite the extreme simplicity of random-projection-based LSH, our results show that the capability of this algorithm has been far underestimated. For the sake of reproducibility, all the codes used in the article are released on GitHub, which can be used as a testing platform for a fair comparison between various hashing algorithms.

Finite element solver for data-driven finite strain elasticity

A nominal finite element solver is proposed for data-driven finite strain elasticity. It bypasses the need for a constitutive model by considering a database of deformation gradient/first Piola–Kirchhoff stress tensors pairs. The boundary value problem is reformulated as the constrained minimization problem of the distance between (i) the mechanical states, i.e. strain–stress, in the body and (ii) the material states coming from the database. The corresponding constraints are of two types: kinematical, i.e. displacement–strain relation, and mechanical, i.e. conservation linear and angular momenta. The solver uses alternated minimization: the material states are determined from a local search in the database using an efficient tree-based nearest neighbor search algorithm, and the mechanical states result from a standard constrained minimization addressed with an augmented Lagrangian approach. The performance of the solver is demonstrated by means of 2D sanity check examples: the data-driven solution converges to the classical finite element solution when the material database increasingly approximates the constitutive model. In addition, we demonstrate that the balance of angular momentum, which was classically not taken into account in previous data-driven studies, must be enforced as a constraint to ensure the convergence of the method.

Research on Industry Question Answering System Based on Near Neighbor Vector Search

In the field of natural language processing, language-based question answering has been widely studied and made great achievements. As an important form of search engine, question answering system can further improve the search quality compared with traditional search engines. Traditional tree-based indexing methods are inefficient in high-dimensional space. This paper focuses on the application of k nearest neighbor search in large-scale image databases, and researches on high-dimensional indexing technology based on vector approximation method. Fast-Hessian detector is used to detect feature points and generate SURF feature description vector; Then, the initial matching point pair is obtained by fast approximate nearest neighbor search algorithm, and then the unidirectional matching results are matched bidirectionally; at the same time, it makes a detailed analysis of the situation that the events in the sentences are verbs of “abstract change”, so as to realize the automatic question and answer after the change of abstract relations between entities.