Efficient Feature Matching Research Articles

Efficient Feature Matching and Pose-graph Initialization for SfM

Abstract. Over the last decades, Structure from Motion (or image orientation) has been widely studied in the fields of photogrammetry and computer vision, mainly thanks to its feasibility for dealing with various image datasets (such as crowdsourced or UAV images). However, due to the fact that images are becoming easy to obtain, nowadays, it is challenging to deal with large scale of datasets, wherein the feature matching and pose-graph generation are the key limitations in terms of time efficiency. In this work, we proposed an efficient method to accelerate the generation of correspondences and two-view geometries. Specifically, based on some already estimated two-view geometries, unknown two-view geometries can be computed via A* algorithm. Then, the corresponding feature matching can be perform in a guided way using an epiploar-hash bins that is derived from the estimated two-view epipolar geometries. The experimental results demonstrated that, our method can improve the speed of generating pose-graph by 3–4 times comparing to two popular packages (colmap and OpenMVG) and is also faster than one SOTA method of Barath et al., (2021), yet the results of SfM are typically on par with them and reprojection error of our works are even better.

Efficient Feature Matching for Large-scale Images based on Cascade Hash and Local Geometric Constraint

Abstract. Feature matching plays a crucial role in 3D reconstruction to provide correspondences between overlapped images. The accuracy and efficiency of feature matching significantly impact the performance of 3D reconstruction. The widely used framework with the exhaustive nearest neighbor searching (NNS) between descriptors and RANSAC-based geometric estimation is, however, low-efficient and unreliable for large-scale UAV images. Inspired by indexing-based NNS, this paper implements an efficient feature matching method for large-scale images based on Cascade Hashing and local geometric constraints. Our proposed method improves upon traditional feature matching approaches by introducing a combination of image retrieval, data scheduling, and GPU-accelerated Cascade Hashing. Besides, it utilizes a local geometric constraint to filter matching results within a matching framework. On the one hand, the GPU-accelerated Cascade Hashing technique generates compact and discriminative hash codes based on image features, facilitating the rapid completion of the initial matching process, and significantly reducing the search space and time complexity. On the other hand, after the initial matching is completed, the method employs a local geometric constraint to filter the initial matching results, enhancing the accuracy of the matching results. This forms a three-tier framework based on data scheduling, GPU-accelerated Cascade Hashing, and local geometric constraints. We conducted experiments using two sets of large-scale UAV image data, comparing our method with SIFTGPU to evaluate its performance in initial matching, outlier rejection, and 3D reconstruction. The results demonstrate that our method achieves a feature matching speed 2.0 times that of SIFTGPU while maintaining matching accuracy and producing comparable reconstruction results. This suggests that our method holds promise for efficiently addressing large-scale image matching.

Efficient feature recognition and matching technology for IoT‐enabled sports training

AbstractWith the development of the Internet of Things (IoT), the IoT technology is gradually applied to the sport training of athletes. According to the training feature of athletes collected by the IoT equipment, this paper proposes to use the method of feature image sequence analysis and feature extraction to automatically identify the training of athletes. The mathematical model of feature image recognition based on gray difference is established, and the pyramid iterative recognition algorithm is used to reduce the recognition error effectively. In addition, a mathematical model of image sequence feature extraction based on moment invariants is established, and the feature table for athlete matching is discussed in detail. Based on the concept of dynamic establishment of search area and the principle of two‐step template feature recognition and matching, through the analysis of the pictures of high jumpers, the change of the athlete angle of left knee in the process of high jump is obtained, which achieves the purpose of automatic identification of key actions. At the same time, the random error existing in manual recognition is completely eliminated.

ParaFormer: Parallel Attention Transformer for Efficient Feature Matching

Heavy computation is a bottleneck limiting deep-learning-based feature matching algorithms to be applied in many real-time applications. However, existing lightweight networks optimized for Euclidean data cannot address classical feature matching tasks, since sparse keypoint based descriptors are expected to be matched. This paper tackles this problem and proposes two concepts: 1) a novel parallel attention model entitled ParaFormer and 2) a graph based U-Net architecture with attentional pooling. First, ParaFormer fuses features and keypoint positions through the concept of amplitude and phase, and integrates self- and cross-attention in a parallel manner which achieves a win-win performance in terms of accuracy and efficiency. Second, with U-Net architecture and proposed attentional pooling, the ParaFormer-U variant significantly reduces computational complexity, and minimize performance loss caused by downsampling. Sufficient experiments on various applications, including homography estimation, pose estimation, and image matching, demonstrate that ParaFormer achieves state-of-the-art performance while maintaining high efficiency. The efficient ParaFormer-U variant achieves comparable performance with less than 50% FLOPs of the existing attention-based models.

HTMatch: An efficient hybrid transformer based graph neural network for local feature matching

Local feature matching plays a vital role in various computer vision tasks. In this work, we present a novel network that combines feature matching and outlier rejection for finding reliable correspondences between image pairs. The proposed method is a hybrid transformer-based graph neural network (GNN), termed HTMatch, which aims to achieve high accuracy and efficient feature matching. Specifically, we first propose a hybrid transformer that integrates self- and cross-attention together to condition the feature descriptors between image pairs. By doing so, the intra/inter-graph attentional aggregation can be realized by a single transformer layer, which achieves more efficient message passing. Then, we introduce a new spatial embedding module to enhance the spatial constraints across images. The spatial information from one image is embedded into another, which can significantly improve matching performance. Finally, we adopt a seeded GNN architecture for establishing a sparse graph, which improves both efficiency and effectiveness. Experiments show that HTMatch reaches state-of-the-art results on several public benchmarks.

Efficient and Robust Feature Matching for High-Resolution Satellite Stereos

Feature matching between high-resolution satellite stereos plays an important role in satellite image orientation. However, images of changed regions, weak-textured regions and occluded regions may generate low-quality matches or even mismatches. Furthermore, matching throughout the entire satellite images often has extremely high time cost. To compute good matching results at low time cost, this paper proposes an image block selection method for high-resolution satellite stereos, which processes feature matching in several optimal blocks instead of the entire images. The core of the method is to formulate the block selection into the optimization of an energy function, and a greedy strategy is designed to compute an approximate solution. The experimental comparisons on various satellite stereos show that the proposed method could achieve similar matching accuracy and much lower time cost when compared with some state-of-the-art satellite image matching methods. Thus, the proposed method is a good compromise between matching accuracy and matching time, which has great potential in large-scale satellite applications.

Visual Loop Closure Detection for a Future Mars Science Helicopter

Future Mars Rotorcrafts will require the ability to precisely navigate to previously visited locations in order to return to a safe landing site or execute precise scientific measurements, such as sample acquisition or targeted sensing. To enable a future Mars Science Helicopter to perform in-flight loop closures, we present an on-board visual loop closure detection system based on a Bag-of-Words (BoW) approach that is efficient enough to run in real-time on the anticipated computationally constrained flight avionics. Our system establishes image-to-image associations between incoming images of a downward-looking navigation camera and previously observed geo-tagged keyframes stored in a database. The system extracts ORB features which are quantized into a BoW histogram using a custom one million words hierarchical vocabulary, trained on synthetic images from a Mars simulation. An efficient database query using an inverted index produces a set of candidate frames which we check for geometrical consistency. For efficient feature matching, we leverage the vocabulary to perform fast approximate nearest neighbor search. The geometrical check accepts loop closure pairs whose essential matrix is supported by a minimum number of feature matches, which are pre-selected based on a rotational consistency check. The vocabulary and the methods used for the geometrical consistency checks were chosen to maximize the performance while allowing single-thread real-time execution on a computationally constrained embedded processor. We demonstrate and evaluate the proposed system both on simulated and real-world data, including flight data from the Mars Helicopter <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ingenuity</i> .

Dual-View Stereovision-Guided Automatic Inspection System for Overhead Transmission Line Corridor

Overhead transmission line corridor detection is important to ensure the safety of power facilities. Frequent and uncertain changes in the transmission line corridor environment requires an efficient and autonomous UAV inspection system, whereas the existing UAV-based inspection systems has some shortcomings in control model and ground clearance measurement. For one thing, the existing manual control model has the risk of striking power lines because it is difficult for manipulators to judge the distance between the UAV fuselage and power lines accurately. For another, the ground clearance methods based on UAV usually depend on LiDAR (Light Detection and Ranging) or single-view visual repeat scanning, with which it is difficult to balance efficiency and accuracy. Aiming at addressing the challenging issues above, a novel UAV inspection system is developed, which can sense 3D information of transmission line corridor by the cooperation of the dual-view stereovision module and an advanced embedded NVIDIA platform. In addition, a series of advanced algorithms are embedded in the system to realize autonomous control of UAVs and ground clearance measurement. Firstly, an edge-assisted power line detection method is proposed to locate the power line accurately. Then, 3D reconstruction of the power line is achieved based on binocular vision, and the target flight points are generated in the world coordinate system one-by-one to guide the UAVs movement along power lines autonomously. In order to correctly detect whether the ground clearances are in the range of safety, we propose an aerial image classification based on a light-weight semantic segmentation network to provide auxiliary information categories of ground objects. Then, the 3D points of ground objects are reconstructed according to the matching points set obtained by an efficient feature matching method, and concatenated with 3D points of power lines. Finally, the ground clearance can be measured and detected according to the generated 3D points of the transmission line corridor. Tests on both corresponding datasets and practical 220-kV transmission line corridors are conducted. The experimental results of different modules reveal that our proposed system can be applied in practical inspection environments and has good performance.

Efficient probability-oriented feature matching using wide field-of-view imaging

Feature matching is a key technique for a wide variety of computer vision and image processing applications such as visual localization. It permits finding correspondences of significant points within the environment that eventually determine the localization of a mobile agent. In this context, this work evaluates an Adaptive Probability-Oriented Feature Matching (APOFM) method that dynamically models the visual knowledge of the environment in terms of the probability of existence of features. Several improvements are proposed to achieve a more robust matching in a visual odometry framework: a study on the classification of the matching candidates, enhanced by a nearest neighbour search policy; a dynamic weighted matching that exploits the probability of feature existence in order to tune the matching thresholds; and an automatic false positive detector. Additionally, a comparison of performance is carried out, considering a publicly available dataset composed of two kinds of wide field-of-view images: catadioptric and fisheye. Overall, the results validate the appropriateness of these contributions, which outperform other well-recognized implementations within this framework, such as the standard visual odometry, a visual odometry method based on RANSAC, as well as the basic APOFM. The analysis shows that fisheye images provide more visual information of the scene, with more feature candidates. Contrarily, omnidirectional images produce fewer feature candidates, but with higher ratios of feature acceptance. Finally, it is concluded that improved precision is obtained when the location problem is solved by this method.

Low-Power Dynamic Object Detection and Classification With Freely Moving Event Cameras.

We present the first purely event-based, energy-efficient approach for dynamic object detection and categorization with a freely moving event camera. Compared to traditional cameras, event-based object recognition systems are considerably behind in terms of accuracy and algorithmic maturity. To this end, this paper presents an event-based feature extraction method devised by accumulating local activity across the image frame and then applying principal component analysis (PCA) to the normalized neighborhood region. Subsequently, we propose a backtracking-free k-d tree mechanism for efficient feature matching by taking advantage of the low-dimensionality of the feature representation. Additionally, the proposed k-d tree mechanism allows for feature selection to obtain a lower-dimensional object representation when hardware resources are limited to implement PCA. Consequently, the proposed system can be realized on a field-programmable gate array (FPGA) device leading to high performance over resource ratio. The proposed system is tested on real-world event-based datasets for object categorization, showing superior classification performance compared to state-of-the-art algorithms. Additionally, we verified the real-time FPGA performance of the proposed object detection method, trained with limited data as opposed to deep learning methods, under a closed-loop aerial vehicle flight mode. We also compare the proposed object categorization framework to pre-trained convolutional neural networks using transfer learning and highlight the drawbacks of using frame-based sensors under dynamic camera motion. Finally, we provide critical insights about the feature extraction method and the classification parameters on the system performance, which aids in understanding the framework to suit various low-power (less than a few watts) application scenarios.

Efficient fuzzy feature matching and optimal feature points for multiple objects tracking in fixed and active camera models

In computer vision, the multiple objects tracking play a vital challenging role. To solve the issues in this research field, various traditional techniques had been developed. In this paper, we consider the problem of tracking multiple persons in a dynamic environment (background) such as illumination changes and shadow moving. Notably, i) Estimating camera motion and ii) Multiple persons tracking are the two main phases involved in our proposed approach. In the first phase, the good features were extracted using both the SIFT features extraction steps and Gaussian noise elimination method. Instead of using the conventional SIFT-based matching method, we have introduced a new fuzzy matching method to create an adaptive matching zone (region). Using this, the two corresponding features from different frames can be matched perfectly. The brightness of a matching feature of interest indicates its size. Additionally, we use the magnitude and direction of the motion for accurate elimination of camera motion. In the second phase, the persons are tracked from the moving object by finding the optimal feature points and clustering of final points are made as the moving persons (objects). Experimental validation was performed on different challenging datasets and promising results are achieved by our proposed method compared to other existing methods.

Efficient Feature Matching via Nonnegative Orthogonal Relaxation

Feature matching problem that incorporates pair-wise constraints can be formulated as an Integer Quadratic Programming (IQP) problem with one-to-one matching constraint. Since it is NP-hard, relaxation models are required. One main challenge for optimizing IQP matching is how to incorporate the discrete one-to-one matching constraint in IQP matching optimization. In this paper, we present a new feature matching relaxation model, called Nonnegative Orthogonal Relaxation (NOR), that aims to optimize IQP matching problem in nonnegative orthogonal domain. One important benefit of the proposed NOR model is that it can naturally incorporate the discrete one-to-one matching constraint in its optimization and can return a desired sparse (approximate discrete) solution for the problem. An efficient and effective update algorithm has been developed to solve the proposed NOR model. Promising experimental results on several benchmark datasets demonstrate the effectiveness and efficiency of the proposed NOR method.

Robust and efficient multi-source image matching method based on best-buddies similarity measure

Robust and efficient feature matching in multi-source image is a difficult task duo to significant noise and intensity changes. Motivated by problems encountered in present methods, a local similarity descriptor, called local central-tendency similarity (LCTS), is proposed to eliminate the effects of noise and intensity changes by establishing the correlation between the central pixels and surrounding regions of interest. Then, an efficient template-matching framework–namely, the best-buddies direction pairs (BBDP)–is developed for multi-source images by calculating the number of nearest-neighbor points between template and target. When tested on 200 pairs of multi-source images acquired from unconstrained environments, the results demonstrate that BBDP achieves higher computational efficiency and better matching performance than many popular multi-source image matching algorithms.

Latent Fingerprint Segmentation Based on Ridge Density and Orientation Consistency

Latent fingerprints are captured from the fingerprint impressions left unintentionally at the surfaces of the crime scene. They are often used as an important evidence to identify criminals in law enforcement agencies. Different from the widely used plain and rolled fingerprints, the latent fingerprints are usually of poor quality consisting of complex background with a lot of nonfingerprint patterns and various noises. Latent fingerprint segmentation is an important image processing step to separate fingerprint foreground from background for more accurate and efficient feature extraction and matching. Traditional methods are usually based on the local features such as gray scale variance and gradients, which are sensitive to noise and cannot work well for latent images. This paper proposes a latent fingerprint segmentation method based on combination of ridge density and orientation consistency, which are global and local features of fingerprints, respectively. First, a texture image is obtained by decomposition of latent image with a total variation model. Second, we propose to detect the ridge segments from the texture image, and then compute the density of ridge segments and ridge orientation consistency to characterize the global and local fingerprint patterns. Finally, fingerprint segmentation is performed by combining the ridge density and orientation consistency for latent images. The proposed method has been evaluated on NIST SD27 latent fingerprint database. Experimental results and comparison demonstrate the promising performance of the proposed method.

On-Board GNSS/IMU Assisted Feature Extraction and Matching for Oblique UAV Images

Feature extraction and matching is a crucial task in the fields of computer vision and photogrammetry. Even though wide researches have been reported, some issues are still existing for oblique images. This paper exploits the use of on-board GNSS/IMU (Global Navigation Satellite System/Inertial Measurement Unit) data to achieve efficient and reliable feature extraction and matching for oblique unmanned aerial vehicle (UAV) images. Firstly, rough POS (Positioning and Orientation System) is calculated for each image with cooperation of on-board GNSS/IMU data and camera installation angles, which enables image rectification and footprint calculation. Secondly, two robust strategies, including the geometric rectification and tile strategy, are considered to address the issues caused by perspective deformations and to relieve the side-effects of image down-sampling. According to the results of individual performance evaluation, four combinations of these two strategies are designed and comprehensively compared in BA (Bundle Adjustment) experiments by using a real oblique UAV dataset. The results reported in this paper demonstrate that the solution with the tiling strategy is superior to the other solutions in terms of efficiency, completeness and accuracy. For feature extraction and matching of oblique UAV images, it is proposed to combine the tiling strategy with existing workflows to achieve an efficient and reliable solution.

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibanez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.

English

Interest points are widely used in computer vision applications such as camera calibration, robot localization and object tracking that require fast and efficient feature matching. A large number of techniques have been proposed in the literature. Such comparative study is crucial for specific applications. It is always necessary to understand the advantages and disadvantages of the existing techniques so that best possible ones can be selected. In this paper a study of Harris, Moravec and SUSAN Corner detection Algorithms has been done for obtaining features required to track and recognize objects in an image. Corner detection of noisy images is a challenging task in image processing. Natural images often get corrupted by noise during acquisition and transmission. As Corner detection of these noisy images does not provide desired results, hence de noising is required.

Formula omitted]-SIFT: SIFT feature extraction and matching for large images in large-scale aerial photogrammetry

The primary contribution of this paper is an efficient feature extraction and matching implementation for large images in large-scale aerial photogrammetry experiments. First, a Block-SIFT method is designed to overcome the memory limitation of SIFT for extracting and matching features from large photogrammetric images. For each pair of images, the original large image is split into blocks and the possible corresponding blocks in the other image are determined by pre-estimating the relative transformation between the two images. Because of the reduced memory requirement, features can be extracted and matched from the original images without down-sampling. Next, a red-black tree data structure is applied to create a feature relationship to reduce the search complexity when matching tie points. Meanwhile, tree key exchange and segment matching methods are proposed to match the tie points along-track and across-track. Finally, to evaluate the accuracy of the features extracted and matched from the proposed L2-SIFT algorithm, a bundle adjustment with parallax angle feature parametrization (ParallaxBA1The ParallaxBA source code is available open-source at OpenSLAM http://openslam.org/ParallaxBA.html. The features of the test datasets in this paper as the input to ParallaxBA are also available at OpenSLAM.1) is applied to obtain the Mean Square Error (MSE) of the feature reprojections, where the feature extraction and matching result is the only information used in the nonlinear optimisation system. Seven different experimental aerial photogrammetric datasets are used to demonstrate the efficiency and validity of the proposed algorithm. It is demonstrated that more than 33 million features can be extracted and matched from the Taian dataset with 737 images within 21h using the L2-SIFT algorithm. In addition, the ParallaxBA involving more than 2.7 million features and 6 million image points can easily converge to an MSE of 0.03874. The C/C++ source code for the proposed algorithm is available at http://services.eng.uts.edu.au/sdhuang/research.htm.

Efficient and Robust Feature Matching via Local Descriptor Generalized Hough Transform

Robust and efficient indistinctive feature matching and outliers removal is an essential problem in many computer vision applications. In this paper we present a simple and fast algorithm named as LDGTH (Local Descriptor Generalized Hough Transform) to handle this problem. The main characteristics of the proposed method include: (1) A novel local descriptor generalized hough transform framework is presented in which the local geometric characteristics of invariant feature descriptors are fused together as a global constraint for feature correspondence verification. (2) Different from standard generalized hough transform, our approach greatly reduces the computational and storage requirements of parameter space through taking advantage of the invariant feature correspondences. (3) The proposed algorithm can be seamlessly embedded into the existing image matching framework, and significantly improve the image matching performance both in speed and robustness in challenge conditions. In the experiment we use both synthetic image data and real world data with high outliers ratio and severe changes in view point, scale, illumination, image blur, compression and noises to evaluate the proposed method, and the results demonstrate that our approach achieves achieves faster and better matching performance compared to the traditional algorithms.

Markerless vision-based augmented reality for enhanced project visualization

Purpose This work aims to develop a system that enables improved visualization of Architectural Engineering and Construction (AEC) 3D-information for application in design, construction, and management of the built environment. Method A novel Augmented Reality (AR) system is presented that uses a single standard digital camera and which, contrary to other investigated approaches, does not rely on any markers inserted in the scene, nor on any positioning and inertial technologies. The system is solely image-based and consists of two stages. In a first offline stage, a 3D-map of the scene is automatically constructed from a set of digital images, and the augmenting information (e.g. the 3D-model of the building asset) is subsequently registered with this map. The 3D-map reconstruction employs structure-from-motion techniques with SURF features (the resulting map consisting of a set 3D-referenced SURF-features) followed by a Poisson mesh reconstruction procedure. The next step consists of online operations. The positions of target digital images (e.g. from video stream or head-mounted camera) are automatically calculated, using a robust SURF feature matching procedure that is optimized for three different situations (initialization, tracking, and resetting) implementing octrees for efficient 3D-pruning, and kd-trees for efficient feature matching. Once each input image is positioned within the map, the view is augmented. A notable feature of dense mesh scene reconstruction conducted in the present work is that it enables static occlusions of the scene on the augmenting data to be taken into account. Results & Discussion Several experiments validate the pr posed system and demonstrate its overall performance: a near real-time processing speed, very accurate and stable positioning. The limitations of the current system are also discussed including: the currently limited processing speed and the need for adequately textured scenes.