Keypoint Matching Research Articles

An Enhanced Multiscale Retinex, Oriented FAST and Rotated BRIEF (ORB), and Scale-Invariant Feature Transform (SIFT) Pipeline for Robust Key Point Matching in 3D Monitoring of Power Transmission Line Icing with Binocular Vision

Power transmission line icing (PTLI) poses significant threats to the reliability and safety of electrical power systems, particularly in cold regions. Accumulation of ice on power lines can lead to severe consequences, such as line breaks, tower collapses, and widespread power outages, resulting in economic losses and infrastructure damage. This study proposes an enhanced image processing pipeline to accurately detect and match key points in PTLI images for 3D monitoring of ice thickness using binocular vision. The pipeline integrates established techniques such as multiscale retinex (MSR), oriented FAST and rotated BRIEF (ORB) and scale-invariant feature transform (SIFT) algorithms, further refined with m-estimator sample consensus (MAGSAC)-based random sampling consensus (RANSAC) optimization. The image processing steps include automatic cropping, image enhancement, feature detection, and robust key point matching, all designed to operate in challenging environments with poor lighting and noise. Experiments demonstrate that the proposed method significantly improves key point matching accuracy and computational efficiency, reducing processing time to make it suitable for real-time applications. The effectiveness of the pipeline is validated through 3D ice thickness measurements, with results showing high precision and low error rates, making it a valuable tool for monitoring power transmission lines in harsh conditions.

Point of Interest Recognition and Tracking in Aerial Video during Live Cycling Broadcasts

Road cycling races, such as the Tour de France, captivate millions of viewers globally, combining competitive sportsmanship with the promotion of regional landmarks. Traditionally, points of interest (POIs) are highlighted during broadcasts using manually created static overlays, a process that is both outdated and labor-intensive. This paper presents a novel, fully automated methodology for detecting and tracking POIs in live helicopter video streams, aiming to streamline the visualization workflow and enhance viewer engagement. Our approach integrates a saliency and Segment Anything-based technique to propose potential POI regions, which are then recognized using a keypoint matching method that requires only a few reference images. This system supports both automatic and semi-automatic operations, allowing video editors to intervene when necessary, thereby balancing automation with manual control. The proposed pipeline demonstrated high effectiveness, achieving over 75% precision and recall in POI detection, and offers two tracking solutions: a traditional MedianFlow tracker and an advanced SAM 2 tracker. While the former provides speed and simplicity, the latter delivers superior segmentation tracking, albeit with higher computational demands. Our findings suggest that this methodology significantly reduces manual workload and opens new possibilities for interactive visualizations, enhancing the live viewing experience of cycling races.

Hierarchical Graph Neural Network: A Lightweight Image Matching Model with Enhanced Message Passing of Local and Global Information in Hierarchical Graph Neural Networks

Graph Neural Networks (GNNs) have gained popularity in image matching methods, proving useful for various computer vision tasks like Structure from Motion (SfM) and 3D reconstruction. A well-known example is SuperGlue. Lightweight variants, such as LightGlue, have been developed with a focus on stacking fewer GNN layers compared to SuperGlue. This paper proposes the h-GNN, a lightweight image matching model, with improvements in the two processing modules, the GNN and matching modules. After image features are detected and described as keypoint nodes of a base graph, the GNN module, which primarily aims at increasing the h-GNN’s depth, creates successive hierarchies of compressed-size graphs from the base graph through a clustering technique termed SC+PCA. SC+PCA combines Principal Component Analysis (PCA) with Spectral Clustering (SC) to enrich nodes with local and global information during graph clustering. A dual non-contrastive clustering loss is used to optimize graph clustering. Additionally, four message-passing mechanisms have been proposed to only update node representations within a graph cluster at the same hierarchical level or to update node representations across graph clusters at different hierarchical levels. The matching module performs iterative pairwise matching on the enriched node representations to obtain a scoring matrix. This matrix comprises scores indicating potential correct matches between the image keypoint nodes. The score matrix is refined with a ‘dustbin’ to further suppress unmatched features. There is a reprojection loss used to optimize keypoint match positions. The Sinkhorn algorithm generates a final partial assignment from the refined score matrix. Experimental results demonstrate the performance of the proposed h-GNN against competing state-of-the-art (SOTA) GNN-based methods on several image matching tasks under homography, estimation, indoor and outdoor camera pose estimation, and 3D reconstruction on multiple datasets. Experiments also demonstrate improved computational memory and runtime, approximately 38.1% and 26.14% lower than SuperGlue, and an average of about 6.8% and 7.1% lower than LightGlue. Future research will explore the effects of integrating more recent simplicial message-passing mechanisms, which concurrently update both node and edge representations, into our proposed model.

VSLAM based on deep learning in low-textured scenes

Abstract Traditional Visual Simultaneous Localization and Mapping (VSLAM) algorithms often suffer from the low positioning accuracy and robustness of the traditional feature extraction methods. These methods typically require environments with clear features, perform poorly or even fail in low-textured scenes. To solve this problem, we propose an RGB-D VSLAM algorithm that integrates a deep learning feature extraction network. First, to solve the problem in which sufficient feature points cannot be extracted, the proposed algorithm uses the R2D2 feature extraction network to extract keypoints and generate descriptors. We also perform lightweight optimization on the R2D2 network, ensuring accuracy while doubling the inference speed. Subsequently, we employ the L2 distance to measure the distance between descriptors, thereby enhancing the accuracy of keypoint matching. Ultimately, we add a dense mapping thread to obtain a dense 3D point map generated by the RGB-D information and poses of the keyframes. The experimental results on the TUM dataset show that the average absolute trajectory error is reduced by 59% and the relative trajectory error is reduced by 56%. The results demonstrate that our algorithm significantly improves localization accuracy and system robustness in low-textured environments.

Development and Evaluation of a Two-Staged 3D Keypoint Based Workflow for the Co-Registration of Unstructured Multi-Temporal and Multi-Modal 3D Point Clouds

Abstract. Robust and automated point cloud registration methods are required in many geoscience applications using multi-temporal and multi-modal 3D point clouds. Therefore, a 3D keypoint-based coarse registration workflow has been implemented, utilizing the ISS keypoint detector and 3DSmoothNet descriptor. This paper contributes to keypoint-based registration research through variations of the standard workflow proposed in the literature, applying a two-staged strategy of global and local keypoint matching as well as prototypical keypoint projection and fine registration based on ICP. Further, by testing the utilized detector and descriptor on unstructured, multi-temporal and multi-source point clouds with variations in point cloud density, generalization ability is tested outside benchmark data. Therefore, data of the Bøverbreen glacier in Jotunheimen, Norway has been acquired in 2022 and 2023, deploying UAV-based image matching and terrestrial laser scanning. The results show good performance of the implemented robust matching algorithm PROSAC, requiring fewer iterations than the well-known RANSAC approach, but solving the rigid body transformation with TEASER++ is faster and more robust to outliers without demanding pre-knowledge of the data. Further, the results identify the keypoint detection as most limiting factor in speed and accuracy. Summarizing, keypoint-based coarse registration on low density point clouds, applying a global and local matching strategy and transformation estimation using TEASER++ is recommended. Keypoint projection shows potential, increasing number and precision in low density clouds, but has to be more robust. Further research needs to be carried out, focusing on identifying a fast and robust keypoint detector.

A Deep-Based Approach for Multi-Descriptor Feature Extraction: Applications on SAR Image Registration

Synthetic aperture radar (SAR) images have found increasing attention in various applications, such as remote sensing, surveillance, and disaster management. Unlike optical sensors, SAR can capture images in any weather conditions at any time, making it particularly valuable. However, it poses unique challenges for image processing tasks, like complex speckle noise, texture, and geometric distortions. For many image processing tasks such as image registration; feature detection and extraction are critical initial steps. In this work, we propose a deep learning-based method for detecting and describing keypoints in SAR images, which can be specifically applied to SAR image registration tasks. To address the challenges of SAR images, we utilize the R2D2 network and propose a training scheme specifically tailored for these images. Furthermore, our approach involves multiple descriptors utilization for outlier removal between image keypoints and a two-stage pipeline consisting of coarse and fine stages for robust feature point matching of SAR images. By leveraging the coarse registration stage, which maintains robustness, and also a fine stage that employs multiple descriptors along with a local search technique, our approach enhances the keypoint matching accuracy. Evaluation results show that the proposed method outperforms most of the traditional and deep learning-based methods in terms of the percentage of correct matches, while achieving comparable or even better root mean squared error (RMSE) results.

A Robust Mismatch Removal Method for Image Matching Based on the Fusion of the Local Features and the Depth

Feature point matching is a fundamental task in computer vision such as vision simultaneous localization and mapping (VSLAM) and structure from motion (SFM). Due to the similarity or interference of features, mismatches are often unavoidable. Therefore, how to eliminate mismatches is important for robust matching. Smoothness constraint is widely used to remove mismatch, but it cannot effectively deal with the issue in the rapidly changing scene. In this paper, a novel LCS-SSM (Local Cell Statistics and Structural Similarity Measurement) mismatch removal method is proposed. LCS-SSM integrates the motion consistency and structural similarity of a local image block as the statistical likelihood of matched key points. Then, the Random Sampling Consensus (RANSAC) algorithm is employed to preserve the isolated matches that do not satisfy the statistical likelihood. Experimental and comparative results on the public dataset show that the proposed LCS-SSM can effectively and reliably differentiate true and false matches compared with state-of-the-art methods, and can be used for robust matching in scenes with fast motion, blurs, and clustered noise.

Visual place recognition for autonomous mobile robot navigation using LoFTR and MAGSAC++

Autonomous mobile robots are defined as robotic entities capable of independent movement and intelligent decision-making, relying on their ability to perceive and analyze their surroundings, including objects in their environment. In Simultaneous Localization and Mapping (SLAM) systems, loop closure is often achieved through visual place recognition techniques, where the system compares the current visual input with previously observed scenes to identify matches. In computer vision applications, Speeded-Up Robust Features (SURF) and Scale-Invariant Feature Transform (SIFT) are popular feature extraction algorithms used for such as key point detection, matching, and image registration tasks. The choice of inlier threshold should be based on the specific characteristics of the application and the nature of the images being processed. It often requires experimentation and tuning to find the optimal balance between robustness and accuracy. It Utilizes the pre-trained Local Feature Transformer (LoFTR) and MAGSAC++ estimator to address these drawbacks by employing the number of inliers to determine the similarity between two images for visual place recognition. Our experiment demonstrates that the number of inliers can determine the similarity of locations between two images. Scale variations and translation in location significantly influence the resulting number of inliers. Comparing images from the same location and from different locations yields varying numbers of inliers. The number of inliers significantly influences the similarity of locations. At the same location, the number of inliers is above 150, while at different locations, the number is below 150.

Discrete Cycle-Consistency Based Unsupervised Deep Graph Matching

We contribute to the sparsely populated area of unsupervised deep graph matching with application to keypoint matching in images. Contrary to the standard supervised approach, our method does not require ground truth correspondences between keypoint pairs. Instead, it is self-supervised by enforcing consistency of matchings between images of the same object category. As the matching and the consistency loss are discrete, their derivatives cannot be straightforwardly used for learning. We address this issue in a principled way by building our method upon the recent results on black-box differentiation of combinatorial solvers. This makes our method exceptionally flexible, as it is compatible with arbitrary network architectures and combinatorial solvers. Our experimental evaluation suggests that our technique sets a new state-of-the-art for unsupervised graph matching.

Restoration of UAV-Based Backlit Images for Geological Mapping of a High-Steep Slope.

Unmanned aerial vehicle (UAV)-based geological mapping is significant for understanding the geological structure in the high-steep slopes, but the images obtained in these areas are inevitably influenced by the backlit effect because of the undulating terrain and the viewpoint change of the camera mounted on the UAV. To handle this concern, a novel backlit image restoration method is proposed that takes the real-world application into account and addresses the color distortion issue existing in backlit images captured in high-steep slope scenes. Specifically, there are two main steps in the proposed method, which consist of the backlit removal and the color and detail enhancement. The backlit removal first eliminates the backlit effect using the Retinex strategy, and then the color and detail enhancement step improves the image color and sharpness. The author designs extensive comparison experiments from multiple angles and applies the proposed method to different engineering applications. The experimental results show that the proposed method has potential compared to other main-stream methods both in qualitative visual effects and universal quantitative evaluation metrics. The backlit images processed by the proposed method are significantly improved by the process of feature key point matching, which is very conducive to the fine construction of 3D geological models of the high-steep slope.

The Psychological Construction of Early Childhood Art Education

The psychological construction of early childhood art education refers to the process of promoting children's mental health, emotional development, and personality shaping through art education activities. Art Education provides a platform for early childhood to express themselves, achieve self-realization, unleash their wisdom, develop own aesthetic standards, learn cooperate and communication skills, achieve psychological balance and establish understanding and respect. This study highlights the strategic shift in the approach to art education in China, with a focus on interdisciplinary education and the integration of art into extracurricular activities. With the issue on knowledge is often imparted in a one-way manner and children is encouraged to passively imitate, does not align with the developmental needs of young children or the goals of comprehensive education, this paper researches the dynamic process that involves teachers, students, and sometimes parents and community members in a cycle of planning, acting, observing, and reflecting to improve educational practices, and outlines the Mission and Vision on Development goals and the level of emphasis placed on families and kindergartens, Professional advantages of promoting mental hygiene through art education and Matching of key points, principles, and strategies.

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

CatTrack: Single-Stage Category-Level 6D Object Pose Tracking via Convolution and Vision Transformer

In the current research, many researchers have focused on instance-level pose tracking, which requires a 3D model of the object in advance, making it challenging to apply in practice. To address this limitation, some researchers have proposed the category-level object pose tracking method. Achieving accurate and speedy monocular category-level pose tracking is an essential research goal. In this paper, we propose CatTrack, a new single-stage keypoints-based monocular category-level multi-object pose tracking network. A significant issue in object pose tracking tasks is utilizing the information from the previous frame to guide pose estimation for the next frame. However, as the object poses and camera information in each frame are different, we need to remove irrelevant information and emphasize useful features. To this end, we propose a transformer-based temporal information capture module to leverage the position information of keypoints from the previous frame. Furthermore, we propose a new keypoint matching module to enable the grouping and matching of object keypoints in complex scenes. We have successfully applied CatTrack to the Objectron dataset and achieved superior results in comparison to existing methods. Furthermore, we have also evaluated the generalization of CatTrack and successfully applied it to track the 6D pose of unseen real-world objects. A video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/Yminjdtsgwk</uri> .

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses.

Neural Radiance Fields (NeRFs) have shown impressive capabilities in synthesizing photorealistic novel views. However, their application to room-size scenes is limited by the requirement of several hundred views with accurate poses for training. To address this challenge, we propose SN 2 eRF, a framework which can reconstruct the neural radiance field with significantly fewer views and noisy poses by exploiting multiple priors. Our key insight is to leverage both multi-view and monocular priors to constrain the optimization of NeRF in the setting of sparse and noisy pose inputs. Specifically, we extract and match key points to constrain pose optimization and use Ray Transformer with a monocular depth estimator to provide dense depth prior for geometry optimization. Benefiting from these priors, our approach achieves state-of-the-art accuracy in novel view synthesis for indoor room scenarios.

Prediction of Clean Coal Ash Content in Coal Flotation through a Convergent Model Unifying Deep Learning and Likelihood Function, Incorporating Froth Velocity and Reagent Dosage Parameters

This study successfully achieved high-precision detection of the clean coal ash content in the coal froth flotation domain by integrating deep learning with the likelihood function. Methodologically, a novel data processing and prediction framework was established by combining a deep learning Keras neural network with the likelihood function from probability statistics. The SIFT algorithm was utilized to extract key feature points and descriptors from the images, and keypoint matching and mean-shift clustering algorithms were employed to accurately obtain information on foam motion trajectories and velocities. For parameter optimization, the maximum likelihood estimation was applied to find the optimal parameter estimates of the likelihood function, ensuring enhanced model accuracy. By incorporating the optimized likelihood function parameters into the Keras deep neural network, an efficient prediction model was constructed for the dosage of flotation reagents, froth velocity, and clean coal ash content. The model’s evaluation involved six performance metrics. The experimental results were highly significant, with R2 at 0.99997%, RMSE at 0.04458%, MAE at 0.00170%, MAPE at 0.02329%, RRSE at 0.00994%, and MAAPE at 0.00067%.

Robust coverless video steganography based on inter-frame keypoint matching

Existing coverless video steganography schemes primarily focus on single-frame features, which manifest poor robustness. To enhance resistance against adversarial attacks, this paper proposes a coverless video steganography method based on inter-frame keypoint matching, leveraging the movement orientation of keypoints within the video for information hiding. Initially, the method divides video frames into blocks, and then constructs an inter-frame keypoint matching network by using the keypoint detection method combined with deep learning. By narrowing down the matching scope and filtering out mismatches, the network effectively discerns the keypoint matching relationships. For keypoints successfully matched within the blocks, a coordinate system is established based on the original keypoint, determining the position of the matched keypoints. The most frequently occurring orientation within the blocks is designated as the orientation of the block. Subsequently, we introduce a mapping rule based on quadrant labels, generating robust hash sequences according to the quadrant labels corresponding to the orientation of the blocks. Compared to the latest coverless video steganography schemes, this method shows superior robustness in resisting 14 common traditional attacks and video attacks. Concurrently, it exhibits commendable hiding capacity and hiding success rate.

A fully‐automatic side‐scan sonar simultaneous localization and mapping framework

AbstractSide‐scan sonar is a lightweight acoustic sensor that is frequently deployed on autonomous underwater vehicles (AUVs) to provide high‐resolution seafloor images. However, using side‐scan images to perform simultaneous localization and mapping (SLAM) remains a challenge when there is a lack of 3D bathymetric information and discriminant features in the side‐scan images. To tackle this, the authors propose a feature‐based SLAM framework using side‐scan sonar, which is able to automatically detect and robustly match keypoints between paired side‐scan images. The authors then use the detected correspondences as constraints to optimise the AUV pose trajectory. The proposed method is evaluated on real data collected by a Hugin AUV, using as a ground truth reference both manually‐annotated keypoints and a 3D bathymetry mesh from multibeam echosounder (MBES). Experimental results demonstrate that this approach is able to reduce drifts from the dead‐reckoning system. The framework is made publicly available for the benefit of the community.

Self-training based augmented reality for robust 3D object registration and task assistance

Typical augmented reality (AR) technologies superimpose virtual information onto the real environment for task guidance and assistance by tracking fiducial markers. However, these have narrow recognition ranges due to the markers, and it is difficult to track markers when occlusion occurs. Therefore, it is challenging to seamlessly register real objects in AR scenes captured by the camera without requiring fiducial markers. Deep learning–based approaches to 3D object registration and task assistance have recently indicated promising directions for superimposing 3D virtual objects onto corresponding real objects or their surroundings seamlessly. However, since supervised learning-based object registration requires many pose annotations on images for the locations and orientations of real objects, this process is a time-consuming and labor-intensive task. Thus, this paper proposes a new self-training based augmented reality method for seamless real object registration and task assistance without time-consuming pose annotations. As the industrial components or objects have their 3D geometric models, the proposed approach utilizes them as their synthetic data. The automatically generated synthetic dataset is used for training the deep learning model, which outputs a colored coordinate map of objects to be detected. 2D–3D key point matching between the 2D colored coordinate map and 3D colored coordinate model is then performed for 3D object registration in an AR environment. In particular, self-training is performed for fine-tuning the deep learning model by generating pseudo-labels with the highest probability for mask intersection over union (IoU) in the unlabeled real object dataset. We compared the proposed approach with previous studies to confirm its effectiveness and advantages. Also, we performed ablation studies by training a custom dataset based on various industrial 3D virtual models. Finally, we present AR-based task guidance and assistance, such as industrial robot manipulation.

Enhancing Feature Detection and Matching in Low-Pixel-Resolution Hyperspectral Images Using 3D Convolution-Based Siamese Networks.

Today, hyperspectral imaging plays an integral part in the remote sensing and precision agriculture field. Identifying the matching key points between hyperspectral images is an important step in tasks such as image registration, localization, object recognition, and object tracking. Low-pixel resolution hyperspectral imaging is a recent introduction to the field, bringing benefits such as lower cost and form factor compared to traditional systems. However, the use of limited pixel resolution challenges even state-of-the-art feature detection and matching methods, leading to difficulties in generating robust feature matches for images with repeated textures, low textures, low sharpness, and low contrast. Moreover, the use of narrower optics in these cameras adds to the challenges during the feature-matching stage, particularly for images captured during low-altitude flight missions. In order to enhance the robustness of feature detection and matching in low pixel resolution images, in this study we propose a novel approach utilizing 3D Convolution-based Siamese networks. Compared to state-of-the-art methods, this approach takes advantage of all the spectral information available in hyperspectral imaging in order to filter out incorrect matches and produce a robust set of matches. The proposed method initially generates feature matches through a combination of Phase Stretch Transformation-based edge detection and SIFT features. Subsequently, a 3D Convolution-based Siamese network is utilized to filter out inaccurate matches, producing a highly accurate set of feature matches. Evaluation of the proposed method demonstrates its superiority over state-of-the-art approaches in cases where they fail to produce feature matches. Additionally, it competes effectively with the other evaluated methods when generating feature matches in low-pixel resolution hyperspectral images. This research contributes to the advancement of low pixel resolution hyperspectral imaging techniques, and we believe it can specifically aid in mosaic generation of low pixel resolution hyperspectral images.

Sparse-to-Dense Matching Network for Large-scale LiDAR Point Cloud Registration.

Point cloud registration is a fundamental problem in 3D computer vision. Previous learning-based methods for LiDAR point cloud registration can be categorized into two schemes: dense-to-dense matching methods and sparse-to-sparse matching methods. However, for large-scale outdoor LiDAR point clouds, solving dense point correspondences is time-consuming, whereas sparse keypoint matching easily suffers from keypoint detection error. In this paper, we propose SDMNet, a novel Sparse-to-Dense Matching Network for large-scale outdoor LiDAR point cloud registration. Specifically, SDMNet performs registration in two sequential stages: sparse matching stage and local-dense matching stage. In the sparse matching stage, we sample a set of sparse points from the source point cloud and then match them to the dense target point cloud using a spatial consistency enhanced soft matching network and a robust outlier rejection module. Furthermore, a novel neighborhood matching module is developed to incorporate local neighborhood consensus, significantly improving performance. The local-dense matching stage is followed for fine-grained performance, where dense correspondences are efficiently obtained by performing point matching in local spatial neighborhoods of high-confidence sparse correspondences. Extensive experiments on three large-scale outdoor LiDAR point cloud datasets demonstrate that the proposed SDMNet achieves state-of-the-art performance with high efficiency.