Keypoint Matching Research Articles

Visual Localization Domain for Accurate V-SLAM from Stereo Cameras

Trajectory estimation from stereo image sequences remains a fundamental challenge in Visual Simultaneous Localization and Mapping (V-SLAM). To address this, we propose a novel approach that focuses on the identification and matching of keypoints within a transformed domain that emphasizes visually significant features. Specifically, we propose to perform V-SLAM in a VIsual Localization Domain (VILD), i.e., a domain where visually relevant feature are suitably represented for analysis and tracking. This transformed domain adheres to information-theoretic principles, enabling a maximum likelihood estimation of rotation, translation, and scaling parameters by minimizing the distance between the coefficients of the observed image and those of a reference template. The transformed coefficients are obtained from the output of specialized Circular Harmonic Function (CHF) filters of varying orders. Leveraging this property, we employ a first-order approximation of the image-series representation, directly computing the first-order coefficients through the application of first-order CHF filters. The proposed VILD provides a theoretically grounded and visually relevant representation of the image. We utilize VILD for point matching and tracking across the stereo video sequence. The experimental results on real-world video datasets demonstrate that integrating visually-driven filtering significantly improves trajectory estimation accuracy compared to traditional tracking performed in the spatial domain.

Implementation of an FPGA-Based System to Process Images and Match Keypoints on High-Resolution Pictures

Processing scenery and finding points of interest is crucial for applications in robotics and aerospace missions. Those areas require efficient and reliable visual input processing. Here, field programmable gate arrays (FPGAs) offer essential advantages, like low power consumption compared to CPUs, performing a large number of calculations simultaneously, and having compact hardware. This paper presents an FPGA system that processes incoming camera data, finds points of interest, and matches them across different images on high-resolution images (2048 × 1088). It is a novel approach to implement the complete image processing pipeline on high-resolution images within the FPGA fabric without additional hardware. For keypoint detection and matching, our work uses a modified SIFT algorithm optimized for FPGA implementation processing and a nearest neighbor-based matching method. It was implemented on a Xilinx Kintex-7 FPGA and partially on a NanoXplore NG-Ultra to evaluate a radiation-hardened FPGA for space applications. On the Kintex-7, the keypoint detection achieves a speed of 33 ms per image, and its features are matched on up to 5 images per second. Judging by the resource utilization of one image processing module on the NG-Ultra, porting the entire system on a radiation-hardened FPGA appears feasible.

EyeLiner: A Deep Learning Pipeline for Longitudinal Image Registration using Fundus Landmarks

ObjectiveDetecting and measuring changes in longitudinal fundus imaging is key to monitoring disease progression in chronic ophthalmic diseases, such as glaucoma and macular degeneration. Clinicians assess changes in disease status by either independently reviewing or manually juxtaposing longitudinally acquired color fundus photos (CFPs). Distinguishing variations in image acquisition due to camera orientation, zoom, and exposure from true disease-related changes can be challenging. This makes manual image evaluation variable and subjective, potentially impacting clinical decision making. We introduce our deep learning pipeline, “EyeLiner,” for registering, or aligning, two-dimensional CFPs. Improved alignment of longitudinal image pairs may compensate for differences that are due to camera orientation, while preserving pathological changes. DesignEyeLiner registers a “moving” image to a “fixed” image using a deep learning-based keypoint matching algorithm. ParticipantsWe evaluate EyeLiner on three longitudinal datasets: Fundus Image REgistration (FIRE), Sequential Fundus for Glaucoma Forecast (SIGF), and our internal glaucoma dataset from the Colorado Ophthalmology Research Information System (CORIS). MethodsAnatomical keypoints along the retinal blood vessels are detected from the moving and fixed images using a convolutional neural network and subsequently matched using a transformer-based algorithm. Finally, transformation parameters are learned using the corresponding keypoints. Main Outcome MeasuresWe computed the mean distance (MD) between manually annotated keypoints from the fixed and the registered moving image. For comparison to existing state-of-the-art retinal registration approaches, we used the mean area under the curve (mAUC) metric introduced in the FIRE dataset study. ResultsEyeLiner effectively aligns longitudinal image pairs from FIRE, SIGF and CORIS, as qualitatively evaluated through registration checkerboards and flicker animations. Quantitative results show that the MD decreased for this model after alignment from 321.32 to 3.74 pixels for FIRE, 9.86 to 2.03 pixels for CORIS, and 25.23 to 5.94 pixels for SIGF. We also obtain an AUC of 0.85, 0.94 and 0.84 on FIRE, CORIS and SIGF respectively, beating the current state-of-the-art SuperRetina (AUCFIRE=0.76, AUCCORIS=0.83, AUCSIGF=0.74). ConclusionsOur pipeline demonstrates improved alignment of image pairs in comparison to the current state-of-the-art methods on three separate datasets. We envision that this method will enable clinicians to align image pairs, and better visualize changes in disease over time.

An Enhanced Multiscale Retinex, Oriented FAST and Rotated BRIEF (ORB), and Scale-Invariant Feature Transform (SIFT) Pipeline for Robust Key Point Matching in 3D Monitoring of Power Transmission Line Icing with Binocular Vision

Power transmission line icing (PTLI) poses significant threats to the reliability and safety of electrical power systems, particularly in cold regions. Accumulation of ice on power lines can lead to severe consequences, such as line breaks, tower collapses, and widespread power outages, resulting in economic losses and infrastructure damage. This study proposes an enhanced image processing pipeline to accurately detect and match key points in PTLI images for 3D monitoring of ice thickness using binocular vision. The pipeline integrates established techniques such as multiscale retinex (MSR), oriented FAST and rotated BRIEF (ORB) and scale-invariant feature transform (SIFT) algorithms, further refined with m-estimator sample consensus (MAGSAC)-based random sampling consensus (RANSAC) optimization. The image processing steps include automatic cropping, image enhancement, feature detection, and robust key point matching, all designed to operate in challenging environments with poor lighting and noise. Experiments demonstrate that the proposed method significantly improves key point matching accuracy and computational efficiency, reducing processing time to make it suitable for real-time applications. The effectiveness of the pipeline is validated through 3D ice thickness measurements, with results showing high precision and low error rates, making it a valuable tool for monitoring power transmission lines in harsh conditions.

Point of Interest Recognition and Tracking in Aerial Video during Live Cycling Broadcasts

Road cycling races, such as the Tour de France, captivate millions of viewers globally, combining competitive sportsmanship with the promotion of regional landmarks. Traditionally, points of interest (POIs) are highlighted during broadcasts using manually created static overlays, a process that is both outdated and labor-intensive. This paper presents a novel, fully automated methodology for detecting and tracking POIs in live helicopter video streams, aiming to streamline the visualization workflow and enhance viewer engagement. Our approach integrates a saliency and Segment Anything-based technique to propose potential POI regions, which are then recognized using a keypoint matching method that requires only a few reference images. This system supports both automatic and semi-automatic operations, allowing video editors to intervene when necessary, thereby balancing automation with manual control. The proposed pipeline demonstrated high effectiveness, achieving over 75% precision and recall in POI detection, and offers two tracking solutions: a traditional MedianFlow tracker and an advanced SAM 2 tracker. While the former provides speed and simplicity, the latter delivers superior segmentation tracking, albeit with higher computational demands. Our findings suggest that this methodology significantly reduces manual workload and opens new possibilities for interactive visualizations, enhancing the live viewing experience of cycling races.

Hierarchical Graph Neural Network: A Lightweight Image Matching Model with Enhanced Message Passing of Local and Global Information in Hierarchical Graph Neural Networks

Graph Neural Networks (GNNs) have gained popularity in image matching methods, proving useful for various computer vision tasks like Structure from Motion (SfM) and 3D reconstruction. A well-known example is SuperGlue. Lightweight variants, such as LightGlue, have been developed with a focus on stacking fewer GNN layers compared to SuperGlue. This paper proposes the h-GNN, a lightweight image matching model, with improvements in the two processing modules, the GNN and matching modules. After image features are detected and described as keypoint nodes of a base graph, the GNN module, which primarily aims at increasing the h-GNN’s depth, creates successive hierarchies of compressed-size graphs from the base graph through a clustering technique termed SC+PCA. SC+PCA combines Principal Component Analysis (PCA) with Spectral Clustering (SC) to enrich nodes with local and global information during graph clustering. A dual non-contrastive clustering loss is used to optimize graph clustering. Additionally, four message-passing mechanisms have been proposed to only update node representations within a graph cluster at the same hierarchical level or to update node representations across graph clusters at different hierarchical levels. The matching module performs iterative pairwise matching on the enriched node representations to obtain a scoring matrix. This matrix comprises scores indicating potential correct matches between the image keypoint nodes. The score matrix is refined with a ‘dustbin’ to further suppress unmatched features. There is a reprojection loss used to optimize keypoint match positions. The Sinkhorn algorithm generates a final partial assignment from the refined score matrix. Experimental results demonstrate the performance of the proposed h-GNN against competing state-of-the-art (SOTA) GNN-based methods on several image matching tasks under homography, estimation, indoor and outdoor camera pose estimation, and 3D reconstruction on multiple datasets. Experiments also demonstrate improved computational memory and runtime, approximately 38.1% and 26.14% lower than SuperGlue, and an average of about 6.8% and 7.1% lower than LightGlue. Future research will explore the effects of integrating more recent simplicial message-passing mechanisms, which concurrently update both node and edge representations, into our proposed model.

VSLAM based on deep learning in low-textured scenes

Abstract Traditional Visual Simultaneous Localization and Mapping (VSLAM) algorithms often suffer from the low positioning accuracy and robustness of the traditional feature extraction methods. These methods typically require environments with clear features, perform poorly or even fail in low-textured scenes. To solve this problem, we propose an RGB-D VSLAM algorithm that integrates a deep learning feature extraction network. First, to solve the problem in which sufficient feature points cannot be extracted, the proposed algorithm uses the R2D2 feature extraction network to extract keypoints and generate descriptors. We also perform lightweight optimization on the R2D2 network, ensuring accuracy while doubling the inference speed. Subsequently, we employ the L2 distance to measure the distance between descriptors, thereby enhancing the accuracy of keypoint matching. Ultimately, we add a dense mapping thread to obtain a dense 3D point map generated by the RGB-D information and poses of the keyframes. The experimental results on the TUM dataset show that the average absolute trajectory error is reduced by 59% and the relative trajectory error is reduced by 56%. The results demonstrate that our algorithm significantly improves localization accuracy and system robustness in low-textured environments.

Development and Evaluation of a Two-Staged 3D Keypoint Based Workflow for the Co-Registration of Unstructured Multi-Temporal and Multi-Modal 3D Point Clouds

Abstract. Robust and automated point cloud registration methods are required in many geoscience applications using multi-temporal and multi-modal 3D point clouds. Therefore, a 3D keypoint-based coarse registration workflow has been implemented, utilizing the ISS keypoint detector and 3DSmoothNet descriptor. This paper contributes to keypoint-based registration research through variations of the standard workflow proposed in the literature, applying a two-staged strategy of global and local keypoint matching as well as prototypical keypoint projection and fine registration based on ICP. Further, by testing the utilized detector and descriptor on unstructured, multi-temporal and multi-source point clouds with variations in point cloud density, generalization ability is tested outside benchmark data. Therefore, data of the Bøverbreen glacier in Jotunheimen, Norway has been acquired in 2022 and 2023, deploying UAV-based image matching and terrestrial laser scanning. The results show good performance of the implemented robust matching algorithm PROSAC, requiring fewer iterations than the well-known RANSAC approach, but solving the rigid body transformation with TEASER++ is faster and more robust to outliers without demanding pre-knowledge of the data. Further, the results identify the keypoint detection as most limiting factor in speed and accuracy. Summarizing, keypoint-based coarse registration on low density point clouds, applying a global and local matching strategy and transformation estimation using TEASER++ is recommended. Keypoint projection shows potential, increasing number and precision in low density clouds, but has to be more robust. Further research needs to be carried out, focusing on identifying a fast and robust keypoint detector.

A Deep-Based Approach for Multi-Descriptor Feature Extraction: Applications on SAR Image Registration

Synthetic aperture radar (SAR) images have found increasing attention in various applications, such as remote sensing, surveillance, and disaster management. Unlike optical sensors, SAR can capture images in any weather conditions at any time, making it particularly valuable. However, it poses unique challenges for image processing tasks, like complex speckle noise, texture, and geometric distortions. For many image processing tasks such as image registration; feature detection and extraction are critical initial steps. In this work, we propose a deep learning-based method for detecting and describing keypoints in SAR images, which can be specifically applied to SAR image registration tasks. To address the challenges of SAR images, we utilize the R2D2 network and propose a training scheme specifically tailored for these images. Furthermore, our approach involves multiple descriptors utilization for outlier removal between image keypoints and a two-stage pipeline consisting of coarse and fine stages for robust feature point matching of SAR images. By leveraging the coarse registration stage, which maintains robustness, and also a fine stage that employs multiple descriptors along with a local search technique, our approach enhances the keypoint matching accuracy. Evaluation results show that the proposed method outperforms most of the traditional and deep learning-based methods in terms of the percentage of correct matches, while achieving comparable or even better root mean squared error (RMSE) results.

A Robust Mismatch Removal Method for Image Matching Based on the Fusion of the Local Features and the Depth

Feature point matching is a fundamental task in computer vision such as vision simultaneous localization and mapping (VSLAM) and structure from motion (SFM). Due to the similarity or interference of features, mismatches are often unavoidable. Therefore, how to eliminate mismatches is important for robust matching. Smoothness constraint is widely used to remove mismatch, but it cannot effectively deal with the issue in the rapidly changing scene. In this paper, a novel LCS-SSM (Local Cell Statistics and Structural Similarity Measurement) mismatch removal method is proposed. LCS-SSM integrates the motion consistency and structural similarity of a local image block as the statistical likelihood of matched key points. Then, the Random Sampling Consensus (RANSAC) algorithm is employed to preserve the isolated matches that do not satisfy the statistical likelihood. Experimental and comparative results on the public dataset show that the proposed LCS-SSM can effectively and reliably differentiate true and false matches compared with state-of-the-art methods, and can be used for robust matching in scenes with fast motion, blurs, and clustered noise.

Visual place recognition for autonomous mobile robot navigation using LoFTR and MAGSAC++

Autonomous mobile robots are defined as robotic entities capable of independent movement and intelligent decision-making, relying on their ability to perceive and analyze their surroundings, including objects in their environment. In Simultaneous Localization and Mapping (SLAM) systems, loop closure is often achieved through visual place recognition techniques, where the system compares the current visual input with previously observed scenes to identify matches. In computer vision applications, Speeded-Up Robust Features (SURF) and Scale-Invariant Feature Transform (SIFT) are popular feature extraction algorithms used for such as key point detection, matching, and image registration tasks. The choice of inlier threshold should be based on the specific characteristics of the application and the nature of the images being processed. It often requires experimentation and tuning to find the optimal balance between robustness and accuracy. It Utilizes the pre-trained Local Feature Transformer (LoFTR) and MAGSAC++ estimator to address these drawbacks by employing the number of inliers to determine the similarity between two images for visual place recognition. Our experiment demonstrates that the number of inliers can determine the similarity of locations between two images. Scale variations and translation in location significantly influence the resulting number of inliers. Comparing images from the same location and from different locations yields varying numbers of inliers. The number of inliers significantly influences the similarity of locations. At the same location, the number of inliers is above 150, while at different locations, the number is below 150.

Discrete Cycle-Consistency Based Unsupervised Deep Graph Matching

We contribute to the sparsely populated area of unsupervised deep graph matching with application to keypoint matching in images. Contrary to the standard supervised approach, our method does not require ground truth correspondences between keypoint pairs. Instead, it is self-supervised by enforcing consistency of matchings between images of the same object category. As the matching and the consistency loss are discrete, their derivatives cannot be straightforwardly used for learning. We address this issue in a principled way by building our method upon the recent results on black-box differentiation of combinatorial solvers. This makes our method exceptionally flexible, as it is compatible with arbitrary network architectures and combinatorial solvers. Our experimental evaluation suggests that our technique sets a new state-of-the-art for unsupervised graph matching.

Restoration of UAV-Based Backlit Images for Geological Mapping of a High-Steep Slope.

Unmanned aerial vehicle (UAV)-based geological mapping is significant for understanding the geological structure in the high-steep slopes, but the images obtained in these areas are inevitably influenced by the backlit effect because of the undulating terrain and the viewpoint change of the camera mounted on the UAV. To handle this concern, a novel backlit image restoration method is proposed that takes the real-world application into account and addresses the color distortion issue existing in backlit images captured in high-steep slope scenes. Specifically, there are two main steps in the proposed method, which consist of the backlit removal and the color and detail enhancement. The backlit removal first eliminates the backlit effect using the Retinex strategy, and then the color and detail enhancement step improves the image color and sharpness. The author designs extensive comparison experiments from multiple angles and applies the proposed method to different engineering applications. The experimental results show that the proposed method has potential compared to other main-stream methods both in qualitative visual effects and universal quantitative evaluation metrics. The backlit images processed by the proposed method are significantly improved by the process of feature key point matching, which is very conducive to the fine construction of 3D geological models of the high-steep slope.

The Psychological Construction of Early Childhood Art Education

The psychological construction of early childhood art education refers to the process of promoting children's mental health, emotional development, and personality shaping through art education activities. Art Education provides a platform for early childhood to express themselves, achieve self-realization, unleash their wisdom, develop own aesthetic standards, learn cooperate and communication skills, achieve psychological balance and establish understanding and respect. This study highlights the strategic shift in the approach to art education in China, with a focus on interdisciplinary education and the integration of art into extracurricular activities. With the issue on knowledge is often imparted in a one-way manner and children is encouraged to passively imitate, does not align with the developmental needs of young children or the goals of comprehensive education, this paper researches the dynamic process that involves teachers, students, and sometimes parents and community members in a cycle of planning, acting, observing, and reflecting to improve educational practices, and outlines the Mission and Vision on Development goals and the level of emphasis placed on families and kindergartens, Professional advantages of promoting mental hygiene through art education and Matching of key points, principles, and strategies.

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

CatTrack: Single-Stage Category-Level 6D Object Pose Tracking via Convolution and Vision Transformer

In the current research, many researchers have focused on instance-level pose tracking, which requires a 3D model of the object in advance, making it challenging to apply in practice. To address this limitation, some researchers have proposed the category-level object pose tracking method. Achieving accurate and speedy monocular category-level pose tracking is an essential research goal. In this paper, we propose CatTrack, a new single-stage keypoints-based monocular category-level multi-object pose tracking network. A significant issue in object pose tracking tasks is utilizing the information from the previous frame to guide pose estimation for the next frame. However, as the object poses and camera information in each frame are different, we need to remove irrelevant information and emphasize useful features. To this end, we propose a transformer-based temporal information capture module to leverage the position information of keypoints from the previous frame. Furthermore, we propose a new keypoint matching module to enable the grouping and matching of object keypoints in complex scenes. We have successfully applied CatTrack to the Objectron dataset and achieved superior results in comparison to existing methods. Furthermore, we have also evaluated the generalization of CatTrack and successfully applied it to track the 6D pose of unseen real-world objects. A video is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/Yminjdtsgwk</uri> .

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses.

Neural Radiance Fields (NeRFs) have shown impressive capabilities in synthesizing photorealistic novel views. However, their application to room-size scenes is limited by the requirement of several hundred views with accurate poses for training. To address this challenge, we propose SN 2 eRF, a framework which can reconstruct the neural radiance field with significantly fewer views and noisy poses by exploiting multiple priors. Our key insight is to leverage both multi-view and monocular priors to constrain the optimization of NeRF in the setting of sparse and noisy pose inputs. Specifically, we extract and match key points to constrain pose optimization and use Ray Transformer with a monocular depth estimator to provide dense depth prior for geometry optimization. Benefiting from these priors, our approach achieves state-of-the-art accuracy in novel view synthesis for indoor room scenarios.

Prediction of Clean Coal Ash Content in Coal Flotation through a Convergent Model Unifying Deep Learning and Likelihood Function, Incorporating Froth Velocity and Reagent Dosage Parameters

This study successfully achieved high-precision detection of the clean coal ash content in the coal froth flotation domain by integrating deep learning with the likelihood function. Methodologically, a novel data processing and prediction framework was established by combining a deep learning Keras neural network with the likelihood function from probability statistics. The SIFT algorithm was utilized to extract key feature points and descriptors from the images, and keypoint matching and mean-shift clustering algorithms were employed to accurately obtain information on foam motion trajectories and velocities. For parameter optimization, the maximum likelihood estimation was applied to find the optimal parameter estimates of the likelihood function, ensuring enhanced model accuracy. By incorporating the optimized likelihood function parameters into the Keras deep neural network, an efficient prediction model was constructed for the dosage of flotation reagents, froth velocity, and clean coal ash content. The model’s evaluation involved six performance metrics. The experimental results were highly significant, with R2 at 0.99997%, RMSE at 0.04458%, MAE at 0.00170%, MAPE at 0.02329%, RRSE at 0.00994%, and MAAPE at 0.00067%.

Robust coverless video steganography based on inter-frame keypoint matching

Existing coverless video steganography schemes primarily focus on single-frame features, which manifest poor robustness. To enhance resistance against adversarial attacks, this paper proposes a coverless video steganography method based on inter-frame keypoint matching, leveraging the movement orientation of keypoints within the video for information hiding. Initially, the method divides video frames into blocks, and then constructs an inter-frame keypoint matching network by using the keypoint detection method combined with deep learning. By narrowing down the matching scope and filtering out mismatches, the network effectively discerns the keypoint matching relationships. For keypoints successfully matched within the blocks, a coordinate system is established based on the original keypoint, determining the position of the matched keypoints. The most frequently occurring orientation within the blocks is designated as the orientation of the block. Subsequently, we introduce a mapping rule based on quadrant labels, generating robust hash sequences according to the quadrant labels corresponding to the orientation of the blocks. Compared to the latest coverless video steganography schemes, this method shows superior robustness in resisting 14 common traditional attacks and video attacks. Concurrently, it exhibits commendable hiding capacity and hiding success rate.

A fully‐automatic side‐scan sonar simultaneous localization and mapping framework

AbstractSide‐scan sonar is a lightweight acoustic sensor that is frequently deployed on autonomous underwater vehicles (AUVs) to provide high‐resolution seafloor images. However, using side‐scan images to perform simultaneous localization and mapping (SLAM) remains a challenge when there is a lack of 3D bathymetric information and discriminant features in the side‐scan images. To tackle this, the authors propose a feature‐based SLAM framework using side‐scan sonar, which is able to automatically detect and robustly match keypoints between paired side‐scan images. The authors then use the detected correspondences as constraints to optimise the AUV pose trajectory. The proposed method is evaluated on real data collected by a Hugin AUV, using as a ground truth reference both manually‐annotated keypoints and a 3D bathymetry mesh from multibeam echosounder (MBES). Experimental results demonstrate that this approach is able to reduce drifts from the dead‐reckoning system. The framework is made publicly available for the benefit of the community.