Keypoint Descriptors Research Articles

Rapid SLAM Method for Star Surface Rover in Unstructured Space Environments

The space environment is characterized by unstructured features, sparsity, and poor lighting conditions. The difficulty in extracting features from the visual frontend of traditional SLAM methods results in poor localization and time-consuming issues. This paper proposes a rapid and real-time localization and mapping method for star chart surveyors in unstructured space environments. Improved localization is achieved using multiple sensor fusion to sense the space environment. We replaced the traditional feature extraction module with an enhanced SuperPoint feature extraction network to tackle the challenge of challenging feature extraction in unstructured space environments. By dynamically adjusting detection thresholds, we achieved uniform detection and description of image keypoints, ultimately resulting in robust and accurate feature association information. Furthermore, we minimized redundant information to achieve precise positioning with high efficiency and low power consumption. We established a star surface rover simulation system and created simulated environments resembling Mars and the lunar surface. Compared to the LVI-SAM system, our method achieved a 20% improvement in localization accuracy for lunar scenarios. In Mars scenarios, our method achieved a positioning accuracy of 0.716 m and reduced runtime by 18.682 s for the same tasks. Our approach exhibits higher localization accuracy and lower power consumption in unstructured space environments.

Key-Point-Descriptor-Based Image Quality Evaluation in Photogrammetry Workflows

Photogrammetry depends critically on the quality of the images used to reconstruct accurate and detailed 3D models. Selection of high-quality images not only improves the accuracy and resolution of the resulting 3D models, but also contributes to the efficiency of the photogrammetric process by reducing data redundancy and computational demands. This study presents a novel approach to image quality evaluation tailored for photogrammetric applications that uses the key point descriptors typically encountered in image matching. Using a LightGBM ranker model, this research evaluates the effectiveness of key point descriptors such as SIFT, SURF, BRISK, ORB, KAZE, FREAK, and SuperPoint in predicting image quality. These descriptors are evaluated for their ability to indicate image quality based on the image patterns they capture. Experiments conducted on various publicly available image datasets show that descriptor-based methods outperform traditional no-reference image quality metrics such as BRISQUE, NIQE, PIQE, and BIQAA and a simple sharpness-based image quality evaluation method. The experimental results highlight the potential of using key-point-descriptor-based image quality evaluation methods to improve the photogrammetric workflow by selecting high-quality images for 3D modeling.

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

ANALYSIS OF IMAGE KEY POINT DETECTION, DESCRIPTION AND MATCHING METHODS

Unsupervised Non-rigid Histological Image Registration Guided by Keypoint Correspondences Based on Learnable Deep Features with Iterative Training.

Histological image registration is a fundamental task in histological image analysis. It is challenging because of substantial appearance differences due to multiple staining. Keypoint correspondences, i.e., matched keypoint pairs, have been introduced to guide unsupervised deep learning (DL) based registration methods to handle such a registration task. This paper proposes an iterative keypoint correspondence-guided (IKCG) unsupervised network for non-rigid histological image registration. Fixed deep features and learnable deep features are introduced as keypoint descriptors to automatically establish keypoint correspondences, the distance between which is used as a loss function to train the registration network. Fixed deep features extracted from DL networks that are pre-trained on natural image datasets are more discriminative than handcrafted ones, benefiting from the deep and hierarchical nature of DL networks. The intermediate layer outputs of the registration networks trained on histological image datasets are extracted as learnable deep features, which reveal unique information for histological images. An iterative training strategy is adopted to train the registration network and optimize learnable deep features jointly. Benefiting from the excellent matching ability of learnable deep features optimized with the iterative training strategy, the proposed method can solve the local non-rigid large displacement problem, an inevitable problem usually caused by misoperation, such as tears in producing tissue slices. The proposed method is evaluated on the Automatic Non-rigid Histology Image Registration (ANHIR) website and AutomatiC Registration Of Breast cAncer Tissue (ACROBAT) website. It ranked 1st on both websites as of August 6th, 2024.

Accelerating Image Classification based on a Model for Estimating Descriptor-to-Class Distance

The article describes a method of image classification based on the estimation of the distance to the etalon class. The implementation of estimates gives a significant gain in classification speed compared to linear search while maintaining a decent level of accuracy. The methodology is based on the use of the triangle inequality for images given by a set of binary vectors as descriptors of the image key points. The evaluation is applied to the "object descriptor – etalon" classification method, which is based on the descriptor voting procedure. An analysis of evaluation options is carried out using the parameters of the etalon sets in the form of a medoid and the closest or farthest points from it. The gain in classification time compared to the traditional method proportionally depends on the number of descriptors in the etalon description. Software simulation of classifiers with the implementation of evaluation shows a gain in speed of 350-450 times for the description of 500 descriptors while maintaining one hundred percent classification accuracy on the training set of similar NFT images. A control sample experiment shows that the classifier with estimation can respond better to image details compared to the traditional method.

Keypoint Detection and Description through Deep Learning in Unstructured Environments

Feature extraction plays a crucial role in computer vision and autonomous navigation, offering valuable information for real-time localization and scene understanding. However, although multiple studies investigate keypoint detection and description algorithms in urban and indoor environments, far fewer studies concentrate in unstructured environments. In this study, a multi-task deep learning architecture is developed for keypoint detection and description, focused on poor-featured unstructured and planetary scenes with low or changing illumination. The proposed architecture was trained and evaluated using a training and benchmark dataset with earthy and planetary scenes. Moreover, the trained model was integrated in a visual SLAM (Simultaneous Localization and Maping) system as a feature extraction module, and tested in two feature-poor unstructured areas. Regarding the results, the proposed architecture provides a mAP (mean Average Precision) in a level of 0.95 in terms of keypoint description, outperforming well-known handcrafted algorithms while the proposed SLAM achieved two times lower RMSE error in a poor-featured area with low illumination, compared with ORB-SLAM2. To the best of the authors’ knowledge, this is the first study that investigates the potential of keypoint detection and description through deep learning in unstructured and planetary environments.

Keypoint-based registration of TLS point clouds using a statistical matching approach

Abstract Laser scanning is a wide-spread practice to capture the environment. Besides the fields of robotics and self-driving cars, it has been applied in the field of engineering geodesy for documentation and monitoring purposes for many years. The registration of scans is still one of the main sources of uncertainty in the final point cloud. This paper presents a new keypoint-based method for terrestrial laser scan (TLS) registration for high-accuracy applications. Based on detected 2D-keypoints, we introduce a new statistical matching approach that tests wheter keypoints, scanned from two scan stations, can be assumed to be identical. This approach avoids the use of keypoint descriptors for matching and also handles wide distances between different scanner stations. The presented approach requires a good coarse registration as initial input, which can be achieved for example by artificial laser scanning targets. By means of two evaluation data sets, we show that our keypoint-based registration leads to the smallest loop closure error when traversing several stations compared to target-based and ICP registrations. Due to the high number of observations compared to the target-based registration, the reliability of the our keypoint-based registration can be increased significantly and the precision of the registration can be increased by about 25 % on average.

A weighted local‐spherical grid based lightweight descriptor for 3D point cloud registration

AbstractEstablishing effective feature descriptors is a crucial step in 3D point cloud registration task. Existing manual‐based methods are noise‐susceptible and time‐consuming when running on low‐cost edge computing devices. To this end, the authors proposed a Local Reference Frame (LRF) based approach that can quickly and robustly register point clouds by using a novel lightweight local‐spherical grid weighted descriptor (LSGWD). Firstly, the LRF of the proposed algorithm is established by the covariance matrix eigenvector of KeyPoint's spherical support set and the centroid vector's projection on its orthogonal plane. Then the spherical support is grided to 32 bins, and the 4D geometric features of each subset are constructed by the centroid moment and the cosine value of the angles between the centroid vector and axes of LRF. Secondly, to restrain the insufficient discriminative information presented in the purely geometric features, the Gaussian projection and gradient mapping are proposed to calculate the smooth density and the correlation of structural characteristics, which are obtained as the distribution information of each bin to weigh the feature representation. Finally, the 32 × 4‐dimensional KeyPoint descriptor is obtained and used in the 3D point cloud registration framework. Experiments are carried out on three test datasets and real scene data. Compared to previous baselines, our descriptor achieves the state‐of‐the‐art performance in terms of efficiency and accuracy owing to its compact structure and noise robustness. The proposed method enhances the recognition and registration performance of 3D point cloud matching in low‐cost edge computing applications.

Joint keypoint detection and description network for color fundus image registration.

Retinal imaging is widely used to diagnose many diseases, both systemic and eye-specific. In these cases, image registration, which is the process of aligning images taken from different viewpoints or moments in time, is fundamental to compare different images and to assess changes in their appearance, commonly caused by disease progression. Currently, the field of color fundus registration is dominated by classical methods, as deep learning alternatives have not shown sufficient improvement over classic methods to justify the added computational cost. However, deep learning registration methods are still considered beneficial as they can be easily adapted to different modalities and devices following a data-driven learning approach. In this work, we propose a novel methodology to register color fundus images using deep learning for the joint detection and description of keypoints. In particular, we use an unsupervised neural network trained to obtain repeatable keypoints and reliable descriptors. These keypoints and descriptors allow to produce an accurate registration using RANdom SAmple Consensus (RANSAC). We train the method using the Messidor dataset and test it with the Fundus Image Registration Dataset (FIRE) dataset, both of which are publicly accessible. Our work demonstrates a color fundus registration method that is robust to changes in imaging devices and capture conditions. Moreover, we conduct multiple experiments exploring several of the method's parameters to assess their impact on the registration performance. The method obtained an overall Registration Score of 0.695 for the whole FIRE dataset (0.925 for category S, 0.352 for P, and 0.726 for A). Our proposal improves the results of previous deep learning methods in every category and surpasses the performance of classical approaches in category A which has disease progression and thus represents the most relevant scenario for clinical practice as registration is commonly used in patients with diseases for disease monitoring purposes.

FRACTIONAL WAVELET TRANSFORM PHASE FOR IRIS IMAGE KEY POINTS MATCHING

Abstract. In this article the fractional phase congruency method for iris image key points descriptors is proposed. The fractional phase congruency is calculated using fractional wavelet transform through the fractional Fourier transform. Fractional Fourier transform is the generalization of the classical Fourier transform. The use of fractional phase congruency can achieve better results compared to the use of the classical phase congruency. The comparison between phase congruency and fractional phase congruency for biometric iris images is given. The optimal parameters of fractional wavelet transform for iris image key points matching are found. The experimental results of the proposed method using the images from CASIA−IrisV4−Interval database are demonstrated.

Investigating keypoint descriptors for camera relocalization in endoscopy surgery.

Recent advances in computer vision and machine learning have resulted in endoscopic video-based solutions for dense reconstruction of the anatomy. To effectively use these systems in surgical navigation, a reliable image-based technique is required to constantly track the endoscopic camera's position within the anatomy, despite frequent removal and re-insertion. In this work, we investigate the use of recent learning-based keypoint descriptors for six degree-of-freedom camera pose estimation in intraoperative endoscopic sequences and under changes in anatomy due to surgical resection. Our method employs a dense structure from motion (SfM) reconstruction of the preoperative anatomy, obtained with a state-of-the-art patient-specific learning-based descriptor. During the reconstruction step, each estimated 3D point is associated with a descriptor. This information is employed in the intraoperative sequences to establish 2D-3D correspondences for Perspective-n-Point (PnP) camera pose estimation. We evaluate this method in six intraoperative sequences that include anatomical modifications obtained from two cadaveric subjects. Show that this approach led to translation and rotation errors of 3.9mm and 0.2radians, respectively, with 21.86% of localized cameras averaged over the six sequences. In comparison to an additional learning-based descriptor (HardNet++), the selected descriptor can achieve a better percentage of localized cameras with similar pose estimation performance. We further discussed potential error causes and limitations of the proposed approach. Patient-specific learning-based descriptors can relocalize images that are well distributed across the inspected anatomy, even where the anatomy is modified. However, camera relocalization in endoscopic sequences remains a persistently challenging problem, and future research is necessary to increase the robustness and accuracy of this technique.

Finding Nemo’s Giant Cousin: Keypoint Matching for Robust Re-Identification of Giant Sunfish

The Giant Sunfish (Mola alexandrini) has unique patterns on its body, which allow for individual identification. By continuously gathering and matching images, it is possible to monitor and track individuals across location and time. However, matching images manually is a tedious and time-consuming task. To automate the process, we propose a pipeline based on finding and matching keypoints between image pairs. We evaluate our pipeline with four different keypoint descriptors, namely ORB, SIFT, RootSIFT, and SuperPoint, and demonstrate that the number of matching keypoints between a pair of images is a strong indicator for the likelihood that they contain the same individual. The best results are obtained with RootSIFT, which achieves an mAP of 75.91% on our test dataset (TinyMola+) without training or fine-tuning any parts of the pipeline. Furthermore, we show that the pipeline generalizes to other domains, such as re-identification of seals and cows. Lastly, we discuss the impracticality of a ranking-based output for real-life tasks and propose an alternative approach by viewing re-identification as a binary classification. We show that the pipeline can be easily modified with minimal fine-tuning to provide a binary output with a precision of 98% and recall of 44% on the TinyMola+ dataset, which basically eliminates the need for time-consuming manual verification on nearly half the dataset.

Merging three‐dimensional occupancy grid maps to support multi‐UAVs cooperative navigation systems

AbstractUnmanned aerial vehicles (UAVs) have been widely used in many applications due to, among other features, their versatility, reduced operation cost, and reduced size. These applications increasingly require that features related to autonomous navigation be employed, such as mapping. However, the reduced capacity of resources such as battery life and hardware (memory and processing capacity) can hinder the development of these applications in UAVs. Thus, the collaborative use of multi‐UAVs for mapping can be used as an alternative to solve this problem, creating a cooperative navigation system. This system requires that individual local maps will be merged into a global map in a distributed way. In this context, this work proposes a system for merging three‐dimensional occupancy grid maps for use in UAVs that compose a cooperative navigation system. The proposed solution consists of a keypoint detector, a keypoint descriptor, and filters for keypoints and correspondences. The keypoint properties, such as orientation, are acquired from potential field gradients. Image processing techniques are applied to remove the map noise and better calculate the transformation parameters. The proposed system is validated by a set of simulation experiments performed in six different environments (indoor and outdoor). First, each component of the proposed system is individually evaluated. Then, the complete pairwise map merging system is evaluated. Finally, the appropriate functioning of the proposed system is validated through a visual evaluation.

Building a High-resolution Digital Terrain Model of Bennu from Laser Altimetry Data

The Origins, Spectral Interpretation, Resource Identification, and Security-Regolith Explorer (OSIRIS-REx) spacecraft orbited the near-Earth asteroid (101955) Bennu to characterize the asteroid prior to sampling. One important aspect of this characterization was the creation of a high-resolution (5–7 cm) global shape model using the OSIRIS-REx Laser Altimeter (OLA). We describe the data collected by OLA, along with the approach used to register overlapping topography using keypoints and keypoint descriptors in order to produce a globally self-consistent set of data. These globally registered sets of topographic scans were used to generate digital terrain models at both global and regional scales. We also describe efforts to correct for a change in behavior of the scanning mirror after the launch and highlight the improvements to the data after implementing an updated calibration of the mirror. The resulting model represents the highest-fidelity global OLA data set.

An Energy-Efficient SIFT Based Feature Extraction Accelerator for High Frame-Rate Video Applications

Visual feature extraction is a key technology of computer vision for intelligent video processing. Efficient feature extraction is a fundamental problem in computer vision applications. Scale-Invariant Feature Transform (SIFT) is one of the most popular feature extraction algorithms because SIFT features are invariant to image scale and rotation and robust to changes in illumination and noise. However, SIFT is a computationally-intensive and power-hungry algorithm, which needs to be accelerated by efficient hardware design to achieve both high-speed feature extraction and high energy efficiency for many high frame-rate video applications at Artificial-intelligent Internet of Things edges. In this work, an energy-efficient SIFT based feature extraction accelerator is proposed. In the Gaussian pyramid and Differences of Gaussian (DoG) pyramid construction process, three design methods are proposed to reduce power consumption and improve information fidelity: a fast and slow dual clock domain design method with a reconfigurable design strategy is proposed to reduce the computation resources; a partial sum reuse design method is proposed to further reduce the computation resources and the amount of computation; a dynamic padding design method is proposed to solve the problem of information loss at image edges and corners after convolution operation. In the keypoint descriptor generation process, an optimized algorithm using circular region and polar coordinates is proposed to parallelize the main orientation assignment and descriptor generation to achieve high-speed processing, while maintaining a comparable matching accuracy with the state-of-the-art designs. The experiment results show that the proposed SIFT hardware accelerator is able to extract features by up to 162 frames per second ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480$ </tex-math></inline-formula> pixels) under 100 MHz, with the power consumption of 364.26 mW and energy efficiency of 2.25 mJ/frame based on 180 nm technology, which is suitable for many high frame-rate AIoT applications including autonomous driving cars and unmanned aerial vehicles.

An Improved ASIFT Image Feature Matching Algorithm Based on POS Information.

The affine scale-invariant feature transform (ASIFT) algorithm is a feature extraction algorithm with affinity and scale invariance, which is suitable for image feature matching using unmanned aerial vehicles (UAVs). However, there are many problems in the matching process, such as the low efficiency and mismatching. In order to improve the matching efficiency, this algorithm firstly simulates image distortion based on the position and orientation system (POS) information from real-time UAV measurements to reduce the number of simulated images. Then, the scale-invariant feature transform (SIFT) algorithm is used for feature point detection, and the extracted feature points are combined with the binary robust invariant scalable keypoints (BRISK) descriptor to generate the binary feature descriptor, which is matched using the Hamming distance. Finally, in order to improve the matching accuracy of the UAV images, based on the random sample consensus (RANSAC) a false matching eliminated algorithm is proposed. Through four groups of experiments, the proposed algorithm is compared with the SIFT and ASIFT. The results show that the algorithm can optimize the matching effect and improve the matching speed.

Robust Feature Matching for 3D Point Clouds with Progressive Consistency Voting.

Feature matching for 3D point clouds is a fundamental yet challenging problem in remote sensing and 3D computer vision. However, due to a number of nuisances, the initial feature correspondences generated by matching local keypoint descriptors may contain many outliers (incorrect correspondences). To remove outliers, this paper presents a robust method called progressive consistency voting (PCV). PCV aims at assigning a reliable confidence score to each correspondence such that reasonable correspondences can be achieved by simply finding top-scored ones. To compute the confidence score, we suggest fully utilizing the geometric consistency cue between correspondences and propose a voting-based scheme. In addition, we progressively mine convincing voters from the initial correspondence set and optimize the scoring result by considering top-scored correspondences at the last iteration. Experiments on several standard datasets verify that PCV outperforms five state-of-the-art methods under almost all tested conditions and is robust to noise, data decimation, clutter, occlusion, and data modality change. We also apply PCV to point cloud registration and show that it can significantly improve the registration performance.

Efficient Shot Boundary Detection with Multiple Visual Representations

Due to the unlimited growth of video-capturing devices and media, searching and finding a particular video in this huge database becomes a laborious as well as expensive task. Information-rich shots are the inevitable factor of the content-based video processing (CBVP) system. Hence, shot boundary detection (SBD) becomes the basic step of all content-based video retrieval processes. The accuracy of the existing SBD methods highly suffers from false positives and false negatives due to the presence of multiple variants. An efficient SBD method with multiple invariant features is proposed in this paper. A right combination of invariant features such as edge change ratio (ECR), colour layout descriptor (CLD), and scale-invariant feature transform (SIFT) key point descriptors helped to improve the accuracy level of SBD. As the selected features are invariant to most of the variants in video frames, such as illuminance changes, motion, scaling, and rotation, a markable reduction in false detection is possible. Support vector machine (SVM) classifier is used for the classification of frames into transition frames and shot frames. This proposed method is experimented and analysed with the standard SBD dataset TRECVid 2007 videos. The experimental results are compared with some state-of-art methods, and our method shows better performance with a 97% of F1 score.

Learning isometry-invariant representations for point cloud analysis

3D shape analysis has drawn broad attention due to its increasing demands in various fields. Despite that impressive performance has been achieved on several databases, most researchers focus their efforts on improving the performance of shape classification, retrieval, segmentation, etc. They neglect the fact that the disturbances, such as orientation and deformation, may impact much on the perception, restricting the capacity of generalizing to real applications where the prior of orientation and pose is often unknown. In this paper, we conduct shape analysis on point clouds and propose the point projection feature, which is rotation-invariant. Specifically, a novel architecture is designed to mine features of different levels. We adopt a PointNet-based backbone to extract global feature for the point cloud, and the graph aggregation operation to perceive local pose variance in the Euclidean space or geodesic space. An efficient key point descriptor is designed to assign each point with different response and help recognize the overall geometry. Furthermore, a novel dataset, PKUnon-rigid, is built that is composed of non-rigid 3D objects, based on which we evaluate the capacity of several mainstream methods in terms of processing non-rigid shapes. Mathematical analyses and experimental results demonstrate that the proposed method can extract isometry-invariant representations for 3D shape analysis tasks without rotation augmentation, and outperforms other state-of-the-art methods. The proposed dataset is publicly available at https://github.com/tasx0823/PKUnon-rigid.

Robust Fingerprint Minutiae Extraction and Matching Based on Improved SIFT Features

Minutiae feature extraction and matching are not only two crucial tasks for identifying fingerprints, but also play an eminent role as core components of automated fingerprint recognition (AFR) systems, which first focus primarily on the identification and description of the salient minutiae points that impart individuality to each fingerprint and differentiate one fingerprint from another, and then matching their relative placement in a candidate fingerprint and previously stored fingerprint templates. In this paper, an automated minutiae extraction and matching framework is presented for identification and verification purposes, in which an adaptive scale-invariant feature transform (SIFT) detector is applied to high-contrast fingerprints preprocessed by means of denoising, binarization, thinning, dilation and enhancement to improve the quality of latent fingerprints. As a result, an optimized set of highly-reliable salient points discriminating fingerprint minutiae is identified and described accurately and quickly. Then, the SIFT descriptors of the local key-points in a given fingerprint are matched with those of the stored templates using a brute force algorithm, by assigning a score for each match based on the Euclidean distance between the SIFT descriptors of the two matched keypoints. Finally, a postprocessing dual-threshold filter is adaptively applied, which can potentially eliminate almost all the false matches, while discarding very few correct matches (less than 4%). The experimental evaluations on publicly available low-quality FVC2004 fingerprint datasets demonstrate that the proposed framework delivers comparable or superior performance to several state-of-the-art methods, achieving an average equal error rate (EER) value of 2.01%.