Local Feature Descriptors Research Articles

DarkMatcher: An efficient self-supervised network for image matching in extremely dark environments

Abstract Low-light visual perception problems, such as night Simultaneous Localization and Mapping (SLAM) or Structure-from-Motion (SfM) have attracted increasing attention, and the performance of keypoint detection and local feature description play a crucial role. Many traditional algorithms and machine learning methods have been widely used to detect and describe local features. However, the performance of the existing techniques in the face of highly low-light scenes will be drastically degraded, resulting in subsequent practical application needs can not be met. Therefore, an efficient self-supervised deep learning model DarkMatcher is proposed, which can directly detect and describe features from images in extremely dark environments in an end-to-end way. This model consists of a backbone based on new dynamic deformable convolutional blocks and a novel DarkMatcher module that combines multiple attention mechanisms to realize cross-scale feature information interaction. The former enhances the feature extraction capability of the model for low-light environment images. The latter effectively strengthens the matching ability of extremely dark environments and weak texture areas, and further improves the feature matching accuracy in low light scenes. In addition, transfer learning and real-time training strategies are used to enhance the generalization and feature representation capabilities of the model. Many experimental results indicate that DarkMatcher possesses the best matching performance and robustness for feature points in extremely dark environment, with an average matching accuracy of 71.24% and an average execution time of 51ms for each pair of images. Besides, the visual pose estimation experiment has also obtained good results as validation.

Establishment and extension of a compact and robust binary feature descriptor for UAV image matching

Abstract Local feature description is a crucial challenge in unmanned aerial vehicle (UAV) images matching. Although scholars have explored a variety of methods for defining local feature descriptions, improving the matching accuracy of UAV images and reducing the memory consumption of descriptor remain worthwhile research issues. A compact and robust binary feature descriptor for UAV image matching is proposed. To achieve rotational invariance of the descriptor, a sampling pattern based on the region overlap error is employed to assign the dominant direction of the feature points. Additionally, a rotation invariant gradient computation method is proposed to reduce the dominant direction assignment errors. Then, the region around the feature point is divided into grids based on the region overlap error, and gradient orientation histogram and average intensity histogram are constructed. Finally, these histograms are binarized to generate a 30-byte binary descriptor. Results from the Oxford dataset, the Iguazu dataset, and a high-resolution UAV image dataset indicate that the proposed descriptor is compact and achieves the highest matching accuracy compared to nine commonly used feature descriptors, including scale-invariant feature transform, speeded up robust feature, binary robust invariant scalable keypoints, KAZE, oriented fast and rotated brief, fast retina keypoint, local intensity order pattern, boosted efficient binary local image descriptor, and triplet-based efficient binary local image descriptor. Notably, on the high-resolution UAV images with translational transformations, scale variations, salt-and-pepper noise, and Gaussian white noise, the accuracies of the proposed descriptor were 87.09%, 80.83%, 92.46%, and 87.42%, respectively. The improvements ranged from 6.95% to 53.20%, 9.32% to 55.90%, 10.39% to 44.29%, and 6.54% to 58.24%.

Corrupted Point Cloud Classification Through Deep Learning with Local Feature Descriptor.

Three-dimensional point cloud recognition is a very fundamental work in fields such as autonomous driving and face recognition. However, in real industrial scenarios, input point cloud data are often accompanied by factors such as occlusion, rotation, and noise. These factors make it challenging to apply existing point cloud classification algorithms in real industrial scenarios. Currently, most studies enhance model robustness from the perspective of neural network structure. However, researchers have found that simply adjusting the neural network structure has proven insufficient in addressing the decline in accuracy caused by data corruption. In this article, we use local feature descriptors as a preprocessing method to extract features from point cloud data and propose a new neural network architecture aligned with these local features, effectively enhancing performance even in extreme cases of data corruption. In addition, we conducted data augmentation to the 10 intentionally selected categories in ModelNet40. Finally, we conducted multiple experiments, including testing the robustness of the model to occlusion and coordinate transformation and then comparing the model with existing SOTA models. Furthermore, in actual scene experiments, we used depth cameras to capture objects and input the obtained data into the established model. The experimental results show that our model outperforms existing popular algorithms when dealing with corrupted point cloud data. Even when the input point cloud data are affected by occlusion or coordinate transformation, our proposed model can maintain high accuracy. This suggests that our method can alleviate the problem of decreased model accuracy caused by the aforementioned factors.

A wavelet and local binary pattern-based feature descriptor for the detection of chronic infection through thoracic X-ray images.

This investigation attempts to propose a novel Wavelet and Local Binary Pattern-based Xception feature Descriptor (WLBPXD) framework, which uses a deep-learning model for classifying chronic infection amongst other infections. Chronic infection (COVID-19 in this study) is identified via RT-PCR test, which is time-consuming and requires a dedicated laboratory (materials, equipment, etc.) to complete the clinical results. X-rays and computed tomography images from chest scans offer an alternative method for identifying chronic infections. It has been demonstrated that chronic infection can be diagnosed from X-ray images acquired in a real-world setting. The images are transformed using the discrete wavelet transform (DWT), combined with the local binary pattern (LBP) technique. Pre-trained deep-learning models, such as AlexNet, Xception, VGG-16 and Inception Resnet50, extract the features. Subsequently, the extracted features are fused using feature-fusion approaches and subjected to classification. The AlexNet, in conjunction with the DWT model, produced 99.7% accurate results, whereas the AlexNet and the LBP model produced 99.6% accurate results. Therefore, the proposed method is efficient as it offers a better detection accuracy and eventually enhances the scope of early detection, thus assisting the clinical perspectives.

Bilinear Distance Feature Network for Semantic Segmentation in PowerLine Corridor Point Clouds.

Semantic segmentation of target objects in power transmission line corridor point cloud scenes is a crucial step in powerline tree barrier detection. The massive quantity, disordered distribution, and non-uniformity of point clouds in power transmission line corridor scenes pose significant challenges for feature extraction. Previous studies have often overlooked the core utilization of spatial information, limiting the network's ability to understand complex geometric shapes. To overcome this limitation, this paper focuses on enhancing the deep expression of spatial geometric information in segmentation networks and proposes a method called BDF-Net to improve RandLA-Net. For each input 3D point cloud data, BDF-Net first encodes the relative coordinates and relative distance information into spatial geometric feature representations through the Spatial Information Encoding block to capture the local spatial structure of the point cloud data. Subsequently, the Bilinear Pooling block effectively combines the feature information of the point cloud with the spatial geometric representation by leveraging its bilinear interaction capability thus learning more discriminative local feature descriptors. The Global Feature Extraction block captures the global structure information in the point cloud data by using the ratio between the point position and the relative position, so as to enhance the semantic understanding ability of the network. In order to verify the performance of BDF-Net, this paper constructs a dataset, PPCD, for the point cloud scenario of transmission line corridors and conducts detailed experiments on it. The experimental results show that BDF-Net achieves significant performance improvements in various evaluation metrics, specifically achieving an OA of 97.16%, a mIoU of 77.48%, and a mAcc of 87.6%, which are 3.03%, 16.23%, and 18.44% higher than RandLA-Net, respectively. Moreover, comparisons with other state-of-the-art methods also verify the superiority of BDF-Net in point cloud semantic segmentation tasks.

2D3D-DescNet: Jointly Learning 2D and 3D Local Feature Descriptors for Cross-Dimensional Matching

The cross-dimensional matching of 2D images and 3D point clouds is an effective method by which to establish the spatial relationship between 2D and 3D space, which has potential applications in remote sensing and artificial intelligence (AI). In this paper, we propose a novel multi-task network, 2D3D-DescNet, to learn 2D and 3D local feature descriptors jointly and perform cross-dimensional matching of 2D image patches and 3D point cloud volumes. The 2D3D-DescNet contains two branches with which to learn 2D and 3D feature descriptors, respectively, and utilizes a shared decoder to generate the feature maps of 2D image patches and 3D point cloud volumes. Specifically, the generative adversarial network (GAN) strategy is embedded to distinguish the source of the generated feature maps, thereby facilitating the use of the learned 2D and 3D local feature descriptors for cross-dimensional retrieval. Meanwhile, a metric network is embedded to compute the similarity between the learned 2D and 3D local feature descriptors. Finally, we construct a 2D-3D consistent loss function to optimize the 2D3D-DescNet. In this paper, the cross-dimensional matching of 2D images and 3D point clouds is explored with the small object of the 3Dmatch dataset. Experimental results demonstrate that the 2D and 3D local feature descriptors jointly learned by 2D3D-DescNet are similar. In addition, in terms of 2D and 3D cross-dimensional retrieval and matching between 2D image patches and 3D point cloud volumes, the proposed 2D3D-DescNet significantly outperforms the current state-of-the-art approaches based on jointly learning 2D and 3D feature descriptors; the cross-dimensional retrieval at TOP1 on the 3DMatch dataset is improved by over 12%.

A 3D local feature description algorithm based on point distribution

It is well known that the local feature descriptor is playing an important role in 3D image recognition. But the current local feature descriptors have poor ability to describe target features in situations such as noise interference, be obscured, and changes in data resolution. On the basis of point distribution, we propose a new 3D local feature descriptor, the Point Distribution Signature Histogram(PDSH), which is relatively simple and easy to implement and has a fast identification speed by simplifying Local Reference Frame(LRF). We also rotate the local surface of the point cloud from multiple angles and project them onto three coordinate planes of the LRF. The projection planes are segmented at the same angle, and the distance information of the edge points and the count of projected points in each segment are statistically accumulated. Finally, the sub-feature vectors of the three coordinate planes are concatenated to obtain the feature descriptor of the key points. Experimental results show that our PDSH descriptor achieves a descriptiveness of 98 % on the pollution-free point cloud data set and better performance than the classical descriptors.

Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (e.g., modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors (such as histograms of oriented gradients) benefits the ViT on IR modality but not RGB or Depth modalities. Second, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. Finally, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document}A2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document}E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document}A2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$^{2}$$\\end{document}E is able to learn more intrinsic task-aware representation and compatible with modality-agnostic (e.g., unimodal, bimodal, and trimodal) downstream settings. Extensive experiments with both unimodal (RGB, Depth, IR) and multimodal (RGB+Depth, RGB+IR, Depth+IR, RGB+Depth+IR) settings conducted on multimodal FAS benchmarks demonstrate the superior performance of the proposed methods. One highlight is that the proposed method is robust under various missing-modality cases where previous multimodal FAS models suffer serious performance drops. We hope these findings and solutions can facilitate the future research for ViT-based multimodal FAS.

Towards stronger illumination robustness of local feature detection and description based on auxiliary learning

Towards stronger illumination robustness of local feature detection and description based on auxiliary learning

Graph-Based Contrastive Learning for Description and Detection of Local Features.

Confronted with the task environment full of repetitive textures, the state-of-art description and detection methods for local features greatly suffer from the "pseudo-negatives," bringing inconsistent optimization objectives during training. To address this problem, this article develops a self-supervised graph-based contrastive learning framework to train the model for local features, GCLFeat. The proposed approach learns to alleviate the pseudo-negatives specifically from three aspects: 1) designing a graph neural network (GNN), which focuses on mining the local transformational invariance across different views and global textual knowledge within individual images; 2) generating the dense correspondence annotations from a diverse natural dataset with a self-supervised paradigm; and 3) adopting a keypoints-aware sampling strategy to compute the loss across the whole dataset. The experimental results show that the unsupervised framework outperforms the state-of-the-art supervised baselines on diverse downstream benchmarks including image matching, 3-D reconstruction and visual localization. The code will be made public and available at https://github.com/RealZihaoWang/GCLFeat.

Advanced Planar Projection Contour (PPC): A Novel Algorithm for Local Feature Description in Point Clouds.

Local feature description of point clouds is essential in 3D computer vision. However, many local feature descriptors for point clouds struggle with inadequate robustness, excessive dimensionality, and poor computational efficiency. To address these issues, we propose a novel descriptor based on Planar Projection Contours, characterized by convex packet contour information. We construct the Local Reference Frame (LRF) through covariance analysis of the query point and its neighboring points. Neighboring points are projected onto three orthogonal planes defined by the LRF. These projection points on the planes are fitted into convex hull contours and encoded as local features. These planar features are then concatenated to create the Planar Projection Contour (PPC) descriptor. We evaluated the performance of the PPC descriptor against classical descriptors using the B3R, UWAOR, and Kinect datasets. Experimental results demonstrate that the PPC descriptor achieves an accuracy exceeding 80% across all recall levels, even under high-noise and point density variation conditions, underscoring its effectiveness and robustness.

Accurate global and local 3D alignment of cryo-EM density maps using local spatial structural features.

Advances in cryo-electron microscopy (cryo-EM) imaging technologies have led to a rapidly increasing number of cryo-EM density maps. Alignment and comparison of density maps play a crucial role in interpreting structural information, such as conformational heterogeneity analysis using global alignment and atomic model assembly through local alignment. Here, we present a fast and accurate global and local cryo-EM density map alignment method called CryoAlign, that leverages local density feature descriptors to capture spatial structure similarities. CryoAlign is a feature-based cryo-EM map alignment tool, in which the employment of feature-based architecture enables the rapid establishment of point pair correspondences and robust estimation of alignment parameters. Extensive experimental evaluations demonstrate the superiority of CryoAlign over the existing methods in terms of both alignment accuracy and speed.

FApSH: An effective and robust local feature descriptor for 3D registration and object recognition

Three-dimensional (3D) local feature descriptor plays an important role in 3D computer vision because it is widely used to build point-to-point correspondences in many 3D vision applications. However, existing descriptors are difficult to have both high descriptiveness and strong robustness to various nuisances (e.g., noise and occlusion). To address this problem, a descriptor named Fully Attribute-pairs Statistical Histogram (FApSH) is proposed. FApSH is constructed on a local reference axis (LRA), and fully encodes the relevancy information of five attributes at each neighbor point by ten attribute-pair statistics. In this process, a priori distribution-based partition strategy is proposed for evenly distributing the attribute values of all neighbor points, and a radial distance-based histogram assignment method is proposed to improve the robustness to noise and outliers. The proposed methods are rigorously evaluated on six benchmark datasets with different application scenarios and nuisances. The results show that FApSH has high descriptiveness and strong robustness. It obviously outperforms the existing handcrafted descriptors, and is comparable to some superior learning-based descriptors. The results also show that the proposed priori distribution-based partition strategy significantly reduces the length and also improves the descriptiveness of FApSH, and the proposed radial distance-based histogram assignment method improves the robustness of FApSH on various datasets.

LSNFS: A Local Feature Descriptor Algorithm with High Discrimination and Strong Robustness

LSNFS: A Local Feature Descriptor Algorithm with High Discrimination and Strong Robustness

An Orientation-Robust Local Feature Descriptor Based on Texture and Phase Congruency for Visible–Infrared Image Matching

An Orientation-Robust Local Feature Descriptor Based on Texture and Phase Congruency for Visible–Infrared Image Matching

Point cloud registration algorithm based on statistical local feature description and matching

Point cloud registration algorithm based on statistical local feature description and matching

Image feature extraction algorithm based on visual information

Abstract Vision is the main sensory organ for human beings to contact and understand the objective world. The results of various statistical data show that more than 60% of all ways for human beings to obtain external information are through the visual system. Vision is of great significance for human beings to obtain all kinds of information needed for survival, which is the most important sense of human beings. The rapid growth of computer technology, image processing, pattern recognition, and other disciplines have been widely applied. Traditional image processing algorithms have some limitations when dealing with complex images. To solve these problems, some scholars have proposed various new methods. Most of these methods are based on statistical models or artificial neural networks. Although they meet the requirements of modern computer vision systems for feature extraction algorithms with high accuracy, high speed, and low complexity, these algorithms still have many shortcomings. For example, many researchers have used different methods for feature extraction and segmentation to get better segmentation results. Scale-invariant feature transform (SIFT) is a description used in the field of image processing. This description has scale invariance and can detect key points in the image. It is a local feature descriptor. A sparse coding algorithm is an unsupervised learning method, which is used to find a set of “super complete” basis vectors to represent sample data more efficiently. Therefore, combining SIFT and sparse coding, this article proposed an image feature extraction algorithm based on visual information to extract image features. The results showed that the feature extraction time of X algorithm for different targets was within 0.5 s when the other conditions were the same. The feature matching time was within 1 s, and the correct matching rate was more than 90%. The feature extraction time of Y algorithm for different targets was within 2 s. The feature matching time was within 3 s, and the correct matching rate was between 80 and 90%, indicating that the recognition effect of X algorithm was better than that of Y algorithm. It indicates the positive relationship between visual information and image feature extraction algorithm.

Point cloud semantic segmentation based on local feature fusion and multilayer attention network

AbstractSemantic segmentation from a three‐dimensional point cloud is vital in autonomous driving, computer vision, and augmented reality. However, current semantic segmentation does not effectively use the point cloud's local geometric features and contextual information, essential for improving segmentation accuracy. A semantic segmentation network that uses local feature fusion and a multilayer attention mechanism is proposed to address these challenges. Specifically, the authors designed a local feature fusion module to encode the geometric and feature information separately, which fully leverages the point cloud's feature perception and geometric structure representation. Furthermore, the authors designed a multilayer attention pooling module consisting of local attention pooling and cascade attention pooling to extract contextual information. Local attention pooling is used to learn local neighbourhood information, and cascade attention pooling captures contextual information from deeper local neighbourhoods. Finally, an enhanced feature representation of important information is obtained by aggregating the features from the two deep attention pooling methods. Extensive experiments on large‐scale point‐cloud datasets Stanford 3D large‐scale indoor spaces and SemanticKITTI indicate that authors network shows excellent advantages over existing representative methods regarding local geometric feature description and global contextual relationships.

A point cloud registration method based on multiple-local-feature matching

A dual-view point cloud registration method has been proposed to enhance the accuracy of point-cloud registration by utilizing multiple local feature descriptors. The method focuses on studying the eigenvalues of the point cloud set at various scales and the impact of descriptors that consist of corresponding curvature radii in reducing misregistration in fast point feature histogram (FPFH). The optimal scale size and number were determined through experimental analysis. SIFT-3D key points of point clouds were extracted to calculate their FPFH and complete initial correspondence matching. Next, a multi-neighborhood local feature descriptor was designed to filter correspondence. A large number of incorrect corresponding point pairs were removed after the initial FPFH matching, and the remaining correct point pairs were used for rough registration. Finally, the improved ICP algorithm was used to reduce registration errors and achieve more accurate dual-view registration. According to the experiments of the public point cloud data set, the dual-view point cloud registration algorithm implemented by the method has good accuracy, high robustness, and more reliable point cloud registration results, which provides a theoretical basis for 3D point-cloud reconstruction.

Novel 3D local feature descriptor of point clouds based on spatial voxel homogenization for feature matching

Obtaining a 3D feature description with high descriptiveness and robustness under complicated nuisances is a significant and challenging task in 3D feature matching. This paper proposes a novel feature description consisting of a stable local reference frame (LRF) and a feature descriptor based on local spatial voxels. First, an improved LRF was designed by incorporating distance weights into Z- and X-axis calculations. Subsequently, based on the LRF and voxel segmentation, a feature descriptor based on voxel homogenization was proposed. Moreover, uniform segmentation of cube voxels was performed, considering the eigenvalues of each voxel and its neighboring voxels, thereby enhancing the stability of the description. The performance of the descriptor was strictly tested and evaluated on three public datasets, which exhibited high descriptiveness, robustness, and superior performance compared with other current methods. Furthermore, the descriptor was applied to a 3D registration trial, and the results demonstrated the reliability of our approach.