3D Feature Descriptors Research Articles

2D3D-DescNet: Jointly Learning 2D and 3D Local Feature Descriptors for Cross-Dimensional Matching

The cross-dimensional matching of 2D images and 3D point clouds is an effective method by which to establish the spatial relationship between 2D and 3D space, which has potential applications in remote sensing and artificial intelligence (AI). In this paper, we propose a novel multi-task network, 2D3D-DescNet, to learn 2D and 3D local feature descriptors jointly and perform cross-dimensional matching of 2D image patches and 3D point cloud volumes. The 2D3D-DescNet contains two branches with which to learn 2D and 3D feature descriptors, respectively, and utilizes a shared decoder to generate the feature maps of 2D image patches and 3D point cloud volumes. Specifically, the generative adversarial network (GAN) strategy is embedded to distinguish the source of the generated feature maps, thereby facilitating the use of the learned 2D and 3D local feature descriptors for cross-dimensional retrieval. Meanwhile, a metric network is embedded to compute the similarity between the learned 2D and 3D local feature descriptors. Finally, we construct a 2D-3D consistent loss function to optimize the 2D3D-DescNet. In this paper, the cross-dimensional matching of 2D images and 3D point clouds is explored with the small object of the 3Dmatch dataset. Experimental results demonstrate that the 2D and 3D local feature descriptors jointly learned by 2D3D-DescNet are similar. In addition, in terms of 2D and 3D cross-dimensional retrieval and matching between 2D image patches and 3D point cloud volumes, the proposed 2D3D-DescNet significantly outperforms the current state-of-the-art approaches based on jointly learning 2D and 3D feature descriptors; the cross-dimensional retrieval at TOP1 on the 3DMatch dataset is improved by over 12%.

B3D-EAR: Binarized 3D descriptors for ear-based human recognition

Traditional 3D feature descriptors often utilize real-valued vectors, posing challenges in terms of computational complexity and space constraints during matching. This study introduces a novel approach for generating binary 3D feature descriptors using correntropy, an online estimate of Renyi’s quadratic entropy. By employing this technique, the efficiency of state-of-the-art 3D descriptors is significantly enhanced. The effectiveness of the proposed methodology is demonstrated through evaluations on two datasets, UND-J2 and an in-house dataset, comprising 3D scans of human ears. Efficient algorithms for matching keypoints and their binary descriptors are employed, addressing both space and time complexities. The matching process is assessed using distance metrics and iterative closest point (ICP) alignment. Experimental results reveal that the method achieves a recognition performance of 98.62% with an equal error rate (EER) of 1.54%, comparable to state-of-the-art techniques. This highlights the efficacy of the method in generating binary descriptors without compromising accuracy.

Matching of crushed highly decomposed granite particles using 3D SHOT descriptors

Owing to the complex morphology and the fragility of highly decomposed granite (HDG), it is still a challenge to track the breakage process of HDG particles within a deformed sample in geomechanics research. An HDG particle matching method is proposed in this paper for the first time to track and match HDG fragments with their mother particles from which they break off. First, the X-ray micro-computed tomography (μCT) images of HDG particles were acquired from a one-dimensional compression test and a certain number of HDG particles and fragments were selected and reconstructed. Second, the original and fracture surfaces of the fragments were identified and segmented. Then, the essential morphological features of the particles and fragments were represented with the key points. The neighbourhood information around the key points was encoded by the three-dimensional (3D) feature descriptor. Next, the mismatches were filtered out by the random sample consensus (RANSAC) algorithm based on the similarity of the local neighbourhood information. Finally, the influences of several parameters on the matching results were investigated. The capability and efficiency of the HDG particle matching method were demonstrated using the μCT data of ten HDG particles and 32 fragments that resulted from the crushing of these particles.

Cross domain 2D-3D descriptor matching for unconstrained 6-DOF pose estimation

This paper presents a novel approach for cross-domain descriptor matching between 2D and 3D modalities. The 2D-3D matching is applied to localize 2D images in 3D point clouds. Direct cross-domain matching allows our technique to localize images in any type of 3D point cloud without any constraints on the nature or mechanism by which it is obtained. We propose a learning based framework, called Desc-Matcher, to directly match features between the two modalities. A dataset of 2D and 3D features with corresponding locations in images and point clouds is generated to train the Desc-Matcher. To estimate the pose of an image in any 3D cloud, keypoints and feature descriptors are extracted from the query image and the point cloud. The trained Desc-Matcher is then used to match the features from the image and the point cloud. A robust pose estimator is used to predict the location and orientation of the query image from the corresponding positions of the matched 2D and 3D features. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results show that the proposed approach can reliably estimate the 6-DOF poses of query cameras in any type of 3D point cloud with high precision. We achieved average median errors of 1.09cm/0.27∘ and 19cm/0.39∘ on the Stanford and Cambridge datasets, respectively.

Application of machine vision image feature recognition in 3D map construction

Simultaneous localization and mapping algorithm is a technique to simultaneously construct a map of the environment by the sensors it carries. In this paper, we focus on the SLAM (Simultaneous Localization and Mapping) algorithm based on machine vision-depth information, overlap region determination among local sub-maps, and fusion algorithm of 3D dense maps, and analyze four modules of front-end visual odometry, back-end optimization, loopback detection, and map construction. The front-end visual odometry adopts the ORB (Oriented FAST and Rotated BRIEF) algorithm based on the feature point method to extract and match the features between image frames and uses the iterative nearest point method to solve the camera motion between adjacent images. Meanwhile, a 3D feature descriptor based on the histogram of spatial distribution is designed to encode the feature information around the key points, and the Kd-Tree algorithm is used for radial nearest neighbor search to complete the feature matching. Finally, the 3D map is fused by the 3D-ICP algorithm. The experiments prove that the low-dimensional local feature descriptor designed has good descriptiveness and is superior, and the 3D map fusion based on this descriptor has a good fusion effect and can meet the accuracy requirements of practical applications.

Does the 3D Feature Descriptor Impact The Registration Accuracy in Laparoscopic Liver Surgery?

Abstract In laparoscopic liver surgery (LLS) image-guided navigation systems could support the surgeon by providing subsurface information such as the positions of tumors and vessels. For this purpose, one option is to perform a registration of preoperative 3D data and 3D surface patches reconstructed from laparoscopic images. Part of an automatic 3D registration pipeline is the feature description, which takes into account various geometric and spatial information. Since there is no leading feature descriptor in the field of LLS, two feature descriptors are compared in this paper: The Fast Point Feature Histogram (FPFH) and Triple Orthogonal Local Depth Images (TOLDI). To evaluate their performance, three perturbations were induced: varying surface patch sizes, spatial displacement, and Gaussian deformation. Registration was performed using the RANSAC algorithm. FPFH outperformed TOLDI for small surface patches and in case of Gaussian deformations in terms of registration accuracy. In contrast, TOLDI showed lower registration errors for patches with spatial displacement. While developing a 3D-3D registration pipeline, the choice of the feature descriptor is of importance, consequently a careful choice suitable for the application in LLS is necessary.

Geometric feature statistics histogram for both real-valued and binary feature representations of 3D local shape

3D local feature description is now at the core of many 3D vision technologies. However, most of the existing 3D feature descriptors can't strike a balance among descriptiveness, robustness, compactness, and efficiency. To overcome the challenges, we propose a real-valued 3D local feature descriptor named Geometric Feature Statistics Histogram (GFSH) and its binary extension descriptor named B-GFSH. A GFSH descriptor first constructs an improved-weighted covariance matrix to solve a stable and reliable Local Reference Frame (LRF), and then achieves a comprehensive description of the 3D local surface by performing statistics on multiple geometric distribution features, namely voxel density, voxel centroid, and projection density. A particular trait of our GFSH descriptor is its seamless extension to the binary representation to reduce storage consumption and accelerate feature matching. For each sub-feature of GFSH, B-GFSH respectively adopts the corresponding binarization strategy, i.e., improved Gray code quantization, thresholding based on coordinates, and neighbor comparison. Extensive experiments on six public datasets prove that both GFSH and B-GFSH have high descriptiveness, strong robustness, and fast real-time performance. In addition, B-GFSH further has the characteristics of fast matching speed, low memory footprint, and high compactness. Finally, we conduct 3D scene registration and 3D object recognition experiments to visually demonstrate the actual effectiveness of GFSH and B-GFSH.

GACM: A Graph Attention Capsule Model for the Registration of TLS Point Clouds in the Urban Scene

Point cloud registration is the foundation and key step for many vital applications, such as digital city, autonomous driving, passive positioning, and navigation. The difference of spatial objects and the structure complexity of object surfaces are the main challenges for the registration problem. In this paper, we propose a graph attention capsule model (named as GACM) for the efficient registration of terrestrial laser scanning (TLS) point cloud in the urban scene, which fuses graph attention convolution and a three-dimensional (3D) capsule network to extract local point cloud features and obtain 3D feature descriptors. These descriptors can take into account the differences of spatial structure and point density in objects and make the spatial features of ground objects more prominent. During the training progress, we used both matched points and non-matched points to train the model. In the test process of the registration, the points in the neighborhood of each keypoint were sent to the trained network, in order to obtain feature descriptors and calculate the rotation and translation matrix after constructing a K-dimensional (KD) tree and random sample consensus (RANSAC) algorithm. Experiments show that the proposed method achieves more efficient registration results and higher robustness than other frontier registration methods in the pairwise registration of point clouds.

Local features of 3D point cloud registration based on Siamese network learning

ABSTRACT To solve the problem of point cloud registration caused by noisy or incomplete data, a deep learning model based on the 3D Siamese convolution neural network is proposed. The key idea of our algorithm is the acquisition of the training data. To accumulate training data for our model, we project 3D point cloud data onto a 2D plane to obtain cumulative projection images, then we use the SIFT algorithm to match the images and obtain the matching point pairs. On this basis, according to the relationship between the matching point pairs and 3D point cloud, the 3D point blocks are obtained, and finally the Siamese convolution neural network is used to learn the 3D point block pairs and obtain the 3D feature descriptor. The point cloud can be effectively matched by using the feature descriptor and Random Sample Consensus (RANSAC) algorithm. Experiments show that the descriptor consistently outperforms other state-of-the-art approaches by a significant margin; therefore, this method offers significant improvement in processing speed and registration accuracy.

Performance Evaluation of 3D Descriptors Paired with Learned Keypoint Detectors

Matching surfaces is a challenging 3D Computer Vision problem typically addressed by local features. Although a plethora of 3D feature detectors and descriptors have been proposed in literature, it is quite difficult to identify the most effective detector-descriptor pair in a certain application. Yet, it has been shown in recent works that machine learning algorithms can be used to learn an effective 3D detector for any given 3D descriptor. In this paper, we present a performance evaluation of the detector-descriptor pairs obtained by learning a 3D detector for the most popular 3D descriptors. Purposely, we address experimental settings dealing with object recognition and surface registration. Our results show how pairing a learned detector to a learned descriptors like CGF leads to effective local features when pursuing object recognition (e.g., 0.45 recall at 0.8 precision on the UWA dataset), while there is not a clear performance gap between CGF and effective hand-crafted features like SHOT for surface registration (0.18 average precision for the former versus 0.16 for the latter).

Segmentation of MRI brain scans using spatial constraints and 3D features.

This paper presents a novel unsupervised algorithm for brain tissue segmentation in magnetic resonance imaging (MRI). The proposed algorithm, named Gardens2, adopts a clustering approach to segment voxels of a given MRI into three classes: cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM). Using an overlapping criterion, 3D feature descriptors and prior atlas information, Gardens2 generates a segmentation mask per class in order to parcellate the brain tissues. We assessed our method using three neuroimaging datasets: BrainWeb, IBSR18, and IBSR20, the last two provided by the Internet Brain Segmentation Repository. Its performance was compared with eleven well established as well as newly proposed unsupervised segmentation methods. Overall, Gardens2 obtained better segmentation performance than the rest of the methods in two of the three databases and competitive results when its performance was measured by class. Graphical Abstract Brain tissue segmentation using 3D features and an adjusted atlas template.

HoPPF: A novel local surface descriptor for 3D object recognition

Three-dimensional feature descriptors play an important role in 3D computer vision because they are widely employed in many 3D perception applications to extract point correspondences between two point clouds. However, most existing description methods suffer from either weak robustness, low descriptiveness, or costly computation. Thus, a 3D local feature descriptor named Histograms of Point Pair Features (HoPPF) is proposed in this paper, and it is aimed at robust representation, high descriptiveness, and efficient computation. First, we propose a novel method to redirect surface normals and use the Poisson-disk sampling strategy to solve the problem of data redundancy in data pre-processing. Second, a new technique is applied to divide the local point pair set of each keypoint into eight regions. Then, the distribution of local point pairs of each region is used to construct the corresponding sub-features. Finally, the proposed HoPPF is generated by concatenating all sub-features into a vector. The performance of the HoPPF method is rigorously evaluated on several standard datasets. The results of the experiments and comparisons with other state-of-the-art methods validate the superiority of the HoPPF descriptor in term of robustness, descriptiveness, and efficiency. Moreover, the proposed technique for division of point pair sets is used to modify the other typical point-pair-based descriptor (i.e., PFH) to show its generalization ability. The proposed HoPPF is also applied to object recognition on real datasets captured by different devices (e.g., Kinect and LiDAR) to verify the feasibility of this method for 3D vision applications.

Correspondence matching in unorganized 3D point clouds using Convolutional Neural Networks

This document presents a novel method based on Convolutional Neural Networks (CNN) to obtain correspondence matchings between sets of keypoints of several unorganized 3D point cloud captures, independently of the sensor used. The proposed technique extends a state-of-the-art method for correspondence matching in standard 2D images to sets of unorganized 3D point clouds. The strategy consists of projecting the 3D neighborhood of the keypoint onto an RGBD patch, and the classification of patch pairs using CNNs. The objective evaluation of the proposed 3D point matching based on CNNs outperforms existing 3D feature descriptors, especially when intensity or color data is available.

False positive reduction in pulmonary nodule classification using 3D texture and edge feature in CT images.

Pulmonary nodule detection can significantly influence the early diagnosis of lung cancer while is confused by false positives. In this study, we focus on the false positive reduction and present a method for accurate and rapid detection of pulmonary nodule from suspective regions with 3D texture and edge feature. This work mainly consists of four modules. Firstly, small pulmonary nodule candidates are preprocessed by a reconstruction approach for enhancing 3D image feature. Secondly, a texture feature descriptor is proposed, named cross-scale local binary patterns (CS-LBP), to extract spatial texture information. Thirdly, we design a 3D edge feature descriptor named orthogonal edge orientation histogram (ORT-EOH) to obtain spatial edge information. Finally, hierarchical support vector machines (H-SVMs) is used to classify suspective regions as either nodules or non-nodules with joint CS-LBP and ORT-EOH feature vector. For the solitary solid nodule, ground-glass opacity, juxta-vascular nodule and juxta-pleural nodule, average sensitivity, average specificity and average accuracy of our method are 95.69%, 96.95% and 96.04%, respectively. The elapsed time in training and testing stage are 321.76s and 5.69s. Our proposed method has the best performance compared with other state-of-the-art methods and is shown the improved precision of pulmonary nodule detection with computationaly low cost.

Toward the Repeatability and Robustness of the Local Reference Frame for 3D Shape Matching: An Evaluation.

The local reference frame (LRF), as an independent coordinate system constructed on the local 3D surface, is broadly employed in 3D local feature descriptors. The benefits of the LRF include rotational invariance and full 3D spatial information, thereby greatly boosting the distinctiveness of a 3D feature descriptor. There are numerous LRF methods in the literature; however, no comprehensive study comparing their repeatability and robustness performance under different application scenarios and nuisances has been conducted. This paper evaluates eight state-of-the-art LRF proposals on six benchmarks with different data modalities (e.g., LiDAR, Kinect, and Space Time) and application contexts (e.g., shape retrieval, 3D registration, and 3D object recognition). In addition, the robustness of each LRF to a variety of nuisances, including varying support radii, Gaussian noise, outliers (shot noise), mesh resolution variation, distance to boundary, keypoint localization error, clutter, occlusion, and partial overlap, is assessed. The experimental study also measures the performance under different keypoint detectors, descriptor matching performance when using different LRFs and feature representation combinations, as well as computational efficiency. Considering the evaluation outcomes, we summarize the traits, advantages, and current limitations of the tested LRF methods.

Multimodal Visual Data Registration for Web-Based Visualization in Media Production

Recent developments of video and sensing technology have led to large volumes of digital media data. Current media production relies on videos from the principal camera together with a wide variety of heterogeneous source of supporting data [photos, light detection and ranging point clouds, witness video camera, high dynamic range imaging, and depth imagery]. Registration of visual data acquired from various 2D and 3D sensing modalities is challenging because current matching and registration methods are not appropriate due to differences in structure, format, and noise characteristics for multimodal data. A combined 2D/3D visualization of this registered data allows an integrated overview of the entire data set. For such a visualization, a Web-based context presents several advantages. In this paper, we propose a unified framework for registration and visualization of this type of visual media data. A new feature description and matching method is proposed, adaptively considering local geometry, semiglobal geometry, and color information in the scene for more robust registration. The resulting registered 2D/3D multimodal visual data are too large to be downloaded and viewed directly via the Web browser, while maintaining an acceptable user experience. Thus, we employ hierarchical techniques for compression and restructuring to enable efficient transmission and visualization over the Web, leading to interactive visualization as registered point clouds, 2D images, and videos in the browser, improving on the current state-of-the-art techniques for Web-based visualization of big media data. This is the first unified 3D Web-based visualization of multimodal visual media production data sets. The proposed pipeline is tested on big multimodal data set typical of film and broadcast production, which are made publicly available. The proposed feature description method shows two times higher precision of feature matching and more stable registration performance than existing 3D feature descriptors.

Histograms of Gaussian normal distribution for 3D feature matching in cluttered scenes

3D feature descriptors provide essential information to find given models in captured scenes. In practical applications, these scenes often contain clutter. This imposes severe challenges on the 3D object recognition leading to feature mismatches between scenes and models. As such errors are not fully addressed by the existing methods, 3D feature matching still remains a largely unsolved problem. We therefore propose our Histograms of Gaussian Normal Distribution (HGND) for capturing salient feature information on a local reference frame (LRF) that enables us to solve this problem. We define a LRF on each local surface patch by using the eigenvectors of the scatter matrix. Different from the traditional local LRF-based methods, our HGND descriptor is based on the combination of geometrical and spatial information without calculating the distribution of every point and its geometrical information in a local domain. This makes it both simple and efficient. We encode the HGND descriptors in a histogram by the geometrical projected distribution of the normal vectors. These vectors are based on the spatial distribution of the points. We use three public benchmarks, the Bologna, the UWA and the Ca’ Foscari Venezia dataset, to evaluate the speed, robustness, and descriptiveness of our approach. Our experiments demonstrate that the HGND is fast and obtains a more reliable matching rate than state-of-the-art approaches in cluttered situations.

BRoPH: An efficient and compact binary descriptor for 3D point clouds

3D feature descriptor plays an essential role in 3D computer vision as it is a pre-requisite step for many 3D vision applications. Despite there exists many 3D feature descriptors currently, they are mostly represented in floating representation, resulting costly computation and storage. In this paper, we propose a 3D binary local feature descriptor, Binary Rotational Projection Histogram (BRoPH), aimed at compactness of representation and efficiency of computation. BRoPH is generated directly from point cloud by turning the description of 3D point cloud into a series binarization of 2D image patches. The exploited local reference frame promotes the construction efficiency meanwhile maintains repeatability and stability, the multi-view mechanism and integration of density distribution and depth information employed in BRoPH complement each other and enhance its descriptiveness, and the multi-scale extension of Center-Symmetric Local Binary Patterns (CS-LBP) provides an efficient and compact way to generate binary string. We compare BRoPH against several representative descriptors on public datasets and demonstrate that it achieves about 14 times more compact, 28 and 4 times more faster in terms of describing and matching time respectively, than the average performance of the compared floating descriptors.

A novel binary shape context for 3D local surface description

3D local surface description is now at the core of many computer vision technologies, such as 3D object recognition, intelligent driving, and 3D model reconstruction. However, most of the existing 3D feature descriptors still suffer from low descriptiveness, weak robustness, and inefficiency in both time and memory. To overcome these challenges, this paper presents a robust and descriptive 3D Binary Shape Context (BSC) descriptor with high efficiency in both time and memory. First, a novel BSC descriptor is generated for 3D local surface description, and the performance of the BSC descriptor under different settings of its parameters is analyzed. Next, the descriptiveness, robustness, and efficiency in both time and memory of the BSC descriptor are evaluated and compared to those of several state-of-the-art 3D feature descriptors. Finally, the performance of the BSC descriptor for 3D object recognition is also evaluated on a number of popular benchmark datasets, and an urban-scene dataset is collected by a terrestrial laser scanner system. Comprehensive experiments demonstrate that the proposed BSC descriptor obtained high descriptiveness, strong robustness, and high efficiency in both time and memory and achieved high recognition rates of 94.8%, 94.1% and 82.1% on the considered UWA, Queen, and WHU datasets, respectively.

B-SHOT: a binary 3D feature descriptor for fast Keypoint matching on 3D point clouds

We present the first attempt in creating a binary 3D feature descriptor for fast and efficient keypoint matching on 3D point clouds. Specifically, we propose a binarization technique and apply it on the state-of-the-art 3D feature descriptor, SHOT (Salti et al., Comput Vision Image Underst 125:251–264, 2014) to create the first binary 3D feature descriptor, which we call B-SHOT. B-SHOT requires 32 times lesser memory for its representation while being six times faster in feature descriptor matching, when compared to the SHOT feature descriptor. Next, we propose a robust evaluation metric, specifically for 3D feature descriptors. A comprehensive evaluation on standard benchmarks reveals that B-SHOT offers comparable keypoint matching performance to that of the state-of-the-art real valued 3D feature descriptors, albeit at dramatically lower computational and memory costs.