3D Local Shape Research Articles

ShapeBench: A new approach to benchmarking local 3D shape descriptors

The ShapeBench evaluation methodology is proposed as an extension to the popular Area Under Precision-Recall Curve (PRC/AUC) for measuring the matching performance of local 3D shape descriptors. It is observed that the PRC inadequately accounts for other similar surfaces in the same or different objects when determining whether a candidate match is a true positive. The novel Descriptor Distance Index (DDI) metric is introduced to address this limitation. In contrast to previous evaluation methodologies, which identify entire objects in a given scene, the DDI metric measures descriptor performance by analysing point-to-point distances. The ShapeBench methodology is also more scalable than previous approaches, by using procedural generation. The benchmark is used to evaluate both old and new descriptors. The results produced by the implementation of the benchmark are fully replicable, and are made publicly available.

Both real-valued and binary multi-feature fusion histograms for 3D local shape representation

Both real-valued and binary multi-feature fusion histograms for 3D local shape representation

3D building similarity for a random single-view-image pair based on a local 3D shape

3D building similarity for a random single-view-image pair based on a local 3D shape

Trigonometric projection statistics histograms for 3D local feature representation and shape description

Feature representation as a significant approach to three-dimensional (3D) shape description has been widely employed in computer vision. However, most existing methods are suffering from the emerging challenges for descriptiveness, robustness and efficiency. This paper presents a novel feature descriptor named trigonometric projection statistics histograms (TPSH). By constructing the repeatable local reference frame based on a multi-attribute weighting strategy, TPSH can address many prevailing nuisances such as noise, occlusion and varying resolution. The trigonometric projection mechanism is originally proposed for TPSH generation, which combines two perspective views to encode both spatial distribution and geometrical measurements from local shape into statistics histograms. The experimental evaluation on public datasets proves that TPSH outperforms state-of-the-art methods in descriptiveness and robustness while maintaining storage compactness and computational efficiency. It is demonstrated that TPSH can not only be suited for 3D object recognition and shape registration, but also generalized across various acquisition devices, data modalities and application scenarios.

SliceLRF: A Local Reference Frame Sliced along the Height on the 3D Surface.

The local reference frame (LRF) plays a vital role in local 3D shape description and matching. Numerous LRF methods have been proposed in recent decades. However, few LRFs can achieve a balance between repeatability and robustness under exposure to a variety of nuisances, including Gaussian noise, mesh resolution variation, clutter, and occlusion. Additionally, most LRFs are heuristic and lack generalizability to different applications and data modalities. In this paper, we first define the degree of distinction to describe the distribution of 2D point clouds and explore the relationship between the relative deviation of the distinction degree and the LRF error through experiments. Based on Gaussian noise and a random sampling analysis, several factors that affect the relative deviation of the distinction degree and result in the LRF error are identified. A scoring criterion is proposed to evaluate the robustness of the point cloud distribution. On this basis, we propose an LRF method (SliceLRF) based on slicing along the Z-axis, which selects the most robust adjacent slices in the point cloud region by scoring criteria for X-axis estimation to improve the repeatability and robustness. SliceLRF is rigorously tested on four public benchmark datasets which have different applications and involve different data modalities. It is also compared with the state-of-the-art LRFs. The experimental results show that the SliceLRF has more comprehensive repeatability and robustness than the other LRFs under exposure to Gaussian noise and random sampling.

Rotational Voxels Statistics Histogram for both real-valued and binary feature representations of 3D local shape

3D Local feature description is an active and fundamental task in 3D computer vision. However, most of the existing descriptors fail to simultaneously achieve satisfactory performance among descriptiveness, robustness, efficiency, and compactness. To address these limitations, we first propose a real-valued descriptor named Rotational Voxels Statistics Histogram (RoVo), which exploits the novel 3D multi-pose processing mechanism proposed in this paper to calculate the 3D voxel density distribution in different 3D poses. Moreover, through well-designed binary encoding algorithms, we conduct the seamless extension of the real-valued RoVo descriptor to three binary representations that have different performance characteristics. Extensive evaluation experiments validate the superiority of the real-valued and three binary RoVo descriptors concerning descriptiveness, robustness, and efficiency. Furthermore, the three binary RoVo descriptors extend the performance of high compactness. Lastly, we perform the experiments of 3D scene registration and 3D object recognition to intuitively present the effectiveness of the four proposed RoVo descriptors.

Uniform and Competency-Based 3D Keypoint Detection for Coarse Registration of Point Clouds with Homogeneous Structure

Recent advances in 3D laser scanner technology have provided a large amount of accurate geo-information as point clouds. The methods of machine vision and photogrammetry are used in various applications such as medicine, environmental studies, and cultural heritage. Aerial laser scanners (ALS), terrestrial laser scanners (TLS), mobile mapping laser scanners (MLS), and photogrammetric cameras via image matching are the most important tools for producing point clouds. In most applications, the process of point cloud registration is considered to be a fundamental issue. Due to the high volume of initial point cloud data, 3D keypoint detection has been introduced as an important step in the registration of point clouds. In this step, the initial volume of point clouds is converted into a set of candidate points with high information content. Many methods for 3D keypoint detection have been proposed in machine vision, and most of them were based on thresholding the saliency of points, but less attention had been paid to the spatial distribution and number of extracted points. This poses a challenge in the registration process when dealing with point clouds with a homogeneous structure. As keypoints are selected in areas of structural complexity, it leads to an unbalanced distribution of keypoints and a lower registration quality. This research presents an automated approach for 3D keypoint detection to control the quality, spatial distribution, and the number of keypoints. The proposed method generates a quality criterion by combining 3D local shape features, 3D local self-similarity, and the histogram of normal orientation and provides a competency index. In addition, the Octree structure is applied to control the spatial distribution of the detected 3D keypoints. The proposed method was evaluated for the keypoint-based coarse registration of aerial laser scanner and terrestrial laser scanner data, having both cluttered and homogeneous regions. The obtained results demonstrate the proper performance of the proposed method in the registration of these types of data, and in comparison to the standard algorithms, the registration error was diminished by up to 56%.

Dual spin-image: A bi-directional spin-image variant using multi-scale radii for 3D local shape description

Obtaining good feature descriptors for 3D local shape under different conditions, such as noise, clutter, occlusion, and limited overlap, is still a challenging task in computer vision. In this paper, we present a variant of spin-image called dual spin-image (Dual-SI) for 3D local shape description that encodes bi-directional information between an oriented point and its surrounding neighbors within a given radius. By accumulating the parameters that encode the 3D point information into a 2D space bidirectionally, we can generate two different signatures. These signatures are then concatenated to form the Dual-SI feature descriptor. Finally, we propose a multi-scale radius approach to address the problem of occlusion and make use of a weighted kernel to address the noise problem. We tested our Dual-SI feature descriptor on popular datasets addressing 3D object registration, 3D object recognition, and shape retrieval scenarios. We also conduct experiments for 3D point cloud registration to further evaluate the effectiveness of our method. Consensus experimental results show that our Dual-SI achieves outstanding performance on datasets with various nuisances and application contexts.

Geometric feature statistics histogram for both real-valued and binary feature representations of 3D local shape

3D local feature description is now at the core of many 3D vision technologies. However, most of the existing 3D feature descriptors can't strike a balance among descriptiveness, robustness, compactness, and efficiency. To overcome the challenges, we propose a real-valued 3D local feature descriptor named Geometric Feature Statistics Histogram (GFSH) and its binary extension descriptor named B-GFSH. A GFSH descriptor first constructs an improved-weighted covariance matrix to solve a stable and reliable Local Reference Frame (LRF), and then achieves a comprehensive description of the 3D local surface by performing statistics on multiple geometric distribution features, namely voxel density, voxel centroid, and projection density. A particular trait of our GFSH descriptor is its seamless extension to the binary representation to reduce storage consumption and accelerate feature matching. For each sub-feature of GFSH, B-GFSH respectively adopts the corresponding binarization strategy, i.e., improved Gray code quantization, thresholding based on coordinates, and neighbor comparison. Extensive experiments on six public datasets prove that both GFSH and B-GFSH have high descriptiveness, strong robustness, and fast real-time performance. In addition, B-GFSH further has the characteristics of fast matching speed, low memory footprint, and high compactness. Finally, we conduct 3D scene registration and 3D object recognition experiments to visually demonstrate the actual effectiveness of GFSH and B-GFSH.

Large depth-of-field 3D measurement with a microscopic structured-light system

Fringe projection techniques have been widely used in three-dimensional (3D) microscopic measurement; however, the working volume is severely restricted due to the shallow depth-of-field (DOF) of the telecentric lenses used. In this paper, a large-DOF 3D measurement method for use with a microscopic structured-light system was proposed. By employing a precisely controlled moving platform, the height of the camera can be adjusted to capture several sets of fringe patterns with different amounts of defocusing. Then the proposed focusing map generation algorithm was employed to calculate the focusing degree of each pixel and generate a focusing map for each set of fringe patterns. Subsequently, local 3D shape data corresponding to in-focus pixels were merged with the guidance of the focusing maps, and global fine 3D measurement data without the influence of lens blur can be obtained. Specifically, to guide the compositing of the 3D shape data, a novel, simple focusing map generation algorithm was proposed to calculate the focusing degree of each set of captured fringe patterns. In contrast to existing DOF-extending 3D measurement techniques using multiple focal distances, the proposed algorithm does not need the object’s texture to calculate the gray-level gradient; hence, complex convolution or iteration is not required. The experimental results demonstrate that the proposed technique can increase the measurement DOF to approximately three times that of a conventional system.

LRF-Net: Learning Local Reference Frames for 3D Local Shape Description and Matching.

The local reference frame (LRF) acts as a critical role in 3D local shape description and matching. However, most existing LRFs are hand-crafted and suffer from limited repeatability and robustness. This paper presents the first attempt to learn an LRF via a Siamese network that needs weak supervision only. In particular, we argue that each neighboring point in the local surface gives a unique contribution to LRF construction and measure such contributions via learned weights. Extensive analysis and comparative experiments on three public datasets addressing different application scenarios have demonstrated that LRF-Net is more repeatable and robust than several state-of-the-art LRF methods (LRF-Net is only trained on one dataset). We show that LRFNet achieves 0.686 MeanCos performance on the UWA 3D modeling (UWA3M) dataset, outperforming the closest method by 0.18. In addition, LRF-Net can significantly boost the local shape description and 6-DoF pose estimation performance when matching 3D point clouds.

Large-scale structured light 3D shape measurement with reverse photography

Combining the advantages of structured light and close range photogrammetry is an effective strategy for large-scale three-dimensional (3D) optical shape measurement. Several coded targets are typically required to be pasted on the object being tested, which induces several shortcomings. This paper proposes a structured light measurement technique based on reverse photography. An auxiliary reverse camera is installed behind the structured light system. When a single local measurement is performed, the reverse camera observes a large liquid crystal display (LCD) screen that is used as an oriented phase target. According to the multiple imaging constraints for the reverse camera, the local 3D shape data of multi-view structured light measurements are unified in a global frame defined on the LCD screen, achieving a holistic 3D shape data fusion. The proposed method possesses the advantage of reverse information transmission and pose transfer, achieving non-contact large-scale 3D shape measurement.

An efficient and compact 3D local descriptor based on the weighted height image

3D local descriptors are the fundamental and essential elements that have been commonly applied in 3D computer vision. This paper proposes a novel and effective 3D local descriptor for describing the 3D local shape. The research focuses on accelerating the descriptor generation by simplifying the Local Reference Frame (LRF) and optimizing the feature space through a Weighted Height Image (WHI). An in-depth theoretical analysis of the LRF is conducted. Then, this study proposes a simplified LRF to reduce the redundant computations of the covariance matrix and share the calculations with the 3D information coding. Besides, the feature space is modeled and analyzed in this paper. Based on the analysis, we propose a weighting function to strengthen the abilities of the feature representation. The experimental results indicate that the proposed WHI descriptor outperforms the state-of-the-art (SOTA) algorithms in terms of accuracy and efficiency. Meanwhile, the compactness of the WHI is about six times more than that of the SOTA algorithms. Moreover, for the application of point cloud registration, the proposed WHI exhibits high effectiveness in terms of both accuracy and real-time capability.

PointCleanNet: Learning to Denoise and Remove Outliers from Dense Point Clouds

AbstractPoint clouds obtained with 3D scanners or by image‐based reconstruction techniques are often corrupted with significant amount of noise and outliers. Traditional methods for point cloud denoising largely rely on local surface fitting (e.g. jets or MLS surfaces), local or non‐local averaging or on statistical assumptions about the underlying noise model. In contrast, we develop a simple data‐driven method for removing outliers and reducing noise in unordered point clouds. We base our approach on a deep learning architecture adapted from PCPNet, which was recently proposed for estimating local 3D shape properties in point clouds. Our method first classifies and discards outlier samples, and then estimates correction vectors that project noisy points onto the original clean surfaces. The approach is efficient and robust to varying amounts of noise and outliers, while being able to handle large densely sampled point clouds. In our extensive evaluation, both on synthetic and real data, we show an increased robustness to strong noise levels compared to various state‐of‐the‐art methods, enabling accurate surface reconstruction from extremely noisy real data obtained by range scans. Finally, the simplicity and universality of our approach makes it very easy to integrate in any existing geometry processing pipeline. Both the code and pre‐trained networks can be found on the project page (https://github.com/mrakotosaon/pointcleannet).

PCPNet Learning Local Shape Properties from Raw Point Clouds

AbstractIn this paper, we propose PCPNET, a deep‐learning based approach for estimating local 3D shape properties in point clouds. In contrast to the majority of prior techniques that concentrate on global or mid‐level attributes, e.g., for shape classification or semantic labeling, we suggest a patch‐based learning method, in which a series of local patches at multiple scales around each point is encoded in a structured manner. Our approach is especially well‐adapted for estimating local shape properties such as normals (both unoriented and oriented) and curvature from raw point clouds in the presence of strong noise and multi‐scale features. Our main contributions include both a novel multi‐scale variant of the recently proposed PointNet architecture with emphasis on local shape information, and a series of novel applications in which we demonstrate how learning from training data arising from well‐structured triangle meshes, and applying the trained model to noisy point clouds can produce superior results compared to specialized state‐of‐the‐art techniques. Finally, we demonstrate the utility of our approach in the context of shape reconstruction, by showing how it can be used to extract normal orientation information from point clouds.

Local voxelized structure for 3D binary feature representation and robust registration of point clouds from low-cost sensors

Local feature-based 3D point cloud registration is a central issue in the fields of 3D computer vision and robotics, and most previously proposed 3D local features are real-valued. In this paper (1) a novel binary descriptor named local voxelized structure (LoVS) for 3D local shape description and (2) a LoVS-based registration algorithm for low-quality, e.g., Kinect-captured, point clouds are proposed. LoVS simply encodes the local shape structure represented by point clouds into bit string using point spatial locations without computing complex geometric attributes, such as normals and curvature, at the feature representation stage. Specifically, the LoVS descriptor is extracted within a local cubic volume around the keypoint. The orientation of the cubic volume is determined by a local reference frame (LRF) to achieve rotation invariance. Then, the cubic volume is uniformly split into a set of voxels. A voxel is labeled 1 if it contains points; otherwise, 0. All the labels are integrated into the LoVS descriptor. Based on the LoVS descriptor, a robust and accurate point cloud registration algorithm was developed, which effectively handles the challenges presented by low-cost sensors, e.g., noise and varying data resolutions. Experiments and extensive comparisons on three descriptor-matching benchmarks and a large-scale Kinect point cloud registration dataset show the effectiveness and the over-all superiority of our proposed LoVS descriptor and LoVS-based point cloud registration algorithm.

Data‐Driven Sparse Priors of 3D Shapes

AbstractWe present a sparse optimization framework for extracting sparse shape priors from a collection of 3D models. Shape priors are defined as point‐set neighborhoods sampled from shape surfaces which convey important information encompassing normals and local shape characterization. A 3D shape model can be considered to be formed with a set of 3D local shape priors, while most of them are likely to have similar geometry. Our key observation is that the local priors extracted from a family of 3D shapes lie in a very low‐dimensional manifold. Consequently, a compact and informative subset of priors can be learned to efficiently encode all shapes of the same family. A comprehensive library of local shape priors is first built with the given collection of 3D models of the same family. We then formulate a global, sparse optimization problem which enforces selecting representative priors while minimizing the reconstruction error. To solve the optimization problem, we design an efficient solver based on the Augmented Lagrangian Multipliers method (ALM). Extensive experiments exhibit the power of our data‐driven sparse priors in elegantly solving several high‐level shape analysis applications and geometry processing tasks, such as shape retrieval, style analysis and symmetry detection.

Surface reconstruction with data-driven exemplar priors

In this paper, we propose a framework to reconstruct 3D models from raw scanned points by learning the prior knowledge of a specific class of objects. Unlike previous work that heuristically specifies particular regularities and defines parametric models, our shape priors are learned directly from existing 3D models under a framework based on affinity propagation. Given a database of 3D models within the same class of objects, we build a comprehensive library of 3D local shape priors. We then formulate the problem to select as-few-as-possible priors from the library, referred to as exemplar priors. These priors are sufficient to represent the 3D shapes of the whole class of objects from where they are generated. By manipulating these priors, we are able to reconstruct geometrically faithful models with the same class of objects from raw point clouds. Our framework can be easily generalized to reconstruct various categories of 3D objects that have more geometrically or topologically complex structures. Comprehensive experiments exhibit the power of our exemplar priors for gracefully solving several problems in 3D shape reconstruction such as preserving sharp features, recovering fine details and so on.

Rotational contour signatures for both real-valued and binary feature representations of 3D local shape

This paper presents a rotational contour signatures (RCS) method for both real-valued and binary descriptions of 3D local shape. RCS comprises several signatures that characterize the 2D contour information derived from 3D-to-2D projection of the local point cloud. The inspiration of our encoding technique comes from that when viewing towards an object, its contour is an effective and robust cue for representing its shape. In order to achieve a comprehensive geometry encoding, the local surface is continually rotated in a predefined local reference frame (LRF) so that multi-view information is obtained. A peculiar trait of our RCS method is its seamless extension to binary representations to accelerate feature matching and reduce storage consumption. Specifically, we resort to three techniques, i.e., thresholding, quantization and geometrical binary encoding, to generate RCS binary strings. In contrast to 2D image area, there are quite rare 3D binary descriptors yet in 3D computer vision. We deploy experiments on three standard datasets including shape retrieval, 3D object recognition and 2.5D point cloud view matching scenarios with a rigorous comparison with six state-of-the-art descriptors. The comparative outcomes confirm numerous merits of our RCS method, e.g., highly discriminative, compact, computational efficient and robust to many nuisances including noise, mesh resolution variation, clutter and occlusion. We also show the versatility of RCS in matching of both LiDAR and Kinect point clouds.

TOLDI: An effective and robust approach for 3D local shape description

Feature description for the 3D local shape in the presence of noise, varying mesh resolutions, clutter and occlusion is a quite challenging task in 3D computer vision. This paper tackles the problem by proposing a new local reference frame (LRF) together with a novel triple orthogonal local depth images (TOLDI) representation, forming the TOLDI method for local shape description. Compared with previous methods, TOLDI manages to perform efficient, distinctive and robust description for the 3D local surface simultaneously under various feature matching contexts. The proposed LRF differs from many prior ones in its calculation of the z-axis and x-axis, the z-axis is calculated using the normal of the keypoint and the x-axis is computed by aggregating the weighted projection vectors of the radius neighbors. TOLDI feature descriptors are then obtained by concatenating three local depth images (LDI) captured from three orthogonal view planes in the LRF into feature vectors. The performance of our TOLDI approach is rigorously evaluated on several public datasets, which contain three major surface matching scenarios, namely shape retrieval, object recognition and 3D registration. Experimental results and comparisons with the state-of-the-arts validate the effectiveness, robustness, high efficiency, and overall superiority of our method. Our method is also applied to aligning 3D object and indoor scene point clouds obtained by different devices (i.e., LiDAR and Kinect), the accurate outcomes further confirm the effectiveness of our method.