Input Point Cloud Research Articles

A local tangent plane distance-based approach to 3D point cloud segmentation via clustering

This paper proposes an effective measure for the planar segmentation problem based on the clustering method. It uses the distance from a point to the local plane as a metric to characterize the relationship between data. As a result, the data points of the coplanar have a high similarity to distinguish each plane. A dissimilarity matrix of the input point cloud can be evaluated, and multidimensional scaling analysis is performed to reconstruct the correlation information between data points in the 3D Euclidean space. The obtained reconstructed point cloud shows the separation between different planes. An adaptive DBSCAN clustering method based on density stratification is developed to perform cluster segmentation on the reconstructed point cloud. Experimental results show that the proposed method can effectively solve the over-segmentation problem, and at the same time provide high segmentation accuracy.

Water Bottom and Surface Classification Algorithm for Bathymetric LiDAR Point Clouds of Very Shallow Waters

The absence of accurate point classification limits the effective use of airborne bathymetric LiDAR (ABL) data for coastal zone mapping. In this study, we propose a classification approach using a custom waveform decomposition technique with the pseudo-waveform generated from ABL point cloud data. Initially, the input point clouds were organized into a 2D grid. Next, the points that fall into a grid cell were organized into a histogram using Z-values to generate the pseudo-waveform. Subsequently, the pseudo-waveform was decomposed into water bottom, column, surface, and noise components using a custom multiple Gaussian curve fitting method. The proposed approach was evaluated with datasets acquired in Florida, USA, using a Riegl VQ-880-G ABL system. With an optimized parameter set, the proposed approach achieved F1 score of 98.944% for the classification of water bottom and an overall accuracy of 91.234% for all the classes. Further, the proposed approach was evaluated with datasets acquired in South Korea using a Seahawk system and compared against MBES data, demonstrating that the water bottom was successfully classified with a vertical error of 0.049 ± 0.167 m.

DeepEgo: Deep Instantaneous Ego-Motion Estimation Using Automotive Radar

The problem of instantaneous ego-motion estimation with mm-wave automotive radar is studied. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepEgo</i> , a deep learning-based method, is proposed for achieving robust and accurate ego-motion estimation. A hybrid approach that uses neural networks to extract complex features from input point clouds and applies weighted least squares (WLS) for motion estimation is utilized in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepEgo</i> . Additionally, a novel loss function, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Doppler loss</i> , is proposed to locate ‘inlier points’ originating from detected stationary objects without human annotation. Finally, a challenging real-world automotive radar dataset is selected for extensive performance evaluation. Compared to other methods selected from the literature, significant improvements in estimation accuracy, long-term stability, and runtime performance of DeepEgo in comparison to other methods are demonstrated.

The Art of Defense: Letting Networks Fool the Attacker

3D perception of objects is critical for many real-world applications, such as autonomous cars and robots. Among them, most state-of-the-art (SOTA) 3D perception systems are based on deep learning models. Recently, the research community found that 3D object classifiers on point cloud based on deep learning are easily fooled by adversarial point cloud craft by attackers. To overcome this, adversarial defenses are considered the most effective ways to improve the robustness of deep learning models, and most adversarial defenses on point cloud are focused on input transformation. However, all previous defense methods decrease the natural accuracy, and the nature of the point cloud classifiers itself has been overlooked. To this end, in this paper, we propose a novel adversarial defense for 3D point cloud classifiers that makes full use of the nature of the point cloud classifiers. Due to the disorder of point cloud, all point cloud classifiers have the property of permutation invariant to the input point cloud. Based on this nature, we design invariant transformations defense (IT-Defense). We show that, even after accounting for obfuscated gradients, our IT-Defense is a resilient defense against SOTA 3D attacks. Moreover, IT-Defense does not hurt clean accuracy compared to previous SOTA 3D defenses. Our code will be available at: https://github.com/cuge1995/IT-Defense.

Hellinger–Kantorovich barycenter between Dirac measures

The Hellinger-Kantorovich (HK) distance is an unbalanced extension of the Wasserstein-2 distance. It was shown recently that the HK barycenter exhibits a much more complex behaviour than the Wasserstein barycenter. Motivated by this observation we study the HK barycenter in more detail for the case where the input measures are an uncountable collection of Dirac measures, in particular the dependency on the length scale parameter of HK, the question whether the HK barycenter is discrete or continuous and the relation between the expected and the empirical barycenter. The analytical results are complemented with numerical experiments that demonstrate that the HK barycenter can provide a coarse-to-fine representation of an input pointcloud or measure.

Up-Sampling Method for Low-Resolution LiDAR Point Cloud to Enhance 3D Object Detection in an Autonomous Driving Environment.

Automobile datasets for 3D object detection are typically obtained using expensive high-resolution rotating LiDAR with 64 or more channels (Chs). However, the research budget may be limited such that only a low-resolution LiDAR of 32-Ch or lower can be used. The lower the resolution of the point cloud, the lower the detection accuracy. This study proposes a simple and effective method to up-sample low-resolution point cloud input that enhances the 3D object detection output by reconstructing objects in the sparse point cloud data to produce more dense data. First, the 3D point cloud dataset is converted into a 2D range image with four channels: x, y, z, and intensity. The interpolation on the empty space is calculated based on both the pixel distance and range values of six neighbor points to conserve the shapes of the original object during the reconstruction process. This method solves the over-smoothing problem faced by the conventional interpolation methods, and improves the operational speed and object detection performance when compared to the recent deep-learning-based super-resolution methods. Furthermore, the effectiveness of the up-sampling method on the 3D detection was validated by applying it to baseline 32-Ch point cloud data, which were then selected as the input to a point-pillar detection model. The 3D object detection result on the KITTI dataset demonstrates that the proposed method could increase the mAP (mean average precision) of pedestrians, cyclists, and cars by 9.2%p, 6.3%p, and 5.9%p, respectively, when compared to the baseline of the low-resolution 32-Ch LiDAR input. In future works, various dataset environments apart from autonomous driving will be analyzed.

Estimation of lidar-based gridded DEM uncertainty with varying terrain roughness and point density

Light detection and ranging (lidar) scanning systems can be used to provide a point cloud with high quality and point density. Gridded digital elevation models (DEMs) interpolated from laser scanning point clouds are widely used due to their convenience, however, DEM uncertainty is rarely provided. This paper proposes an end-to-end workflow to quantify the uncertainty (i.e., standard deviation) of a gridded lidar-derived DEM. A benefit of the proposed approach is that it does not require independent validation data measured by alternative means. The input point cloud requires per point uncertainty which is derived from lidar system observational uncertainty. The propagated uncertainty caused by interpolation is then derived by the general law of propagation of variances (GLOPOV) with simultaneous consideration of both horizontal and vertical point cloud uncertainties. Finally, the interpolated uncertainty is then scaled by point density and a measure of terrain roughness to arrive at the final gridded DEM uncertainty. The proposed approach is tested with two lidar datasets measured in Waikoloa, Hawaii, and Sitka, Alaska. Triangulated irregular network (TIN) interpolation is chosen as the representative gridding approach. The results indicate estimated terrain roughness/point density scale factors ranging between 1 (in flat areas) and 7.6 (in high roughness areas), with a mean value of 2.3 for the Waikoloa dataset and between 1 and 9.2 with a mean value of 1.2 for the Sitka dataset. As a result, the final gridded DEM uncertainties are estimated between 0.059 m and 0.677 m with a mean value of 0.164 m for the Waikoloa dataset and between 0.059 m and 1.723 m with a mean value of 0.097 m for the Sitka dataset.

UPSNet: Universal Point Cloud Sampling Network Without Knowing Downstream Tasks

With the development of three-dimensional sensing technology, the data volume of point cloud grows rapidly. Therefore, point cloud is usually down-sampled in advance so as to save memory space and reduce the computational complexity for its downstream processing tasks such as classification, segmentation, reconstruction in learning based point cloud processing. Obviously, the sampled point clouds should be well representative and maintain the geometric structure of the original point clouds so that the downstream tasks can achieve satisfied performance based on the point clouds sampled from the original ones. Traditional point cloud sampling methods such as farthest point sampling and random sampling mainly heuristically select a subset of the original point cloud. However, they do not make full use of high-level semantic representation of point clouds, are sensitive to outliers. Some of other sampling methods are task oriented. In this paper, a Universal Point cloud Sampling Network without knowing downstream tasks (denoted as UPSNet) is proposed. It consists of three modules. The importance learning module is responsible for learning the mutual information between the points of input point cloud and calculating a group of variational importance probabilities to represent the importance of each point in the input point cloud, based on which a mask is designed to discard the points with lower importance so that the number of remaining points is controlled. Then, the regional learning module learns from the input point cloud to get the high dimensional space embedding of each region, and the global feature of each region are obtained by weighting the high dimensional space embedding with the variational importance probability. Finally, through the coordinate regression module, the global feature and the high dimensional space embedding of each region are cascaded for learning to obtain the sampled point cloud. A series of experiments are implemented in which the point cloud classification, segmentation, reconstruction and retrieval are performed on the reconstructed point clouds sampled with different point cloud sampling methods. The experimental results show that the proposed UPSNet can provide more reasonable sampling result of the input point cloud for the downstream tasks of classification, segmentation, reconstruction and retrieval, and is superior to the existing sampling methods without knowing the downstream tasks. The proposed UPSNet is not oriented to specific downstream tasks, so it has wide applicability.

Learning a Task-Specific Descriptor for Robust Matching of 3D Point Clouds

Existing learning-based point feature descriptors are usually task-agnostic, which pursue describing the individual 3D point clouds as accurate as possible. However, the matching task aims at describing the corresponding points consistently across different 3D point clouds. Therefore these too accurate features may play a counterproductive role due to the inconsistent point feature representations of correspondences caused by the unpredictable noise, partiality, deformation, \etc, in the local geometry. In this paper, we propose to learn a robust task-specific feature descriptor to consistently describe the correct point correspondence under interference. Born with an Encoder and a Dynamic Fusion module, our method EDFNet develops from two aspects. First, we augment the matchability of correspondences by utilizing their repetitive local structure. To this end, a special encoder is designed to exploit two input point clouds jointly for each point descriptor. It not only captures the local geometry of each point in the current point cloud by convolution, but also exploits the repetitive structure from paired point cloud by Transformer. Second, we propose a dynamical fusion module to jointly use different scale features. There is an inevitable struggle between robustness and discriminativeness of the single scale feature. Specifically, the small scale feature is robust since little interference exists in this small receptive field. But it is not sufficiently discriminative as there are many repetitive local structures within a point cloud. Thus the resultant descriptors will lead to many incorrect matches. In contrast, the large scale feature is more discriminative by integrating more neighborhood information. ...

Point Cloud Instance Segmentation With Semi-Supervised Bounding-Box Mining.

Point cloud instance segmentation has achieved huge progress with the emergence of deep learning. However, these methods are usually data-hungry with expensive and time-consuming dense point cloud annotations. To alleviate the annotation cost, unlabeled or weakly labeled data is still less explored in the task. In this paper, we introduce the first semi-supervised point cloud instance segmentation framework (SPIB) using both labeled and unlabelled bounding boxes as supervision. To be specific, our SPIB architecture involves a two-stage learning procedure. For stage one, a bounding box proposal generation network is trained under a semi-supervised setting with perturbation consistency regularization (SPCR). The regularization works by enforcing an invariance of the bounding box predictions over different perturbations applied to the input point clouds, to provide self-supervision for network learning. For stage two, the bounding box proposals with SPCR are grouped into some subsets, and the instance masks are mined inside each subset with a novel semantic propagation module and a property consistency graph module. Moreover, we introduce a novel occupancy ratio guided refinement module to refine the instance masks. Extensive experiments on the challenging ScanNet v2 dataset demonstrate our method can achieve competitive performance compared with the recent fully-supervised methods.

A Neural Galerkin Solver for Accurate Surface Reconstruction

To reconstruct meshes from the widely-available 3D point cloud data, implicit shape representation is among the primary choices as an intermediate form due to its superior representation power and robustness in topological optimizations. Although different parameterizations of the implicit fields have been explored to model the underlying geometry, there is no explicit mechanism to ensure the fitting tightness of the surface to the input. We present in response, NeuralGalerkin, a neural Galerkin-method-based solver designed for reconstructing highly-accurate surfaces from the input point clouds. NeuralGalerkin internally discretizes the target implicit field as a linear combination of a set of spatially-varying basis functions inferred by an adaptive sparse convolution neural network. It then solves differentiably for a variational problem that incorporates both positional and normal constraints from the data in closed form within a single forward pass, highly respecting the raw input points. The reconstructed surface extracted from the implicit interpolants is hence very accurate and incorporates useful inductive biases benefiting from the training data. Extensive evaluations on various datasets demonstrate our method's promising reconstruction performance and scalability.

RFEPS

Feature lines are important geometric cues in characterizing the structure of a CAD model. Despite great progress in both explicit reconstruction and implicit reconstruction, it remains a challenging task to reconstruct a polygonal surface equipped with feature lines, especially when the input point cloud is noisy and lacks faithful normal vectors. In this paper, we develop a multistage algorithm, named RFEPS , to address this challenge. The key steps include (1) denoising the point cloud based on the assumption of local planarity, (2) identifying the feature-line zone by optimization of discrete optimal transport, (3) augmenting the point set so that sufficiently many additional points are generated on potential geometry edges, and (4) generating a polygonal surface that interpolates the augmented point set based on restricted power diagram. We demonstrate through extensive experiments that RFEPS, benefiting from the edge-point augmentation and the feature preserving explicit reconstruction, outperforms state of the art methods in terms of the reconstruction quality, especially in terms of the ability to reconstruct missing feature lines.

Point Cloud Geometry Compression Based on Multi-Layer Residual Structure.

Point cloud data are extensively used in various applications, such as autonomous driving and augmented reality since it can provide both detailed and realistic depictions of 3D scenes or objects. Meanwhile, 3D point clouds generally occupy a large amount of storage space that is a big burden for efficient communication. However, it is difficult to efficiently compress such sparse, disordered, non-uniform and high dimensional data. Therefore, this work proposes a novel deep-learning framework for point cloud geometric compression based on an autoencoder architecture. Specifically, a multi-layer residual module is designed on a sparse convolution-based autoencoders that progressively down-samples the input point clouds and reconstructs the point clouds in a hierarchically way. It effectively constrains the accuracy of the sampling process at the encoder side, which significantly preserves the feature information with a decrease in the data volume. Compared with the state-of-the-art geometry-based point cloud compression (G-PCC) schemes, our approach obtains more than 70-90% BD-Rate gain on an object point cloud dataset and achieves a better point cloud reconstruction quality. Additionally, compared to the state-of-the-art PCGCv2, we achieve an average gain of about 10% in BD-Rate.

Local offset point cloud transformer based implicit surface reconstruction

AbstractImplicit neural representations, such as MLP, can well recover the topology of watertight object. However, MLP fails to recover geometric details of watertight object and complicated topology due to dealing with point cloud in a point‐wise manner. In this paper, we propose a point cloud transformer called local offset point cloud transformer (LOPCT) as a feature fusion module. Before using MLP to learn the implicit function, the input point cloud is first fed into the local offset transformer, which adaptively learns the dependency of the local point cloud and obtains the enhanced features of each point. The feature‐enhanced point cloud is then fed into the MLP to recover the geometric details and sharp features of watertight object and complex topology. Extensive reconstruction experiments of watertight object and complex topology demonstrate that our method achieves comparable or better results than others in terms of recovering sharp features and geometric details. In addition, experiments on watertight objects demonstrate the robustness of our method in terms of average result.

End-to-End Learning to Grasp via Sampling From Object Point Clouds

The ability to grasp objects is an essential skill that enables many robotic manipulation tasks. Recent works have studied point cloud-based methods for object grasping by starting from simulated datasets and have shown promising performance in real-world scenarios. Nevertheless, many of them still rely on ad-hoc geometric heuristics to generate grasp candidates, which fail to generalize to objects with significantly different shapes with respect to those observed during training. Several approaches exploit complex multi-stage learning strategies and local neighborhood feature extraction while ignoring semantic global information. Furthermore, they are inefficient in terms of number of training samples and time required for inference. In this letter, we propose an end-to-end learning solution to generate 6-DOF parallel-jaw grasps starting from the 3D partial view of the object. Our Learning to Grasp (L2G) method gathers information from the input point cloud through a new procedure that combines a differentiable sampling strategy to identify the visible contact points, with a feature encoder that leverages local and global cues. Overall, L2G is guided by a multi-task objective that generates a diverse set of grasps by optimizing contact point sampling, grasp regression, and grasp classification. With a thorough experimental analysis, we show the effectiveness of L2G as well as its robustness and generalization abilities.

Voting and Attention-Based Pose Relation Learning for Object Pose Estimation From 3D Point Clouds

Estimating the 6DOF pose of objects is an important function in many applications, such as robot manipulation or augmented reality. However, accurate and fast pose estimation from 3D point clouds is challenging, because of the complexity of object shapes, measurement noise, and presence of occlusions. We address this challenging task using an end-to-end learning approach for object pose estimation given a raw point cloud input. Our architecture pools geometric features together using a self-attention mechanism and adopts a deep Hough voting scheme for pose proposal generation. To build robustness to occlusion, the proposed network generates candidates by casting votes and accumulating evidence for object locations. Specifically, our model learns higher-level features by leveraging the dependency of object parts and object instances, thereby boosting the performance of object pose estimation. Our experiments show that our method outperforms state-of-the-art approaches in public benchmarks including the Siléane dataset [35 and the Fraunhofer IPA dataset [36]. We also deploy our proposed method to a real robot pick-and-place based on the estimated pose.

Point2Roof: End-to-end 3D building roof modeling from airborne LiDAR point clouds

Three-dimensional (3D) building roof reconstruction from airborne LiDAR point clouds is an important task in photogrammetry and computer vision. To automatically reconstruct the 3D building models at Level of Detail 2 (LoD-2) from airborne LiDAR point clouds, the data-driven approaches usually need to be performed in two steps: geometric primitive extraction and roof structure inference. Obviously, the traditional approaches are not end-to-end, the accumulated errors in different stages cannot be avoided and the final 3D roof models may not be optimal. In addition, the results of 3D roof models largely depend on the accuracy of geometric primitives (planes, lines, etc.). To solve these problems, we present a deep learning-based approach to directly reconstruct building roofs from airborne LiDAR point clouds, named Point2Roof. In our method, we start by extracting the deep features for each input point using PointNet++. Then, we identify a set of candidate corner points from the input point clouds using the extracted deep features. In addition, we also regress the offset for each candidate corner point to refine their locations. After that, these candidates are clustered into a set of initial vertices, and we further refine their locations to obtain the final accurate vertices. Finally, we propose a Paired Point Attention (PPA) module to predict the true model edges from an exhaustive set of candidate edges between the vertices. Unlike traditional roof modeling approaches, the proposed Point2Roof is end-to-end. However, due to the lack of a building reconstruction dataset, we construct a large-scale synthetic dataset to verify the effectiveness and robustness of the proposed Point2Roof. The experimental results conducted on the synthetic benchmark demonstrate that the proposed Point2Roof significantly outperforms the traditional roof modeling approaches. The experiments also show that the network trained on the synthetic dataset can be applied to the real point clouds after fine-tuning the trained model on a small real dataset. The large-scale synthetic dataset, the small real dataset and the source code of our approach are publicly available in https://github.com/Li-Li-Whu/Point2Roof.

A Deep Learning Approach for LiDAR Resolution-Agnostic Object Detection

Existing neural network-based object detection approaches process LiDAR point clouds trained from one kind of LiDAR sensor. In the case of a different point cloud input, the trained network performs with less efficiency, especially when the given point cloud has low resolution. In this paper, we propose a new object detection approach, which is more resilient to variations in point cloud resolution. Firstly, layers from the point cloud are randomly discarded during the training phase in order to increase the variability of the data processed by the network. Secondly, the obstacles are described as Gaussian functions, grouping multiple parameters into a single representation. A Bhattacharyya distance is used as a loss function. This approach is tested on a LiDAR-based network and on an architecture using camera and LiDAR sensors. The networks are trained exclusively on the KITTI dataset and tested on Pandaset and the nuScenes Mini dataset. Experiments show that our method improves the performance of the tested networks on low-resolution point clouds without decreasing the ability to process high-resolution data.

Real-Time Scene-Aware LiDAR Point Cloud Compression Using Semantic Prior Representation

Existing LiDAR point cloud compression (PCC) methods tend to treat compression as a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fidelity</i> issue, without sufficiently addressing its <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">machine perception</i> aspect. The latter issue is often encountered by the decoder agents that might aim to conduct scene-understanding related tasks only, such as computing the localization information. For tackling this challenge, a novel LiDAR PCC system is proposed to compress the point cloud geometry, which contains a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">back channel</i> for allowing the decoder to initiate such request to the encoder. The key success of our PCC method lies in our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semantic prior representation</i> (SPR) and its lossy encoding algorithm with variable precision to generate the final bitstream; the entire process is fast and achieves real-time performance. Note that our SPR is a compact and effective representation of three-dimensional (3D) input point clouds, and it consists of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">labels, predictions</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">residuals</i> . These information can be generated by first exploiting a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scene-aware object segmentation</i> to a set of 2D range images (frames) individually, which were generated from the 3D point clouds via a projection process. Based on the generated labels, the pixels associated with those moving objects are considered as noisy information and should be removed for not only saving bit budget on transmission but also, most importantly, improving the accuracy of localization computed at the decoder. Experimental results conducted on the commonly-used test dataset have shown that our proposed system outperforms the MPEG’s G-PCC (TMC13-v14.0) in a large bitrate range. In fact, the performance gap will become even larger when more and/or large moving objects are involved in the input point clouds.

VRNet: Learning the Rectified Virtual Corresponding Points for 3D Point Cloud Registration

3D point cloud registration is fragile to outliers, which are labeled as the points without corresponding points. To handle this problem, a widely adopted strategy is to estimate the relative pose based only on some accurate correspondences, which is achieved by building correspondences on the identified inliers or by selecting reliable ones. However, these approaches are usually complicated and time-consuming. By contrast, the virtual point-based methods learn the virtual corresponding points (VCPs) for all source points uniformly without distinguishing the outliers and the inliers. Although this strategy is time-efficient, the learned VCPs usually exhibit serious collapse degeneration due to insufficient supervision and the inherent distribution limitation. In this paper, we propose to exploit the best of both worlds and present a novel robust 3D point cloud registration framework. We follow the idea of the virtual point-based methods but learn a new type of virtual points called rectified virtual corresponding points (RCPs), which are defined as the point set with the same shape as the source and with the same pose as the target. Hence, a pair of consistent point clouds, i.e. source and RCPs, is formed by rectifying VCPs to RCPs (VRNet), through which reliable correspondences between source and RCPs can be accurately obtained. Since the relative pose between source and RCPs is the same as the relative pose between source and target, the input point clouds can be registered naturally. Specifically, we first construct the initial VCPs by using an estimated soft matching matrix to perform a weighted average on the target points. Then, we design a correction-walk module to learn an offset to rectify VCPs to RCPs, which effectively breaks the distribution limitation of VCPs. Finally, we develop a hybrid loss function to enforce the shape and geometry structure consistency ...