Real Point Cloud Research Articles

Lift path planning for tower cranes based on environmental point clouds

The planning of lift paths is a critical task in crane operations. Traditionally, crane operators and assistants perform operation tasks based on their observation and experience, this is a tedious and error-prone process. Previous studies mainly focus on the optimization of path length or planning time, but seldom consider the accessibility of information for planning and the execution ability of crane operators. As a result, existing solutions may not be suitable for lifting practices due to the ideal validation environment. To address the gap, this research proposes an automated lift path planning method for tower cranes based on environmental point clouds. The method utilizes an octree-based sampling strategy to generate a roadmap from point clouds in the configuration space and a novel collision checking method for point clouds. Furthermore, an optimized A* algorithm is employed to identify executable paths. The proposed method is validated using real point clouds from a construction site. The result shows that the method can automatically generate more executable paths with higher efficiency only using point cloud data. Specifically, the proposed method reduces searching time, path length and the number of inflection points by 57%, 50% and 40% respectively, when compared to the RRT algorithm.

Cascade Graph Neural Networks for Few-Shot Learning on Point Clouds

Point cloud data, a flexible 3D object representation, is critical for various applications such as autonomous driving, robotics and remote sensing. Despite the recent success of deep neural networks (DNNs) on supervised point cloud analysis tasks, they still rely on tedious manual annotation of point clouds and cannot make predictions for new classes. Unlike few-shot learning for 2D images with the advantages of large-scale datasets and high-quality deep pre-trained models like ResNet, for 3D few-shot learning, obtaining discriminative representations of unseen classes with high intra-class similarity and inter-class difference is very challenging. To address this issue, this work proposes a novel cascade graph neural network for few-shot learning on point clouds, termed as CGNN, in which two cascade GNNs are adopted to extract the intra-object topological information and learn the inter-object relations respectively. To further increase the discriminability of point cloud features, we first design a novel discriminative edge label to model the intra-class similarity and inter-class dissimilarity based on channel-wise feature variance and class consistency. Second, we propose a novel few-shot circle loss which classifies the nodes into two subsets, i.e., support to support pairs and support to query pairs, and optimizes the pair-wise similarity on two subsets independently. Extensive experiments on benchmark CAD and real LiDAR point cloud datasets have demonstrated that CGNN improves accuracy by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.98\%$</tex-math> </inline-formula> over the state-of-the-art GNN-based few-shot classification methods.

UPU-SNet: Siamese network for unsupervised point cloud upsampling based on spatial-aware transformers

In the majority of cases, the paired dataset is not easy to obtain, which makes it difficult to train a supervised point cloud upsampling network directly. Therefore, we propose a novel siamese network architecture for the unsupervised upsampling task, called UPU-SNet. This architecture contains two branches based on hierarchical spatial-aware transformers. Specifically, one branch includes a global-local transformer (GLT) module and a standard reconstruction module for coarse point generation. The other contains a shared GLT module, a transformer shuffle (TranShuffle) module and a shared reconstruction module for dense point generation. For multi-level feature extraction, the GLT module we propose aims to explore feature fusions of global-to-local and local-to-global, respectively, with the aid of the multi-head cross-attention mechanism of the transformers. Then, the TranShuffle module is constructed closely after the GLT module to further optimize and expand the extracted features. Moreover, taking full account of sparse points, coarse points and dense points, we design a joint loss function, so as to support the network to generate denser and more uniform point clouds without using ground-truth. Through extensive experiments, we show how our proposal not only outperforms existing unsupervised methods but also achieves competitive results against previous supervised and self-supervised methods. Also, we present various surface reconstruction results on both synthetic and real point clouds, and show our network can enable accurate 3D reconstructions.

From CAD Models to Soft Point Cloud Labels: An Automatic Annotation Pipeline for Cheaply Supervised 3D Semantic Segmentation

We propose a fully automatic annotation scheme that takes a raw 3D point cloud with a set of fitted CAD models as input and outputs convincing point-wise labels that can be used as cheap training data for point cloud segmentation. Compared with manual annotations, we show that our automatic labels are accurate while drastically reducing the annotation time and eliminating the need for manual intervention or dataset-specific parameters. Our labeling pipeline outputs semantic classes and soft point-wise object scores, which can either be binarized into standard one-hot-encoded labels, thresholded into weak labels with ambiguous points left unlabeled, or used directly as soft labels during training. We evaluate the label quality and segmentation performance of PointNet++ on a dataset of real industrial point clouds and Scan2CAD, a public dataset of indoor scenes. Our results indicate that reducing supervision in areas that are more difficult to label automatically is beneficial compared with the conventional approach of naively assigning a hard “best guess” label to every point.

Automatic generation of as-is BEM models of buildings

This paper shows a method with which to create a proprietary as-is 3D semantic thermal model of a building and transform it into standard BEM formats. Geometric and relational information regarding the essential structural elements of the building (i.e., zone, ceiling, floor, wall, door, etc.) is, together with their corresponding as-is temperatures, first extracted from thermal points clouds of the scene and then adapted to the standard geometric BEM requirements. Non-geometric properties concerning definitions and properties of envelopes, windows and doors are also defined as essential information. After all this information has been collected, a BEM file is eventually generated in order to be used in energy simulation software. One of the specific features of this work is the generation of as-is BIM formats of the interior of a building that are later adapted for energy simulation applications in EnergyPlus and TRNSYS software. An ASCII file containing the temperatures of the main structural elements of the building is also created. The method has been tested on real thermal point clouds of the ground floor of a building, yielding promising results.

An Automatic Hierarchical Clustering Method for the LiDAR Point Cloud Segmentation of Buildings via Shape Classification and Outliers Reassignment

Shape segmentation in urban environments forms the foundation for tasks such as classification and reconstruction. Most artificial buildings with complex structures are composed of multiple simple geometric primitives. Based on this assumption, this paper proposes a divisive hierarchical clustering algorithm that uses shape classification and outliers reassignment to segment LiDAR point clouds in order to effectively identify the various shapes of structures that make up buildings. The proposed method adopts a coarse-to-fine strategy. Firstly, based on the geometric properties of different primitives in a Gaussian sphere space, coarse extraction is performed using Gaussian mapping and the DBSCAN algorithm to identify the primary structure of various shapes. Then, the error functions are constructed after parameterizing the recognized shapes. Finally, a minimum energy loss function is built by combining the error functions and binary integer programming (BIP) to redistribute the outlier points. Thereby, the accurate extraction of geometric primitives is achieved. Experimental evaluations on real point cloud datasets show that the indicators of precision, accuracy, and F1 score of our method are 0.98, 0.95, and 0.96 (point assignment) and 0.97, 0.95, and 0.95 (shape recognition), respectively. Compared with other state-of-the-art methods, the proposed method can efficiently segment planar and non-planar structures with higher quality from building point clouds.

Generating synthetic laser scanning data of forests by combining forest inventory information, a tree point cloud database and an open-source laser scanning simulator

Abstract Airborne laser scanning (ALS) data are routinely used to estimate and map structure-related forest inventory variables. The further development, refinement and evaluation of methods to derive forest inventory variables from ALS data require extensive datasets of forest stand information on an individual tree-level and corresponding ALS data. A cost-efficient method to obtain such datasets is the combination of virtual forest stands with a laser scanning simulator. We present an approach to simulate ALS data of forest stands by combining forest inventory information, a tree point cloud database and the laser scanning simulation framework HELIOS++. ALS data of six 1-ha plots were simulated and compared to real ALS data of these plots. The synthetic 3D representations of the forest stands were composed of real laser scanning point clouds of individual trees that were acquired by an uncrewed aerial vehicle (UAV), and, for comparison, simplified tree models with cylindrical stems and spheroidal crowns. The simulated ALS point clouds of the six plots were compared with the real point clouds based on canopy cover, height distribution of returns and several other point cloud metrics. In addition, the performance of biomass models trained using these synthetic data was evaluated. The comparison revealed that, in general, both the real tree models and the simplified tree models can be used to generate synthetic data. The results differed for the different study sites and depending on whether all returns or only first returns were considered. The measure of canopy cover was better represented by the data of the simplified tree models, whereas the height distribution of the returns was – for most of the study sites – better represented by the real tree model data. Training biomass models with metrics derived from the real tree model data led to an overestimation of biomass, while using metrics of the simplified tree model data resulted in an underestimation of biomass. Still, the accuracy of models trained with simulated data was only slightly lower compared to models trained with real ALS data. Our results suggest that the presented approach can be used to generate ALS data that are sufficiently realistic for many applications. The synthetic data may be used to develop new or refine existing ALS-based forest inventory methods, to systematically investigate the relationship between point cloud metrics and forest inventory variables and to analyse how this relationship is affected by laser scanning acquisition settings and field reference data characteristics.

Automatic quality assessment of terrestrial laser scans

Abstract This work addresses the topic of a quality modelling of terrestrial laser scans, including different quality measures such as precision, systematic deviations in distance measurement and completeness. For this purpose, the term “quality” is first defined in more detail in the field of TLS. A distinction is made between a total of seven categories that affect the quality of the TLS point cloud. The focus in this work lies on the uncertainty modeling of the TLS point clouds especially the distance measurement. It is demonstrated that influences such as the intensity and the incidence angle can lead to systematic deviations in the distance measurement of more than 1 mm. Based on these findings, it is presented that systematic deviations in distance measurement can be divided into four classes using machine learning classification approaches. The predicted classes can be useful for deformation analysis or for processing steps like registration. At the end of this work the entire quality assessment process is demonstrated using a real TLS point cloud (40 million points).

MSG-Point-GAN: Multi-Scale Gradient Point GAN for Point Cloud Generation

The generative adversarial network (GAN) has recently emerged as a promising generative model. Its application in the image field has been extensive, but there has been little research concerning point clouds.The combination of a GAN and a graph convolutional network has been the state-of-the-art method for generating point clouds. However, there is a significant gap between the generated point cloud and the point cloud used for training. In order to improve the quality of the generated point cloud, this study proposed multi-scale gradient point GAN (MSG-Point-GAN). The training of the GAN is a dynamic game process, and we expected the generation and discrimination capabilities to be symmetric, so that the network training would be more stable. Based on the concept of progressive growth, this method used the network structure of a multi-scale gradient GAN (MSG-GAN) to stabilize the training process. The discriminator of this method used part of the PointNet structure to resolve the problem of the disorder and rotation of the point cloud. The discriminator could effectively determine the authenticity of the generated point cloud. This study also analyzed the optimization process of the objective function of the MSG-Point-GAN. The experimental results showed that the training process of the MSG-Point-GAN was stable, and the point cloud quality was superior to other methods in subjective vision. From the perspective of performance metrics, the gap between the point cloud generated by the proposed method and the real point cloud was significantly smaller than that generated by other methods. Based on the practical analysis, the point cloud generated by the proposed method for training the point-cloud classification network was improved by about 0.2%, as compared to the original network. The proposed method provided a stable training framework for point cloud generation. It can effectively promote the development of point-cloud-generation technology.

Large-scale 3D object segmentation employing conditional angular clustering technique

This paper aims to develop an automatic 3D object segmentation method for the large-scale point clouds. Given a range image, the preprocessing is first applied to get the optimal 3D point cloud. A [Formula: see text]-nearest neighbor is built, and a segmentation algorithm based on the conditional angular clustering technique is used to segment the objects from the point cloud. The algorithm is tested on the real point cloud datasets. The experiment results demonstrated that the developed segmentation method can be used to localize the object with the relative uncertainty of 0.27%.

IPC-Net: Incomplete point cloud classification network based on data augmentation and similarity measurement

Existing point cloud classification researches are usually conducted on datasets with complete structure and clear semantics. However, in real point cloud scenes, the occlusion and truncation may destroy the completeness of objects affecting the classification performance. To solve this problem, we propose an incomplete point cloud classification network (IPC-Net) with data augmentation and similarity measurement. The proposed network learns the feature representation of incomplete point clouds and the semantic differences compared to the complete ones for classification. Specifically, IPC-Net adopts a random erasing-based data augmentation to deal with incomplete point clouds. IPC-Net also introduces an auxiliary loss function weighted by attention scores to measure the similarity between the incomplete and the complete point clouds. Extensive experiments verify that IPC-Net has the ability to classify incomplete point clouds and significantly improves the robustness of point cloud classification under different completeness.

3D full-field fringe projection measurement based on imaging view transformation.

To address the loss of measurement information caused by limitation of the field of view in fringe projection profilometry, a 3D full-field measurement based on imaging view transformation with a mirror auxiliary is proposed in this work. In this system, the principle of ideal specular reflection is applied to create a symmetrical structure between the real and virtual images for subsequent imaging view transformation. The phase of images is obtained by a multifrequency and multiple phase-shift sinusoidal fringe projection algorithm, and the real and virtual point clouds of measured objects are obtained by system calibration. The specular reflection matrix is obtained by calculating the relationship between virtual and real corner points. As a result, incomplete real point clouds can be compensated with virtual point clouds for the 3D full-field measurement by imaging view transformation. Compared to the standard ladder, the mean absolute error and the maximum relative error are 0.0706mm and 0.9834%, respectively.

Three-dimensional object detection network based on geometric information supplement strategy

Three-dimensional object detection from point clouds greatly advances autonomous driving. The real point clouds are unbalanced, where some occluded or distant objects with fewer points suffer from insufficient features, resulting in limited detection accuracy. To address this issue, we propose a novel three-dimensional object detection network based on a Geometric Information Supplement (GIS) strategy. Concretely, we design a novel shape completion model and a point attention fusion module to implement the GIS strategy. The shape completion model based on a structure-aware Transformer first recovers the complete shape points of objects within proposals. Then, the point attention fusion module supplies the recovered geometric shape cues to the original spatial features of objects. By the GIS strategy, the network learns enriched geometric features of objects for proposal refinement, thus, generating accurate detection boxes. Comparative experiments on KITTI dataset show that our method significantly outperforms other state-of-the-art methods while maintaining modest model complexity.

Multi-Dimensional Graph Interactional Network for Progressive Point Cloud Completion

Point cloud completion refers to inferring the complete and visually plausible shape from a partial input. Existing point cloud completion methods focus on recovering the global integrity of partial point clouds but lack local structural details. Furthermore, they seldom consider the shape faithfulness of completed results, that some completed points fail to fall into the ground-truth position faithfully. To meet the above challenges, we present a multidimensional graph interactional network for progressive point cloud completion. Specifically, we propose a multiresolution multidimensional graph encoder (MRMD-GE) to capture the information from both within-dimension and cross-dimension interactions for the purpose of enhancing the perception of local geometry. Inspired by the FPN, we develop a recursive point cloud pyramid decoder (RPPD) for generating multistage completed point clouds progressively, which incorporates extra feedback connections into the bottom-up backbone layers. In addition, we design a depth map discriminator combined with differentiable rendering to match the distribution of generated and real point clouds, making the completed point clouds more faithful to the ground truth. Quantitative and qualitative experiments on Completion3D, Shapenet-Part, and KITTI datasets demonstrate that our proposed method has compelling advantages over the state-of-the-art methods.

Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network

Multispectral LiDAR technology can simultaneously acquire spatial geometric data and multispectral wavelength intensity information, which can provide richer attribute features for semantic segmentation of point cloud scenes. However, due to the disordered distribution and huge number of point clouds, it is still a challenging task to accomplish fine-grained semantic segmentation of point clouds from large-scale multispectral LiDAR data. To deal with this situation, we propose a deep learning network that can leverage contextual semantic information to complete the semantic segmentation of large-scale point clouds. In our network, we work on fusing local geometry and feature content based on 3D spatial geometric associativity and embed it into a backbone network. In addition, to cope with the problem of redundant point cloud feature distribution found in the experiment, we designed a data preprocessing with principal component extraction to improve the processing capability of the proposed network on the applied multispectral LiDAR data. Finally, we conduct a series of comparative experiments using multispectral LiDAR point clouds of real land cover in order to objectively evaluate the performance of the proposed method compared with other advanced methods. With the obtained results, we confirm that the proposed method achieves satisfactory results in real point cloud semantic segmentation. Moreover, the quantitative evaluation metrics show that it reaches state-of-the-art.

Edge-Cooperative Privacy-Preserving Object Detection Over Random Point Cloud Shares for Connected Autonomous Vehicles

Connected autonomous vehicles (CAVs) employ the point cloud data captured by LiDAR to enhance the capability of object recognition and detection. Edge computing with its inherent advantages can help CAVs alleviate resource constraints and enable faster situational awareness and data processing. However, the point cloud data contains private information, such as vehicle identity, location and trajectory, directly uploading the raw point cloud to the edge nodes or other vehicle will lead to serious privacy leakage. To the best of our knowledge, we are the first to try to tackle this challenge and propose a privacy-preserving object detection framework over random point cloud shares for CAVs (referred to SecPCV), aiming to guarantee the privacy of both point cloud and object detection results. In SecPCV, CAVs split point cloud into two random shares based on additive secret sharing (ASS) and upload them to two competing edge nodes, respectively, which greatly compress the computational load of CAVs. Without changing the object detection network in plaintext environment, the edge nodes can cooperatively and securely extract, regress, and classify over point cloud shares. Theoretical analysis ensure the efficiency and security of the SecPCV framework. Experimental results with the real KITTI point cloud dataset indicate that SecPCV can achieve the consistent object detection accuracy as that in plaintext environment, and provide a feasible solution for CAVs secure sharing of point cloud data.

Missing Detection of The Clutch-Driven Disc Spring Assembly

A detection system for the missing assembly of clutch-driven plate springs is established. The algorithms used in this system, such as point cloud data preprocessing, spring area positioning, and spring missing detection, are studied. Firstly, according to the characteristics of the clutch-driven plate spring point cloud, the Gaussian statistical method is used to eliminate the outliers incurred by system error and other reasons, and the two-sided filtering is performed on the clutch-driven plate spring point cloud that eliminates the outliers to eliminate the noise points with small ups and downs. Then, the theoretical division area is constructed by using the theoretical width between the clutch-driven plate springs and the theoretical width of the springs. Compared with actually recorded positions, the theoretical area is expanded so that the real spring point cloud data are in the corresponding positioning area; finally, the spring in the region is identified by using the characteristics of an inconsistent number of point clouds in different assembly conditions, so as to detect the missing assembly of the clutch-driven plate spring. The experimental results show that the detection accuracy of the clutch-driven plate spring assembly is as high as 100%. It can basically meet the detection requirements for the missing assembly of clutch-driven plate springs.

Mitigating Shadows in LIDAR Scan Matching Using Spherical Voxels

In this paper we propose an approach to mitigate shadowing errors in Lidar scan matching, by introducing a preprocessing step based on spherical gridding. Because the grid aligns with the Lidar beam, it is relatively easy to eliminate shadow edges which cause systematic errors in Lidar scan matching. As we show through simulation, our proposed algorithm provides better results than ground-plane removal, the most common existing strategy for shadow mitigation. Unlike ground plane removal, our method applies to arbitrary terrains (e.g. shadows on urban walls, shadows in hilly terrain) while retaining key Lidar points on the ground that are critical for estimating changes in height, pitch, and roll. Our preprocessing algorithm can be used with a range of scan-matching methods; however, for voxel-based scan matching methods, it provides additional benefits by reducing computation costs and more evenly distributing Lidar points among voxels.

Enhanced GPIS Learning Based on Local and Global Focus Areas

Implicit surface learning is one of the most widely used methods for 3D surface reconstruction from raw point cloud data. Current approaches employ deep neural networks or Gaussian process models with the trade-offs across computational performance, object fidelity, and generalization capabilities. We propose a novel method based on Gaussian process regression to build implicit surfaces for 3D surface reconstruction (GPIS), which leads to better accuracy in comparison to the standard GPIS formulation. Our approach encodes local and global shape information from the data to maintain the correct topology of the underlying shape. The proposed pipeline works on dense, sparse, and noisy raw point clouds and can be parallelized to improve computational efficiency. We evaluate our approach on synthetic and real point cloud datasets including data from robot visual and tactile sensors. Results show that our approach leads to high accuracy compared to baselines.

Point2Roof: End-to-end 3D building roof modeling from airborne LiDAR point clouds

Three-dimensional (3D) building roof reconstruction from airborne LiDAR point clouds is an important task in photogrammetry and computer vision. To automatically reconstruct the 3D building models at Level of Detail 2 (LoD-2) from airborne LiDAR point clouds, the data-driven approaches usually need to be performed in two steps: geometric primitive extraction and roof structure inference. Obviously, the traditional approaches are not end-to-end, the accumulated errors in different stages cannot be avoided and the final 3D roof models may not be optimal. In addition, the results of 3D roof models largely depend on the accuracy of geometric primitives (planes, lines, etc.). To solve these problems, we present a deep learning-based approach to directly reconstruct building roofs from airborne LiDAR point clouds, named Point2Roof. In our method, we start by extracting the deep features for each input point using PointNet++. Then, we identify a set of candidate corner points from the input point clouds using the extracted deep features. In addition, we also regress the offset for each candidate corner point to refine their locations. After that, these candidates are clustered into a set of initial vertices, and we further refine their locations to obtain the final accurate vertices. Finally, we propose a Paired Point Attention (PPA) module to predict the true model edges from an exhaustive set of candidate edges between the vertices. Unlike traditional roof modeling approaches, the proposed Point2Roof is end-to-end. However, due to the lack of a building reconstruction dataset, we construct a large-scale synthetic dataset to verify the effectiveness and robustness of the proposed Point2Roof. The experimental results conducted on the synthetic benchmark demonstrate that the proposed Point2Roof significantly outperforms the traditional roof modeling approaches. The experiments also show that the network trained on the synthetic dataset can be applied to the real point clouds after fine-tuning the trained model on a small real dataset. The large-scale synthetic dataset, the small real dataset and the source code of our approach are publicly available in https://github.com/Li-Li-Whu/Point2Roof.