Complete Point Cloud Research Articles

DVGG: Deep Variational Grasp Generation for Dextrous Manipulation

Grasping with anthropomorphic robotic hands involves much more hand-object interactions compared to parallel-jaw grippers. Modeling hand-object interactions is essential to the study of multi-finger hand dextrous manipulation. This work presents DVGG, an efficient grasp generation network that takes single-view observation as input and predicts high-quality grasp configurations for unknown objects. In general, our generative model consists of three components: 1) Point cloud completion for the target object based on the partial observation; 2) Diverse sets of grasps generation given the complete point cloud; 3) Iterative grasp pose refinement for physically plausible grasp optimization. To train our model, we build a large-scale grasping dataset that contains about 300 common object models with 1.5 M annotated grasps in simulation. Experiments in simulation show that our model can predict robust grasp poses with a wide variety and high success rate. Real robot platform experiments demonstrate that the model trained on our dataset performs well in the real world. Remarkably, our method achieves a grasp success rate of 70.7% for novel objects in the real robot platform, which is a significant improvement over the baseline methods.

Temporal Point Cloud Completion With Pose Disturbance

Point clouds collected by real-world sensors are always unaligned and sparse, which makes it hard to reconstruct the complete shape of object from a single frame of data. In this work, we manage to provide complete point clouds from sparse input with pose disturbance by limited translation and rotation. We also use temporal information to enhance the completion model, refining the output with a sequence of inputs. With the help of gated recovery units(GRU) and attention mechanisms as temporal units, we propose a point cloud completion framework that accepts a sequence of unaligned and sparse inputs, and outputs consistent and aligned point clouds. Our network performs in an online manner and presents a refined point cloud for each frame, which enables it to be integrated into any SLAM or reconstruction pipeline. As far as we know, our framework is the first to utilize temporal information and ensure temporal consistency with limited transformation. Through experiments in ShapeNet and KITTI, we prove that our framework is effective in both synthetic and real-world datasets.

Point cloud completion by dynamic transformer with adaptive neighbourhood feature fusion

Point cloud completion aims to reconstruct detailed structures from partial observations. However, previous methods often suffered from inaccurate neighbourhood feature extraction and rough reconstruction of complex structures, which is difficult to complete detailed semantic shapes. To solve this problem, we present a method that applies dynamic transformers with adaptive neighbourhood feature fusion operations to resume complete point clouds. Firstly, we propose an adaptive neighbourhood feature extraction module, which contains a learnable global neighbourhood selection strategy and a traditional local k-nearest neighbour strategy to dynamically select neighbourhood points according to the shape of different objects. Secondly, we observed that traditional point generation methods based on folding-series operations limit their capacity of generating complex and faithful shapes. Inspired by cell division, we regard the process of reconstructing as point splitting and propose a genetic hierarchical point generation module, which means current points can inherit shape features from previous points and generate more detailed structures of their own. Extensive experiments of point cloud completion are carried out on PCN and Completion3D datasets to verify the effectiveness of our method. The average category chamfer distances of our method are 7.17(×10−3) in PCN and 7.96(×10−4) in Completion3D, which has better completion performance than other methods.

DPG-Net: Densely progressive-growing network for point cloud completion

This paper proposes a Densely Progressive-growing Network (DPG-Net) for 3D point cloud completion based on the Generative Adversarial Network (GAN). It aims to learn useful features from incomplete 3D objects to infer the complete geometric shape. Most methods usually acquire global features directly from the incomplete point cloud. However, the global feature often loses local information from the input point cloud. To solve this problem, we design a novel network named DPG-Net for point cloud completion by proposing a multi-resolution dense contextual feature mechanism, a progressive growth process, and combining the adversarial process. Firstly, we study a Multi-resolution Densely Contextual Encoder to infer the potential features with local information of the partial point cloud. The encoder can obtain the global information of the point cloud while fusing the local structure details. Secondly, we propose a Progressive Growth Decoder, which can make full use of global information, to gradually refine local areas and generate a complete shape. In addition, the discriminator is used to control the network for a realistic point cloud. Our model composed of the above modules can extract the features from incomplete point clouds, and then generate detailed complete point clouds. The performance (chamfer distance) of the proposed method is better than other methods on different datasets. The experiment proves the effectiveness of our method on point cloud completion.

Traffic Sign Three‐Dimensional Reconstruction Based on Point Clouds and Panoramic Images

AbstractTraffic signs are a very important source of information for drivers and pilotless automobiles. With the advance of Mobile LiDAR System (MLS), massive point clouds have been applied in three‐dimensional digital city modelling. However, traffic signs in MLS point clouds are low density, colourless and incomplete. This paper presents a new method for the reconstruction of vertical rectangle traffic sign point clouds based on panoramic images. In this method, traffic sign point clouds are extracted based on arc feature and spatial semantic features analysis. Traffic signs in images are detected by colour and shape features and a convolutional neural network. Traffic sign point cloud and images are registered based on outline features. Finally, traffic sign points match traffic sign pixels to reconstruct the traffic sign point cloud. Experimental results have demonstrated that this proposed method can effectively obtain colourful and complete traffic sign point clouds with high resolution.

The “Fuzzy” Repair of Urban Building Facade Point Cloud Based on Distribution Regularity

The integrity of point cloud is the basis for smoothly ensuring subsequent data processing and application. For “Smart City” and “Scan to Building Information Modeling (BIM)”, complete point cloud data is essential. At present, the most commonly used methods for repairing point cloud holes are multi-source data fusion and interpolation. However, these methods either make it difficult to obtain data, or they are ineffective at repairs or labor-intensive. To solve these problems, we proposed a point cloud “fuzzy” repair algorithm based on the distribution regularity of buildings, aiming at the façade of a building in an urban scene, especially for the vehicle Lidar point cloud. First, the point cloud was rotated to be parallel to the plane XOZ, and the feature boundaries of buildings were extracted. These boundaries were further classified as horizontal or vertical. Then, the distance between boundaries was calculated according to the Euclidean distance, and the points were divided into grids based on this distance. Finally, the holes in the grid that needed to be repaired were filled from four adjacent grids by the “copy–paste” method, and the final hole repairs were realized by point cloud smoothing. The quantitative results showed that data integrity improved after the repair and conformed to the state of the building. The angle and position deviation of the repaired grid were less than 0.54° and 3.25 cm, respectively. Compared with human–computer interaction and other methods, our method required less human intervention, and it had high efficiency. This is of promotional significance for the repair and modeling of point cloud in urban buildings.

Extraction of Maize Leaf Base and Inclination Angles Using Terrestrial Laser Scanning (TLS) Data

Leaf base and inclination angles are two critical 3-D structural parameters in agronomy and remote sensing for breeding and modeling. Terrestrial laser scanning (TLS) has been proven to be a promising tool to quantify leaf base and inclination angles. However, previous TLS studies often focused on leaf base and inclination angles of certain trees or plants with flat leaves, such as European beech. Few studies have worked on leaf base and inclination angles of maize plants due to their curved and elongated characteristics. In this study, a machine learning-based [support vector machine (SVM)] method and a structure-based [skeleton extraction (SE)] method were presented to extract the leaf base and inclination angles of maize plants. After separating individual leaf points from the complete point cloud and skeleton points of maize plants and then extracting geometric features, the machine learning- and structure-based methods were used to calculate leaf base and inclination angles. Our results show that the leaf base and inclination angles extracted using these two methods agreed well with ground truth, and the estimation accuracy of the machine learning-based method was obviously higher than that of the structure-based method. The mean absolute error (MAE), root-mean-squared error (RMSE), and relative RMSE (rRMSE) of the leaf base and inclination angles using the machine learning-based method were 4.56°, 6.17°, and 19.04% and 7.95°, 10.00°, and 20.24%, respectively; and those from the structure-based method were 6.22°, 7.47°, and 23.30% and 8.99°, 12.57°, and 25.85%, respectively. The machine learning-based method was also applied to a field with dense mature maize, and their MAE, RMSE, and rRMSE were 6.04°, 8.12°, and 25.90% and 11.30°, 13.52°, and 26.5%, respectively. It is demonstrated that both the machine learning- and structure-based methods are effective to estimate the leaf base and inclination angles of maize plants, although the machine learning-based method appears to outperform the structure-based method.

A Rover Team Based 3D Map Building Using Low Cost 2D Laser Scanners

This paper presents a solution to develop 3D map of indoor places using a team of rovers equipped with low cost 2D laser scanners. An orthogonal combination of two RPLidar 2D laser scanners has been used on every rover to build 3D point cloud map of the explored regions. Each rover of the team has been distantly placed inside the surveying environment in order to independently scan and built the unique 3D point cloud map of the region. The interfacing of scanners and the processing of simultaneous localization and mapping (SLAM) technique for each moving rover have been achieved using Robot Operating System (ROS). The individually build 3D maps have been merged in offline mode through sensor fusion application using Kalman Filter (KF) technique. Different indoor vicinities have been tested and complete 3D point cloud maps have been established which have been found accurate when compared to ground truths. The developed point cloud map has been further utilized to establish Building Information Model (BIM) and found valid for the surveyed region. In comparison to existing surveying and scanning solutions of the regional market, the presented method has been found precise, quick and convenient to provide structural details of surveyed entities at highly affordable rates.

PS-Net: Point Shift Network for 3-D Point Cloud Completion

Point cloud completion aims to infer the complete point clouds from incomplete ones, which is used in remote sensing applications such as reconstructing and autonomous driving. However, most existing methods cannot recover accurate structure details of the object. In this paper, we propose point shift network (PS-Net). Our main contributions lie in the following three-folds. First, we propose a multi-resolution encoder, which extracts and fuses multi-resolution point cloud features hierarchically, thus avoiding information loss caused by a single global feature. Second, we design a multi-resolution point cloud generation structure, which can be combined with the multi-resolution encoder to generate gradually dense point clouds, avoiding the problem of non-uniformly density of the single-layer decoder. Third, we design the shift network, which is used to generate shift vectors to shift the coordinates of each point cloud, so as to further finetune the coordinate positions of point clouds, achieving more accurate prediction. We conduct comprehensive experiments on ShapeNet, KITTI, ScanObjectNN, and ModelNet40 datasets, which demonstrate that the proposed PS-Net achieves better performance than existing methods and verify the robustness of the proposed method. This paper contributes a new method to point cloud completion, realizes fine point cloud shape completion, and brings new possibilities to the research of autonomous driving, registration, and reconstruction.

Dense Point Cloud Completion Based on Generative Adversarial Network

Point cloud completion aims to reconstruct complete point clouds from partial point clouds, which is widely used in various fields such as autonomous driving and robotics. Most existing methods are sparse point cloud completion, where the number of point clouds after completion is relatively small and the details are insufficient. This article proposes a novel end-to-end generative adversarial network-based dense point cloud completion architecture (DPCG-Net). We design two generative adversarial network (GAN)-based modules that translate point cloud completion into mapping between global feature distributions obtained by encoding partial point clouds and ground truth, respectively. The first designed generator module proposes skip connections to fully connected layer-based network for regenerating global feature and changing the global feature distribution derived from the encoder module to approximate the ground truth global feature distribution. The second proposed discriminator module divides high-dimensional global feature vectors into several smaller batches for judgment to guarantee the similarity between the regenerated global feature and the ground truth. We perform quantitative and qualitative experiments on the ShapeNet and KITTI datasets. Experiments on ShapeNet demonstrate that our model outperforms other models in cases where the lack of a large proportion of point clouds results in a large loss of spatial structure, especially when 80% of point clouds are missing. Moreover, KITTI experiments reveal that it is also valid for realistic situations. In addition, application in classification shows that the classification accuracy of point clouds completed with DPCG-Net is as high as 86.5% under the condition of 80% missing point clouds.

Snowflake Point Deconvolution for Point Cloud Completion and Generation With Skip-Transformer.

Most existing point cloud completion methods suffer from the discrete nature of point clouds and the unstructured prediction of points in local regions, which makes it difficult to reveal fine local geometric details. To resolve this issue, we propose SnowflakeNet with snowflake point deconvolution (SPD) to generate complete point clouds. SPD models the generation of point clouds as the snowflake-like growth of points, where child points are generated progressively by splitting their parent points after each SPD. Our insight into the detailed geometry is to introduce a skip-transformer in the SPD to learn the point splitting patterns that can best fit the local regions. The skip-transformer leverages attention mechanism to summarize the splitting patterns used in the previous SPD layer to produce the splitting in the current layer. The locally compact and structured point clouds generated by SPD precisely reveal the structural characteristics of the 3D shape in local patches, which enables us to predict highly detailed geometries. Moreover, since SPD is a general operation that is not limited to completion, we explore its applications in other generative tasks, including point cloud auto-encoding, generation, single image reconstruction, and upsampling. Our experimental results outperform state-of-the-art methods under widely used benchmarks.

Bidirectional interaction between BIM and construction processes using a multisource geospatial data enabled point cloud model

Building Information Modeling (BIM) has increasingly been adopted as an as-planned construction state to provide essential support for fine construction. However, some deviations are inevitably induced between the BIM model and actual construction states in practical construction processes, and these deviations obstruct the unidirectional interaction between the BIM model and actual construction states, thus seriously degrading the fine construction quality. This paper proposes a bidirectional interaction mechanism between BIM and construction processes using a multisource geospatial data enabled point cloud model. In the forward interaction, parametric BIM model are obtained to provide essential information for fine construction. Particularly, a multisource geospatial data enabled point cloud modeling strategy is the core of the proposed method, which overcomes complex buildings characterized by severe occlusions, specular surfaces and similar components to capture accurate and complete 3D point cloud model that reflects the actual construction state. In the backward interaction, the point cloud model provides feedback to adjust the BIM model for ensuring a fit between the BIM model and actual construction state. Moreover, the resulting up-to-date BIM model can further instruct the subsequent construction processes; thus, BIM and construction become a closed task-oriented loop. The proposed method was applied to the curtain wall construction of the main stadium for the Chengdu 2022 31st Summer World University Games and compared with four other state-of-the-art point cloud registration methods. The results demonstrated the superior accuracy of our method over other methods. The results also showed that the proposed method could effectively maintain the interaction between the BIM model and actual construction states throughout the construction lifecycle and ensure the quality of fine construction.

Point Projection Network: A Multi-View-Based Point Completion Network with Encoder-Decoder Architecture

Recently, unstructured 3D point clouds have been widely used in remote sensing application. However, inevitable is the appearance of an incomplete point cloud, primarily due to the angle of view and blocking limitations. Therefore, point cloud completion is an urgent problem in point cloud data applications. Most existing deep learning methods first generate rough frameworks through the global characteristics of incomplete point clouds, and then generate complete point clouds by refining the framework. However, such point clouds are undesirably biased toward average existing objects, meaning that the completion results lack local details. Thus, we propose a multi-view-based shape-preserving point completion network with an encoder–decoder architecture, termed a point projection network (PP-Net). PP-Net completes and optimizes the defective point cloud in a projection-to-shape manner in two stages. First, a new feature point extraction method is applied to the projection of a point cloud, to extract feature points in multiple directions. Second, more realistic complete point clouds with finer profiles are yielded by encoding and decoding the feature points from the first stage. Meanwhile, the projection loss in multiple directions and adversarial loss are combined to optimize the model parameters. Qualitative and quantitative experiments on the ShapeNet dataset indicate that our method achieves good results in learning-based point cloud shape completion methods in terms of chamfer distance (CD) error. Furthermore, PP-Net is robust to the deletion of multiple parts and different levels of incomplete data.

Raw Scanned Point Cloud Registration with Repetition for Aircraft Fuel Tank Inspection

The aircraft fuel tank, as the source of fuel supply, is one of the key components for an aircraft. It is fastened to the fuselage with thousands of rivets, in which each rivet is sealed carefully to prevent the leakage. Thus, the sealing quality of the aircraft fuel tank needs to be precisely measured to ensure its normal function. To achieve the measurement, the complete 3D point clouds captured by 3D scanning are usually utilized. Therefore, it is necessary to align the different views of the aircraft fuel tank point clouds, to obtain the complete data. However, the aircraft fuel tank consists of abundant repetitive structures, such as the regular-distributed rivets, which can lead to mismatches in the aircraft fuel tank point clouds if using the conventional registration methods. In order to address the challenges, we propose a new registration framework, which is able to handle the point cloud match under the repetitive and complex scenes. First, a novel 3D descriptor called LP-PPF is constructed with the point and line features from the aircraft fuel tank point clouds, so as to identify the repetitive structures correctly. Then, according to the proposed descriptor, the accurate registration can be achieved between the neighboring point cloud pairs. Finally, based on the results of pairwise registration, the complete aircraft fuel tank point cloud can be obtained via the multi-view registration method. Experiments demonstrate that our method achieves favorable results on both synthetic and raw scanned point cloud data.

Hole Repairing Algorithm for 3D Point Cloud Model of Symmetrical Objects Grasped by the Manipulator.

For the engineering application of manipulator grasping objects, mechanical arm occlusion and limited imaging angle produce various holes in the reconstructed 3D point clouds of objects. Acquiring a complete point cloud model of the grasped object plays a very important role in the subsequent task planning of the manipulator. This paper proposes a method with which to automatically detect and repair the holes in the 3D point cloud model of symmetrical objects grasped by the manipulator. With the established virtual camera coordinate system and boundary detection, repair and classification of holes, the closed boundaries for the nested holes were detected and classified into two kinds, which correspond to the mechanical claw holes caused by mechanical arm occlusion and the missing surface produced by limited imaging angle. These two kinds of holes were repaired based on surface reconstruction and object symmetry. Experiments on simulated and real point cloud models demonstrate that our approach outperforms the other state-of-the-art 3D point cloud hole repair algorithms.

ResDLPS-Net: Joint residual-dense optimization for large-scale point cloud semantic segmentation

Semantic segmentation methods based on three-dimensional (3D) point clouds are mostly limited to input point clouds that have been divided into blocks for training. This is mainly attributed to the requirement of constant trade-offs between computational resources and accuracy for directly processing large-scale point clouds. Specifically, the block dividing strategy will add the data preprocessing time to some extent and may disturb the complete geometry of the object. Therefore, this paper proposes a large-scale point cloud semantic segmentation network without block dividing operation, referred to as ResDLPS-Net. This network can take the complete point cloud of the whole large scene as input and process up to nearly a million points on one single GPU. In particular, a novel feature extraction module is designed to efficiently extract neighbor, geometric, and semantic features. The learned features are then aggregated through the attention mechanism to form local feature descriptors. In addition, the proposed ResDLPS-Net is jointly trained by residual connections and dense convolutional connections to optimize the feature aggregation operation. As a result, the ResDLPS-Net performs brilliantly on multiple objects, such as windows, road markings, fences, etc. For example, the Mean Intersection over Union (mIoU) of road markings on the Toronto-3D dataset is 37.76% higher than the state-of-the-art algorithm. Moreover, this paper outperforms most deep learning methods on three well-known benchmark datasets, including the indoor dataset S3DIS and the outdoor large-scale scene datasets Semantic3D and Toronto-3D. The proposed ResDLPS-Net achieves the best performance on the S3DIS dataset. The average accuracy (mA) and overall accuracy (OA) are 82.3% and 88.1%, respectively. Notably, the proposed ResDLPS-Net attains a mIoU of 80.27% on the Toronto-3D dataset, which is 6.00% higher than the best results published currently.

Graph-Guided Deformation for Point Cloud Completion

For a long time, the point cloud completion task has been regarded as a pure generation task. After obtaining the global shape code through the encoder, a complete point cloud is generated using the shape priorly learnt by the networks. However, such models are undesirably biased towards prior average objects and inherently limited to fit geometry details. In this letter, we propose a Graph-Guided Deformation Network, which respectively regards the input data and intermediate generation as controlling and supporting points, and models the optimization guided by a graph convolutional network(GCN) for the point cloud completion task. Our key insight is to simulate the least square Laplacian deformation process via mesh deformation methods, which brings adaptivity for modeling variation in geometry details. By this means, we also reduce the gap between the completion task and the mesh deformation algorithms. As far as we know, we are the first to refine the point cloud completion task by mimicing traditional graphics algorithms with GCN-guided deformation. We have conducted extensive experiments on both the simulated indoor dataset ShapeNet, outdoor dataset KITTI, and our self-collected autonomous driving dataset Pandar40. The results show that our method outperforms the existing state-of-the-art algorithms in the 3D point cloud completion task.

FinerPCN: High fidelity point cloud completion network using pointwise convolution

3D scanners often obtain partial point clouds due to occlusion and limitation of viewing angles. Point cloud completion aims at inferring the full shape of an object from an incomplete point set. Existing deep learning models either do not consider local information or easily degrade the sharp details of the input, thereby losing some existing structures. In this paper, we propose a high fidelity point cloud completion network using pointwise convolution, called FinerPCN. FinerPCN generates complete and fine point clouds in a coarse-to-fine manner. FinerPCN consists of two subnetworks: an encoder-decoder for generating a coarse shape and pointwise convolution for refining its local structure. By repeatedly feeding partial input into the second subnetwork, FinerPCN effectively considers local information and alleviates structural blur of input while maintaining global shape. Experimental results show that FinerPCN generates finer detailed completion results than state-of-the-art methods while successfully keeping the shape of the input.

BOAT-BASED MOBILE LASER SCANNING FOR SHORELINE MONITORING OF LARGE LAKES

Abstract. In many applications, it is necessary to measure the shore areas of water bodies in three dimensions, for example for the planning of shore protection measures or regularly for water body monitoring. The complete recording of long shorelines can be very time-consuming with conventional methods. This paper presents the surveying of two lakes using a mobile laser scanner system. In order to close non-visible areas, the measurement was supplemented with terrestrial laser scanning and with UAV-supported photogrammetric methods. The result is an almost complete point cloud of the shoreline, each more than 8 km long, with a point spacing of a few millimetres.

ASHF-Net: Adaptive Sampling and Hierarchical Folding Network for Robust Point Cloud Completion

Estimating the complete 3D point cloud from an incomplete one lies at the core of many vision and robotics applications. Existing methods typically predict the complete point cloud based on the global shape representation extracted from the incomplete input. Although they could predict the overall shape of 3D objects, they are incapable of generating structure details of objects. Moreover, the partial input point sets obtained from range scans are often sparse, noisy and non-uniform, which largely hinder shape completion. In this paper, we propose an adaptive sampling and hierarchical folding network (ASHF-Net) for robust 3D point cloud completion. Our main contributions are two-fold. First, we propose a denoising auto-encoder with an adaptive sampling module, aiming at learning robust local region features that are insensitive to noise. Second, we propose a hierarchical folding decoder with the gated skip-attention and multi-resolution completion goal to effectively exploit the local structure details of partial inputs. We also design a KL regularization term to evenly distribute the generated points. Extensive experiments demonstrate that our method outperforms existing state-of-the-art methods on multiple 3D point cloud completion benchmarks.