Entire Point Cloud Research Articles

A New Framework for Generating Indoor 3D Digital Models from Point Clouds

Three-dimensional indoor models have wide applications in fields such as indoor navigation, civil engineering, virtual reality, and so on. With the development of LiDAR technology, automatic reconstruction of indoor models from point clouds has gained significant attention. We propose a new framework for generating indoor 3D digital models from point clouds. The proposed method first generates a room instance map of an indoor scene. Walls are detected and projected onto a horizontal plane to form line segments. These segments are extended, intersected, and, by solving an integer programming problem, line segments are selected to create room polygons. The polygons are converted into a raster image, and image connectivity detection is used to generate a room instance map. Then the roofs of the point cloud are extracted and used to perform an overlap analysis with the generated room instance map to segment the entire roof point cloud, obtaining the roof for each room. Room boundaries are defined by extracting and regularizing the roof point cloud boundaries. Finally, by detecting doors and windows in the scene in two steps, we generate the floor plans and 3D models separately. Experiments with the Giblayout dataset show that our method is robust to clutter and furniture point clouds, achieving high-accuracy models that match real scenes. The mean precision and recall for the floorplans are both 0.93, and the Point–Surface Distance (PSD) and standard deviation of the PSD for the 3D models are 0.044 m and 0.066 m, respectively.

A robust assessment method of point cloud quality for enhancing 3D robotic scanning

Point clouds are widely used to construct models of workpieces using 3D scanners, especially where high-quality robotic and automatic 3D scanning is required in industries and manufacturing. In recent years, Point Cloud Quality Assessment (PCQA) has garnered increasing attention as it provides quality scores for entire point clouds, addressing issues such as downsampling and compression distortions. However, current PCQA methods cannot provide specific and local quality scores, which are necessary to facilitate rescanning and recompletion in 3D robotic scanning. Additionally, 3D data produced by robotic view-planning algorithms are usually considered the final result, where PCQA is typically not involved. In this paper, we bridge the gap between PCQA methods and practical robotic 3D scanning. We propose a no-reference PCQA method that recognizes sparse regions during 3D scanning, providing both local and overall quality scores. Unlike traditional methods that primarily consider density as a key metric, our method assumes that an expected 3D scan will have a uniformly distributed point cloud on surfaces. We analyze the quality of points by using geometric information from surfaces fitted to these points, which are mapped to a 2D distribution based on specified distances and angles. We conducted experiments on various datasets, including both synthetic and public datasets, to evaluate the accuracy and robustness of our method. The results show that our method can represent the quality on surfaces more accurately and robustly than density calculation methods. Additionally, it outperforms most existing PCQA methods in scenarios of downsampling, which is a common challenge in high-quality 3D scanning applications. The performance of our quality enhancement experiments on practical 3D scanning, conducted towards the end of our study, demonstrates significant potential for real-world applications. The related code is released at https://github.com/leihui6/PQA.

Accurately identifying the defects of bubbles and foreign objects under the protective films of electric vehicle batteries by using 3D point clouds

Abstract For the defects of bubbles and foreign objects under the protective film of electric vehicle batteries, it is difficult to accurately identify them over traditional 2D optical images. In this paper, we first propose a supervoxel-based region growing algorithm for pre-segmentation of point clouds. Secondly, we utilize radial basis function interpolation and threshold segmentation methods to accurately segment defect point clouds from the entire point cloud. Finally, we develop a feature descriptor and combine it with support vector machine to classify bubbles and foreign objects under the film. This paper achieves the identification of bubbles and foreign objects under the film through two steps: point cloud segmentation and point cloud classification. Experimental results demonstrate that the proposed point cloud segmentation method exhibits high robustness to noise and the intrinsic curvature of the workpiece. Additionally, in the classification scenario presented in this paper, the proposed feature descriptor outperforms classical feature descriptors. Compared to image-based deep learning methods, the defect recognition algorithm proposed in this paper has clear principles and superior performance, with precision and recall of 95.63% and 96.95%, and an intersection over union metric of 0.926.

Terrain point cloud inpainting via signal decomposition

The rapid development of 3D acquisition technology has made it possible to obtain point clouds of real-world terrains. However, due to limitations in sensor acquisition technology or specific requirements, point clouds often contain defects such as holes with missing data. Inpainting algorithms are widely used to patch these holes. However, existing traditional inpainting algorithms rely on precise hole boundaries, which limits their ability to handle cases where the boundaries are not well-defined. On the other hand, learning-based completion methods often prioritize reconstructing the entire point cloud instead of solely focusing on hole filling. Based on the fact that real-world terrain exhibits both global smoothness and rich local detail, we propose a novel representation for terrain point clouds. This representation can help to repair the holes without clear boundaries. Specifically, it decomposes terrains into low-frequency and high-frequency components, which are represented by B-spline surfaces and relative height maps respectively. In this way, the terrain point cloud inpainting problem is transformed into a B-spline surface fitting and 2D image inpainting problem. By solving the two problems, the highly complex and irregular holes on the terrain point clouds can be well-filled, which not only satisfies the global terrain undulation but also exhibits rich geometric details. The experimental results also demonstrate the effectiveness of our method.

WS-SSD: Achieving faster 3D object detection for autonomous driving via weighted point cloud sampling

Due to the limited computational resources of the onboard computing devices of autonomous vehicles, the development of lightweight 3D object detectors is essential. Point-based detectors that progressively sample raw point clouds reduce numerous redundant computations and facilitate the implementation of high-speed 3D object detectors. The farthest point sampling based on Euclidean distance (D-FPS) is frequently utilized in point-based 3D detectors for point cloud sampling to reduce computation overhead. D-FPS can ensure uniform point sampling, covering the entire point cloud space as much as possible. The ratio of foreground and background points does not change significantly in the sampling results; hence, foreground objects do not receive sufficient attention. In addition, the number of road reflection points is large, and these points are close to foreground objects, reducing the sampling accuracy for faint objects. We propose weighted farthest point sampling based on Euclidean distance (W-DFPS), which selectively discards some road reflection points in the point sampling process, thus increasing the weight of foreground points in sampling results. It reduces the likelihood of faint objects being lost in the sampling results, such that a small number of sampling points can also cover most foreground objects. W-DFPS replaces D-FPS in the single-stage detector based on instance-aware (IA-SSD), and the network structure is slightly modified, namely the single-stage detector based on weighted sampling (WS-SSD). We evaluate WS-SSD in the KITTI dataset with a single A100 GPU. When the number of sampling points for the first module of WS-SSD is consistent with IA-SSD, the pedestrian detection accuracy is improved by an average of 4.06 compared to IA-SSD. Although the number of sampling points for the first module of WS-SSD is reduced to 25% of IA-SSD, the object detection accuracy remains competitive; it also achieves a state-of-the-art inference speed of 150.76 frames per second (FPS), which is a 46% improvement over IA-SSD.

Automatic Identification and Segmentation of Long‐Span Rail‐and‐Road Cable‐Stayed Bridges Using UAV LiDAR Point Cloud

Bridge information models are essential for bridge inspection, assessment, and management. LiDAR technology, particularly UAV LiDAR, offers a cost‐effective means to capture dense and accurate 3D coordinates of a bridge’s surface. However, the structure of large‐scale bridges is complex, and existing commercial software still demands substantial manual effort to segment the components when constructing bridge information models for large‐scale bridges. This study introduces a novel approach to automatically segment the components of a long‐span rail‐and‐road cable‐stayed bridge from the entire point cloud obtained through UAV LiDAR. In this proposed approach, the geometric and topological constraints of various bridge components are thoroughly examined, and a combination of the coarse‐to‐fine concept and top‐down strategy is employed. The key structural elements, including piers, cable towers, wind fairing plate, stay‐cable, main truss, railway surfaces, and deck surfaces, are identified and segmented. The proposed methodology achieves an average accuracy of over 96% at the point level validated using datasets acquired by UAV LiDAR.

Adaptive local adversarial attacks on 3D point clouds

Modern artificial intelligence systems rely heavily on deep learning techniques. However, deep neural networks are easily disturbed by adversarial objects. Adversarial examples are beneficial to improve the robustness of the 3D neural network model and enhance the stability of the artificial intelligence system. At present, most 3D adversarial attack methods perturb the entire point cloud to generate adversarial examples, which results in high perturbation costs and low operability in the physical world. In this paper, we propose an adaptive local adversarial attack method (AL-Adv) on 3D point clouds to generate adversarial point clouds. First, we analyze the vulnerability of the 3D network model and extract the salient regions of the input point cloud, namely the vulnerable regions. Second, we propose an adaptive gradient attack algorithm that targets salient regions. The proposed attack algorithm adaptively assigns different disturbances in different directions of the three-dimensional coordinates of the point cloud. Experimental results show that our proposed AL-Adv method achieves a higher attack success rate than the global attack method. Specifically, the adversarial examples generated by AL-Adv demonstrate good imperceptibility and small generation costs.

Adaptive LiDAR Reconstruction by Convolutional Compressive Sensing Network and Multivariate Empirical Mode Decomposition

With the recent advancement of LiDAR data processing pipelines integrated with signal processing techniques, it is now possible to rapidly acquire, process, and infer multidimensional information of 3D surroundings. This work proposes a novel compressive sensing (CS) framework (named MEMD-CSNet), based on Multivariate Empirical Mode Decomposition (MEMD) for adaptive 3D reconstruction of airborne urban LiDAR point clouds. EMD is a data-driven transform for decomposing the input data into Intrinsic Mode Functions (IMFs), capturing the implicit features in terms of variation in spatial frequency. Our approach generates a new filtered 3D signal constituting an optimal subset of IMFs based on the understanding of their statistical significance. Further, this filtered signal hierarchically extracts the overall shape in the last IMF (Residual function). Our proposed MEMD-CSNet is a deep convolutional network that (1) computes a multivariate EMD on input raw 3D LiDAR point clouds for generating data-dependent feature-rich filtered signal and (2) additionally augments this signal with convolutional features, thus learning a data-centric representation function (Ψ) for transforming the input into a low-dimensional sparse basis. We present extensive qualitative and quantitative analysis using the proposed MEMD-CSNet for a set of chosen CS measurement ratios. MEMD-CSNet achieves a maximum reconstruction PSNR of 108.22 dB (approx. 30 dB/1.36x better than without using MEMD) for reconstruction fidelity on selected urban LiDAR scenes. We envisage that our proposed MEMD-CSNet has a high potential (as a data-centric framework) for generating data-driven sparse feature maps and reconstructing entire 3D LiDAR point clouds from a few sparse measurements.

TreeNet: Structure preserving multi-class 3D point cloud completion

Generating the missing data of 3D object point clouds from partial observations is a challenging task. Existing state-of-the-art learning-based 3D point cloud completion methods tend to use a limited number of categories/classes of training data and regenerate the entire point cloud based on the training datasets. As a result, output 3D point clouds generated by such methods may lose details (i.e. sharp edges and topology changes) due to the lack of multi-class training. These methods also lose the structural and spatial details of partial inputs due to the models do not separate the reconstructed partial input from missing points in the output.In this paper, we propose a novel deep learning network - TreeNet for 3D point cloud completion. TreeNet has two networks in hierarchical tree-based structures: TreeNet-multiclass focuses on multi-class training with a specific class of the completion task on each sub-tree to improve the quality of point cloud output; TreeNet-binary focuses on generating points in missing areas and fully preserving the original partial input. TreeNet-multiclass and TreeNet-binary are both network decoders and can be trained independently. TreeNet decoder is the combination of TreeNet-multiclass and TreeNet-binary and is trained with an encoder from existing methods (i.e. PointNet encoder). We compare the proposed TreeNet with five state-of-the-art learning-based methods on fifty classes of the public Shapenet dataset and unknown classes, which shows that TreeNet provides a significant improvement in the overall quality and exhibits strong generalization to unknown classes that are not trained.

Building Maps Using Monocular Image-feeds from Windshield-mounted Cameras in a Simulator Environment

3-dimensional, accurate, and up-to-date maps are essential for vehicles with autonomous capabilities, whose functionality is made possible by machine learning-based algorithms. Since these solutions require a tremendous amount of data for parameter optimization, simulation-to-reality (Sim2Real) methods have been proven immensely useful for training data generation. For creating realistic models to be used for synthetic data generation, crowdsourcing techniques present a resource-efficient alternative. In this paper, we show that using the Carla simulation environment, a crowdsourcing model can be created that mimics a multi-agent data gathering and processing pipeline. We developed a solution that yields dense point clouds based on monocular images and location information gathered by individual data acquisition vehicles. Our method provides scene reconstructions using the robust Structure-from-Motion (SfM) solution of Colmap. Moreover, we introduce a solution for synthesizing dense ground truth point clouds originating from the Carla simulator using a simulated data acquisition pipeline. We compare the results of the Colmap reconstruction with the reference point cloud after aligning them using the iterative closest point algorithm. Our results show that a precise point cloud reconstruction was feasible with this crowdsourcing-based approach, with 54\% of the reconstructed points having an error under 0.05 m, and a weighted root mean square error of 0.0449 m for the entire point cloud.

Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation.

Visual Question Answering on 3D Point Cloud (VQA-3D) is an emerging yet challenging field that aims at answering various types of textual questions given an entire point cloud scene. To tackle this problem, we propose the CLEVR3D, a large-scale VQA-3D dataset consisting of 171K questions from 8,771 3D scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Through such a manner, we initially generated 44K questions from 1,333 real-world scenes. Moreover, a more challenging setup is proposed to remove the confounding bias and adjust the context from a common-sense layout. Such a setup requires the network to achieve comprehensive visual understanding when the 3D scene is different from the general co-occurrence context (e.g., chairs always exist with tables). To this end, we further introduce the compositional scene manipulation strategy and generate 127K questions from 7,438 augmented 3D scenes, which can improve VQA-3D models for real-world comprehension. Built upon the proposed dataset, we baseline several VQA-3D models, where experimental results verify that the CLEVR3D can significantly boost other 3D scene understanding tasks.

CTpoint: A novel local and global features extractor for point cloud

Local convolutional operation and global attention-based transformer operation can extract features of the point cloud from two different scales respectively, but few methods can combine them effectively. In this paper, we propose a novel point cloud feature extractor that has the advantages of both convolutional and transformer operation, which is named as CTpoint. Specifically, CTpoint is composed of two branches, where ”C” represents the convolutional branch and “T” represents the transformer branch. The convolutional branch is responsible for extracting the local feature from grouped neighbor points, and the transformer branch performs the attention operation on the entire point cloud to capture the global feature. To make the two branches communicate with each other, the Feature Transmission Element (FTE) is proposed for the alignment of the spatial-feature dimension and the semantic space between the local and global features, so that the extracted local and global features can be effectively fused and the two branches can coordinately learn expressive features. We utilize CTpoint to construct point cloud classification and segmentation networks and evaluate their performance in several public datasets. In addition, we visualize the feature learned by CTpoint. The experimental results show that the expressive features learned by CTpoint make the networks achieve the state of the art performance on the point cloud classification and segmentation tasks.

PointCaps: Raw point cloud processing using capsule networks with Euclidean distance routing

Raw point cloud processing using capsule networks is widely adopted in classification, reconstruction, and segmentation due to its ability to preserve spatial agreement of the input data. However, most of the existing capsule based network approaches are computationally heavy and fail at representing the entire point cloud as a single capsule. We address these limitations in existing capsule network based approaches by proposing PointCaps, a novel convolutional capsule architecture with parameter sharing. Along with PointCaps, we propose a novel Euclidean distance routing algorithm and a class-independent latent representation. The latent representation captures physically interpretable geometric parameters of the point cloud, with dynamic Euclidean routing, PointCaps well-represents the spatial (point-to-part) relationships of points. PointCaps has a significantly lower number of parameters and requires a significantly lower number of FLOPs while achieving better reconstruction with comparable classification and segmentation accuracy for raw point clouds compared to state-of-the-art capsule networks.

Improved Registration Algorithm Based on Double Threshold Feature Extraction and Distance Disparity Matrix.

Entire surface point clouds in complex objects cannot be captured in a single direction by using noncontact measurement methods, such as machine vision; therefore, different direction point clouds should be obtained and registered. However, high efficiency and precision are crucial for registration methods when dealing with huge number of point clouds. To solve this problem, an improved registration algorithm based on double threshold feature extraction and distance disparity matrix (DDM) is proposed in this study. Firstly, feature points are extracted with double thresholds using normal vectors and curvature to reduce the number of points. Secondly, a fast point feature histogram is established to describe the feature points and obtain the initial corresponding point pairs. Thirdly, obviously wrong corresponding point pairs are eliminated as much as possible by analysing the Euclidean invariant features of rigid body transformation combined with the DDM algorithm. Finally, the sample consensus initial alignment and the iterative closest point algorithms are used to complete the registration. Experimental results show that the proposed algorithm can quickly process large data point clouds and achieve efficient and precise matching of target objects. It can be used to improve the efficiency and precision of registration in distributed or mobile 3D measurement systems.

Colored Point Cloud Completion for a Head Using Adversarial Rendered Image Loss.

Recent advances in depth measurement and its utilization have made point cloud processing more critical. Additionally, the human head is essential for communication, and its three-dimensional data are expected to be utilized in this regard. However, a single RGB-Depth (RGBD) camera is prone to occlusion and depth measurement failure for dark hair colors such as black hair. Recently, point cloud completion, where an entire point cloud is estimated and generated from a partial point cloud, has been studied, but only the shape is learned, rather than the completion of colored point clouds. Thus, this paper proposes a machine learning-based completion method for colored point clouds with XYZ location information and the International Commission on Illumination (CIE) LAB () color information. The proposed method uses the color difference between point clouds based on the Chamfer Distance (CD) or Earth Mover’s Distance (EMD) of point cloud shape evaluation as a color loss. In addition, an adversarial loss to images rendered from the output point cloud can improve the visual quality. The experiments examined networks trained using a colored point cloud dataset created by combining two 3D datasets: hairstyles and faces. Experimental results show that using the adversarial loss with the colored point cloud renderer in the proposed method improves the image domain’s evaluation.

A new point cloud simplification method with feature and integrity preservation by partition strategy

The advancement of 3D scanning technology has gradually brought to light the issue of complex and time-consuming processing of high-density point cloud data. To address this need, this article proposes a method for point cloud simplification based on the concept of partitioning, which divides the point cloud's points into edge points, feature points, and non-feature points. On the basis of the normal angular difference, the edge points are extracted from the entire point cloud. The point cloud is then segmented by curvature into feature and non-feature regions using a region growing segmentation method. The feature points are determined by calculating the information entropy of the points in the feature region, whereas the non-feature points are extracted by voxel down-sampling the non-feature points. The experiment demonstrates that the proposed method effectively preserves the features and integrity of the point cloud while requiring less computational effort.

An Accuracy Assessment of Snow Depth Measurements in Agro-Forested Environments by UAV Lidar

This study assesses the performance of UAV lidar system in measuring high-resolution snow depths in agro-forested landscapes in southern Québec, Canada. We used manmade, mobile ground control points in summer and winter surveys to assess the absolute vertical accuracy of the point cloud. Relative accuracy was determined by a repeat flight over one survey block. Estimated absolute and relative errors were within the expected accuracy of the lidar (~5 and ~7 cm, respectively). The validation of lidar-derived snow depths with ground-based measurements showed a good agreement, however with higher uncertainties observed in forested areas compared with open areas. A strip alignment procedure was used to attempt the correction of misalignment between overlapping flight strips. However, the significant improvement of inter-strip relative accuracy brought by this technique was at the cost of the absolute accuracy of the entire point cloud. This phenomenon was further confirmed by the degraded performance of the strip-aligned snow depths compared with ground-based measurements. This study shows that boresight calibrated point clouds without strip alignment are deemed to be adequate to provide centimeter-level accurate snow depth maps with UAV lidar. Moreover, this study provides some of the earliest snow depth mapping results in agro-forested landscapes based on UAV lidar.

An Accurate Digital Subsidence Model for Deformation Detection of Coal Mining Areas Using a UAV-Based LiDAR

Coal mine surface subsidence detection determines the damage degree of coal mining, which is of great importance for the mitigation of hazards and property loss. Therefore, it is very important to detect deformation during coal mining. Currently, there are many methods used to detect deformations in coal mining areas. However, with most of them, the accuracy is difficult to guarantee in mountainous areas, especially for shallow seam mining, which has the characteristics of active, rapid, and high-intensity surface subsidence. In response to these problems, we made a digital subsidence model (DSuM) for deformation detection in coal mining areas based on airborne light detection and ranging (LiDAR). First, the entire point cloud of the study area was obtained by coarse to fine registration. Second, noise points were removed by multi-scale morphological filtering, and the progressive triangulation filtering classification (PTFC) algorithm was used to obtain the ground point cloud. Third, the DEM was generated from the clean ground point cloud, and an accurate DSuM was obtained through multiple periods of DEM difference calculations. Then, data mining was conducted based on the DSuM to obtain parameters such as the maximum surface subsidence value, a subsidence contour map, the subsidence area, and the subsidence boundary angle. Finally, the accuracy of the DSuM was analyzed through a comparison with ground checkpoints (GCPs). The results show that the proposed method can achieve centimeter-level accuracy, which makes the data a good reference for mining safety considerations and subsequent restoration of the ecological environment.

A Representation Separation Perspective to Correspondence-Free Unsupervised 3-D Point Cloud Registration

3D point cloud registration in remote sensing field has been greatly advanced by deep learning based methods, where the rigid transformation is either directly regressed from the two point clouds (correspondences-free approaches) or computed from the learned correspondences (correspondences-based approaches). Existing correspondences-free methods generally learn the holistic representation of the entire point cloud, which is fragile for partial and noisy point clouds. In this paper, we propose a correspondences-free unsupervised point cloud registration (UPCR) method from the representation separation perspective. First, we model the input point cloud as a combination of pose-invariant representation and pose-related representation. Second, the pose-related representation is used to learn the relative pose wrt a "latent canonical shape" for the source and target point clouds respectively. Third, the rigid transformation is obtained from the above two learned relative poses. Our method not only filters out the disturbance in pose-invariant representation but also is robust to partial-to-partial point clouds or noise. Experiments on benchmark datasets demonstrate that our unsupervised method achieves comparable if not better performance than state-of-the-art supervised registration methods.

Deep-PCAC: An End-to-End Deep Lossy Compression Framework for Point Cloud Attributes

The large data volume of point clouds poses severe challenges for efficient storage and transmission in recent years. In this paper, we propose the first<monospace>--</monospace>to our best knowledge<monospace>--</monospace>end-to-end deep framework for compressing point cloud attributes. Specifically, we propose a point cloud lossy attribute autoencoder, which directly encodes and decodes point cloud attributes with the help of geometry, instead of voxelizing or projecting the points. In the autoencoder, we propose a second-order point convolution that utilizes the spatial correlations between more points and the nonlinear relationship between attribute features. We introduce a dense point-inception block, which derives from a combination of an inception-style block and a dense block, to improve feature propagation. In addition, we devise a multiscale loss to guide the autoencoder in focusing attention on the coarse-grained points with better coverage of the entire point cloud, which makes it easier for the autoencoder to obtain better optimization of the qualities of all points. Experimental results show that our proposed framework still has a performance gap compared with the state-of-the-art algorithms in the MPEG G-PCC reference software TMC13. However, it does outperform the RAHT-RLGR, which is one of the core transforms used in TMC13 without many well-designed techniques that make TMC13 what it is today. It outperforms RAHT-RLGR by 2.63 dB, 1.77 dB, and 3.40 dB on average in terms of the BD-PSNR for the Y, U, and V components. A subjective quality comparison demonstrates that our framework can preserve more textures and reduce blocking and color noise artifacts.