Large-scale 3D Point Clouds Research Articles

2L-LSH: A Locality-Sensitive Hash Function-Based Method For Rapid Point Cloud Indexing

Abstract The development of 3D scanning technology has enabled the acquisition of massive point cloud models with diverse structures and large scales, thereby presenting significant challenges in point cloud processing. Fast neighboring points search is one of the most common problems, which is frequently used in model reconstruction, classification, retrieval and feature visualization. Hash function is well known for its high-speed and accurate performance in searching high-dimensional data, which is also the core of the proposed 2L-LSH. Specifically, the 2L-LSH algorithm adopts a two-step hash function strategy, in which the popular step divides the bounding box of the point cloud model and the second step constructs a generalized table-based data structure. The proposed 2L-LSH offers a highly efficient and accurate solution for fast neighboring points search in large-scale 3D point cloud models, making it a promising technique for various applications in the field. The proposed algorithm is compared with the well-known methods including Kd-tree and Octree; the obtained results demonstrated that the proposed method outperforms Kd-tree and Octree in terms of speed, i.e. the time consumption of kNN search can be 51.111% and 94.159% lower than Kd-tree and Octree, respectively. And the RN search time can be 54.519% and 41.840% lower than Kd-tree and Octree, respectively.

Multi-Scale Classification and Contrastive Regularization: Weakly Supervised Large-Scale 3D Point Cloud Semantic Segmentation

With the proliferation of large-scale 3D point cloud datasets, the high cost of per-point annotation has spurred the development of weakly supervised semantic segmentation methods. Current popular research mainly focuses on single-scale classification, which fails to address the significant feature scale differences between background and objects in large scenes. Therefore, we propose MCCR (Multi-scale Classification and Contrastive Regularization), an end-to-end semantic segmentation framework for large-scale 3D scenes under weak supervision. MCCR first aggregates features and applies random downsampling to the input data. Then, it captures the local features of a random point based on multi-layer features and the input coordinates. These features are then fed into the network to obtain the initial and final prediction results, and MCCR iteratively trains the model using strategies such as contrastive learning. Notably, MCCR combines multi-scale classification with contrastive regularization to fully exploit multi-scale features and weakly labeled information. We investigate both point-level and local contrastive regularization to leverage point cloud augmentor and local semantic information and introduce a Decoupling Layer to guide the loss optimization in different spaces. Results on three popular large-scale datasets, S3DIS, SemanticKITTI and SensatUrban, demonstrate that our model achieves state-of-the-art (SOTA) performance on large-scale outdoor datasets with only 0.1% labeled points for supervision, while maintaining strong performance on indoor datasets.

NLA-GCL-Net: semantic segmentation of large-scale surveying point clouds based on neighborhood label aggregation (NLA) and global context learning (GCL)

For large-scale 3D point clouds from surveying, the RandLA-Net semantic segmentation network model is unable to learn global context features efficiently during training, leading to suboptimal feature acquisition of globally related features. Furthermore, inconsistent predictions are made in the vicinity of data boundaries due to the oversimplified usage of multi-layer perceptrons (MLPs) and non-linear activation functions following decoding, which has a detrimental impact on both model training and performance. This study provides two solutions to address these issues: a neighborhood label aggregation (NLA) classifier that aggregates neighborhood information for segmentation tasks, and a global context learning (GCL) module that learns global volume-relative information to improve the semantic segmentation network. Overall accuracy (OA) obtained via experimental validation on large-scale SensatUrban and S3DIS datasets is 91.80% and 88.8%, respectively. The proposed model outperforms RandLA-Net by 3.8% and 3.0% in overall mean Intersection over Union (mIOU), respectively, significantly improving model performance and generalization abilities. This work offers new perspectives on how to effectively segment point cloud data.

A Review on the Deep Learning-based Surface Reconstruction from the Point Clouds

Background: Point cloud has become one of the most important data formats for 3D presentation because of the increased availability of acquisition devices and its wide applications. Deep learning has the most powerful ability to capture features from data and has successfully solved various problems in the field of image, such as classification, segmentation, and generation. Deep learning is commonly used to process data with a structured grid, while point cloud is irregular and unstructured. The irregularity of point clouds makes it difficult to use deep learning to solve the problems represented by point clouds. Recently, numerous approaches have been proposed to process point clouds with deep learning to solve various problems. Objective: The objective of this study is to serve as a guide to new scholars in the field of deep learning on 3D surface reconstruction from point clouds as it presents the recent progress in deep learning-based surface reconstruction for point clouds. It helps scholars to grasp the current research situation better and further explore the search direction. Method: This study reviews the recent progress in deep learning-based methods used for surface reconstruction from point clouds and large-scale 3D point cloud benchmark datasets commonly used. Results: Several relevant articles on deep learning used for surface reconstruction from point clouds and some recent patents on deep learning applications are collected and reviewed in this paper. The difficulty of irregularity of point clouds can be overcome by deep learning methods, thus achieving remarkable progress in surface reconstruction. Conclusion: Deep learning for 3D surface reconstruction from point clouds is becoming a research hotspot due to its performance in terms of anti-interference and generalization. Although the advance is remarkable, there are still some challenges that need to be further studied.

Development of large-scale synthetic 3D point cloud datasets for vision-based bridge structural condition assessment

Visual recognition of 3D point cloud data of bridge inspection scenes is a key step in automating the visual inspection process, which is currently largely manual and inefficient. To alleviate the lack of large-scale annotated point cloud datasets for training such 3D visual recognition algorithms, this research investigates an approach for developing large-scale synthetic point cloud datasets. The proposed approach proceeds in four steps: (1) random generation of different types of bridges in computer graphics environments; (2) sampling of camera trajectories that represent data collection scenarios during bridge inspection; (3) 3D reconstruction using Structure from Motion (SfM) applied to rendered synthetic images; (4) automated annotation of the reconstructed point cloud using ground truth masks obtained with synthetic images. Besides, this research proposes to store point uncertainty information defined by the error between the ground truth depth and the depth calculated from the SfM results. Prior to training, thresholds can be applied to this uncertainty information to control the levels of outliers in the dataset. This research demonstrates the proposed approach by generating point cloud datasets for two data collection scenarios. The effectiveness of the generated datasets is investigated by training 3D semantic segmentation algorithms and evaluating the performance on real and synthetic point cloud data. The proposed approach for point cloud dataset generation will facilitate the development of generalizable and high level-of-detail 3D recognition algorithms toward autonomous bridge inspection.

Weakly Supervised Learning Method for Semantic Segmentation of Large-Scale 3D Point Cloud Based on Transformers

Abstract. Nowadays, semantic segmentation results of 3D point cloud have been widely applied in the fields of robotics, autonomous driving, and augmented reality etc. Thanks to the development of relevant deep learning models (such as PointNet), supervised training methods have become hotspot, in which two common limitations exists: inferior feature representation of 3D points and massive annotations. To improve 3D point feature, inspired by the idea of transformer, we employ a so-call LCP network that extracts better feature by investigating attentions between target 3D points and its corresponding local neighbors via local context propagation. Training transformer-based network needs amount of training samples, which itself is a labor-intensive, costly and error-prone work, therefore, this work proposes a weakly supervised framework, in particular, pseudo-labels are estimated based on the feature distances between unlabeled points and prototypes, which are calculated based on labeled data. The extensive experimental results show that, the proposed PL-LCP can yield considerable results (67.6% mIOU for indoor and 67.3% for outdoor) even if only using 1% real labels, and comparing to several state-of-the-art method using all labels, we achieve superior results in mIOU, OA for indoor (65.9%, 89.2%).

Dynamic Downsampling Algorithm for 3D Point Cloud Map Based on Voxel Filtering

In response to the challenge of handling large-scale 3D point cloud data, downsampling is a common approach, yet it often leads to the problem of feature loss. We present a dynamic downsampling algorithm for 3D point cloud maps based on an improved voxel filtering approach. The algorithm consists of two modules, namely, dynamic downsampling and point cloud edge extraction. The former adapts voxel downsampling according to the features of the point cloud, while the latter preserves edge information within the 3D point cloud map. Comparative experiments with voxel downsampling, grid downsampling, clustering-based downsampling, random downsampling, uniform downsampling, and farthest-point downsampling were conducted. The proposed algorithm exhibited favorable downsampling simplification results, with a processing time of 0.01289 s and a simplification rate of 91.89%. Additionally, it demonstrated faster downsampling speed and showcased improved overall performance. This enhancement not only benefits productivity but also highlights the system’s efficiency and effectiveness.

Large-scale Point Cloud Segmentation based on Multi-feature Local Enhanced Fusion

This paper introduces a framework for large-scale 3D point cloud semantic segmentation - the MLEF-Net model. The model aims to improve the segmentation accuracy of large-scale point clouds by innovatively combining Manhattan distance-based KNN neighborhood search with feature aggregation techniques. This approach uniquely handles spatial, color, and normal vector attributes, thereby improving the segmentation results. The superiority of the model is validated through comprehensive testing on the SemanticKITTI and nuScenes datasets, demonstrating its potential to enhance point cloud segmentation through advanced feature fusion strategies.

Regional-to-Local Point-Voxel Transformer for Large-Scale Indoor 3D Point Cloud Semantic Segmentation

Semantic segmentation of large-scale indoor 3D point cloud scenes is crucial for scene understanding but faces challenges in effectively modeling long-range dependencies and multi-scale features. In this paper, we present RegionPVT, a novel Regional-to-Local Point-Voxel Transformer that synergistically integrates voxel-based regional self-attention and window-based point-voxel self-attention for concurrent coarse-grained and fine-grained feature learning. The voxel-based regional branch focuses on capturing regional context and facilitating inter-window communication. The window-based point-voxel branch concentrates on local feature learning while integrating voxel-level information within each window. This unique design enables the model to jointly extract local details and regional structures efficiently and provides an effective and efficient solution for multi-scale feature fusion and a comprehensive understanding of 3D point clouds. Extensive experiments on S3DIS and ScanNet v2 datasets demonstrate that our RegionPVT achieves competitive or superior performance compared with state-of-the-art approaches, attaining mIoUs of 71.0% and 73.9% respectively, with significantly lower memory footprint.

Class-aware tiny object recognition over large-scale 3D point clouds

Although tremendous strides have been made in object recognition over large-scale 3D point clouds, one of the remaining open challenges is detecting tiny objects. The signal and appearance information of tiny objects are generally sparse and insufficient due to their tiny size, which makes the tiny object difficult to be detected with existing studies both in tiny 2D object recognition and general 3D object recognition since 3D topology relations are lost due to feature projection in tiny 2D object recognition and massive background may dominate the feature learner in general 3D object recognition. To the best of our knowledge, we explore tiny object recognition for the first time over large-scale 3D point clouds by designing deep neural network. To cope with this problem, a novel two-stage approach is proposed in this paper, which can effectively learn an informative and discriminative feature representation and efficiently process large-scale point clouds. Specifically, a class-aware filtering strategy is designed to dispense with the redundant background information. After that, we introduce a novel aggregating and sampling scheme in a supervised manner to progressively increase the receptive field for each 3D point, which can adaptively capture useful signals from tiny objects and learn complex geometric structures. Last, we propose a biased loss function that is inversely correlated to the number of tiny object points, thereby effectively handling the sparsity of tiny objects. Extensive experimental results over two real-world data sets validate the effectiveness and efficiency of the proposed approach.

Design of constrained dynamic path planning algorithms in large-scale 3D point cloud maps for UAVs

The technology of unmanned aerial vehicles (UAVs) has been growing rapidly in recent years. To achieve highly autonomous flight capability, collision-free path generation becomes a key research topic. However, for a large-scale terrain map, an efficient path searching algorithm is always a challenging problem. Based on the Goal-bias Rapidly-exploring Random Tree Star (GB-RRT*) together with the aid of a bidirectional search technique, in this paper, a new constrained path planning algorithm, called Parent Revisited Goal and Parent Bias RRT* (PRGPB-RRT*), is proposed. Three major objectives are considered as the development benchmark: 1). shorter total distance, 2). smoother flight path, and 3). high computational efficiency. To meet different task requirements, a dynamic path planning method is investigated to diminish the high costs due to large-scale environments and keep track of the changing destination by finding local target points. In addition, unknown object avoidance is an essential part of flight path planning. As a consequence, the Fitting-Octree is presented to partition the point clouds into multiple cubes to represent the obstacles precisely. Moreover, the method of grouping point clouds into clusters effectively increases the efficiency of the implementation under a large number of obstacles. Finally, the feasibility of the proposed method is verified by a real-world 3D large-scale point cloud map. Experiments show that the proposed method successfully enhances the path generation efficiency as well as the trajectory smoothness. Moreover, the presented PRGPB-RRT* can be applied to different intelligent UAVs subject to flight kinematics constrain.

3D Mapping for a Large Crane Using Rotating 2D-Lidar and IMU Attached to the Crane Boom

This paper describes a novel method for large-scale 3D mapping for construction cranes with an arbitrary motion of the sensor system (2D lidar and IMU) attached to the crane boom. A heavy lidar with a slowly rotating base is needed to make a large-scale map both vertically and horizontally for cranes. This sensor configuration and mapping conditions entail handling each 2D scan separately, making it difficult to adopt existing rotating 2D lidar-based methods that construct a virtual 3D scan from a set of 2D scans. In the proposed method, we introduce a complementary filter with moving average filtering for lidar pose estimation, which is more robust to severe vibration than Kalman filter-based methods. As there are only a small amount of overlaps between 2D lidar scans, we propose a map correction method based on a pose graph optimization with planar environmental constraints. We evaluate the proposed method in a simulation and a small-sized real environment and compare it with one of the state-of-the-art methods. The evaluation results reveal that the proposed method can accurately estimate the sensor poses, thereby generating a high-quality, large-scale 3D point cloud map.

ROBUST TECHNIQUES FOR BUILDING FOOTPRINT EXTRACTION IN AERIAL LASER SCANNING 3D POINT CLOUDS

Abstract. The building footprint is crucial for a volumetric 3D representation of a building that is applied in urban planning, 3D city modeling, cadastral and topographic map generation. Aerial laser scanning (ALS) has been recognized as the most suitable means of large-scale 3D point cloud data (PCD) acquisition. PCD can produce geometric detail of a scanned surface. However, it is almost impossible to get point clouds without noise and outliers. Besides, data incompleteness and occlusions are two common phenomena for PCD. Most of the existing methods for building footprint extraction employ classification, segmentation, voting techniques (e.g., Hough-Transform or RANSAC), or Principal Component Analysis (PCA) based methods. It is known that classical PCA is highly sensitive to outliers, even RANSAC which is known as a robust technique for shape detection is not free from outlier effects. This paper presents a novel algorithm that employs MCMD (maximum consistency within minimum distance), MSAC (a robust variant of RANSAC) and a robust regression to extract reliable building footprints in the presence of outliers, missing points and irregular data distributions. The algorithm is successfully demonstrated through two sets of ALS PCD.

MFOC-CliqueNet: A CliqueNet-Based Optimal Combination of Multidimensional Features Classification Method for Large-Scale Laser Point Clouds

As large-scale laser 3D point clouds data contains massive and complex data, it faces great challenges in the automatic intelligent processing and classification of large-scale 3D point clouds. Aiming at the problem that 3D point clouds in complex scenes are self-occluded or occluded, which could reduce the object classification accuracy, we propose a multidimension feature optimal combination classification method named MFOC-CliqueNet based on CliqueNet for large-scale laser point clouds. The optimal combination matrix of multidimension features is constructed by extracting the three-dimensional features and multidirectional two-dimension features of 3D point cloud. This is the first time that multidimensional optimal combination features are introduced into cyclic convolutional networks CliqueNet. It is important for large-scale 3D point cloud classification. The experimental results show that the MFOC-CliqueNet framework can realize the latest level with fewer parameters. The experiments on the Large-Scale Scene Point Cloud Oakland dataset show that the classification accuracy of our method is 98.9%, which is better than other classification algorithms mentioned in this paper.

DPC-Net: Distributed Point Convolution Network for large-scale point clouds semantic segmentation

Background: Applying convolution neural networks for large-scale 3D point clouds semantic segmentation is quiet challenging, due to the unordered characteristics of 3D data and the computation burden of large-scale point clouds. Methods: To solve these problems, we designed DPC-Net (Distributed Point Convolution Network). The input point clouds of DPC-Net are partitioned by the K-nearest neighbor strategy and reordered based on Euclidean distance. For reducing computation and memory consumption while retaining critical features, the random sampling strategy is used and a distributed point convolution operation is designed. Our novel convolution method extracts parallel local geometric information including space distance and angle features, respectively. Furthermore, our proposed method could be easily and efficiently embedded into many networks for point clouds semantic segmentation. Results: Extensive experimental results on the Semantic3D and CSPC (Complex Scene Point Cloud) datasets indicate that the proposed DPC-Net not only obtains state-of-the-art performances but also reduces semantic segmentation time. Conclusions: In general, we present an efficient and lightweight deep convolutional network, DPC-Net, which captures local geometric features and local contextual information to predict point labels.

MSIDA-Net: Point Cloud Semantic Segmentation via Multi-Spatial Information and Dual Adaptive Blocks

Large-scale 3D point clouds are rich in geometric shape and scale information but they are also scattered, disordered and unevenly distributed. These characteristics lead to difficulties in learning point cloud semantic segmentations. Although many works have performed well in this task, most of them lack research on spatial information, which limits the ability to learn and understand the complex geometric structure of point cloud scenes. To this end, we propose the multispatial information and dual adaptive (MSIDA) module, which consists of a multispatial information encoding (MSI) block and dual adaptive (DA) blocks. The MSI block transforms the information of the relative position of each centre point and its neighbouring points into a cylindrical coordinate system and spherical coordinate system. Then the spatial information among the points can be re-represented and encoded. The DA blocks include a Coordinate System Attention Pooling Fusion (CSAPF) block and a Local Aggregated Feature Attention (LAFA) block. The CSAPF block weights and fuses the local features in the three coordinate systems to further learn local features, while the LAFA block weights the local aggregated features in the three coordinate systems to better understand the scene in the local region. To test the performance of the proposed method, we conducted experiments on the S3DIS, Semantic3D and SemanticKITTI datasets and compared the proposed method with other networks. The proposed method achieved 73%, 77.8% and 59.8% mean Intersection over Union (mIoU) on the S3DIS, Semantic3D and SemanticKITTI datasets, respectively.

Paying attention for adjacent areas: Learning discriminative features for large-scale 3D scene segmentation

Despite recent improvements in analyzing large-scale 3D point clouds, several problems still exist: (a) segmentation models suffer from intra-class inconsistency and inter-class indistinction; (b) the existing methods ignore the inherent long-tailed class distribution of real-world 3D data. These problems result in unsatisfactory semantic segmentation predictions, especially in object adjacent areas. To handle these problems, this paper proposes a novel Adjacent areas Refinement Network (ARNet). Specifically, an Adjacent areas Refinement (AR) module is designed, which consists of two parallel attention blocks. Besides, our proposed attention blocks can process a large number of points (N∼105) with a slight increase in the computational complexity and time cost. Additionally, to deal with the inherent long-tailed class distribution in real-world 3D data, imbalance adjustment loss and occupancy regression loss are introduced. Based on this, the proposed network can handle the classification of both majority and minority classes, which is essential in distinguishing the ambiguous parts in large-scale 3D scenes. The proposed AR module and the loss functions can be easily integrated into the cutting-edge backbone networks, contributing to better performance in modeling semantic inter-dependencies and significantly improving the accuracy of the state-of-the-art semantic segmentation methods on indoor and outdoor scenes.

An encoder-decoder deep learning method for multi-class object segmentation from 3D tunnel point clouds

Discovering seepage is widely thought to be critical for maintaining the healthy conditions of the tunnel. Unfortunately, most of the seepage surveys are still manual with tedious, time-consuming, and inefficient as well as work-related physical injuries. To address this problem, this research proposes an encoder-decoder deep learning method combined with point cloud techniques for multi-class object segmentation, including seepage, from 3D tunnel point clouds. This method develops data processing and feature extraction techniques to perform normalization of 3D point clouds with full consideration of point features, followed by constructing voxels as input to the proposed encoder-decoder architecture for learning. In the training process, an optimal model is selected with a learning rate of 0.0001, a batch size of 256, and a voxel boundary of 8. Subsequently, the optimal well-trained model is applied to the testing set, achieving excellent performance. Comparisons with other state-of-the-art methods and four data processing strategies are conducted, demonstrating that the proposed method outperforms in segmenting large-scale 3D point clouds. Overall, the proposed method performs excellently, beneficially contributing to the multi-class object segmentation from 3D tunnel point clouds with great practical potential.

RETRACTED: SHREC 2021: 3D point cloud change detection for street scenes

The rapid development of 3D acquisition devices enables us to collect billions of points in a few hours. However, the analysis of the output data is a challenging task, especially in the field of 3D point cloud change detection. In this Shape Retrieval Challenge (SHREC) track, we provide a street-scene dataset for 3D point cloud change detection. The dataset consists of 866 3D object pairs in year 2016 and 2020 from 78 large-scale street scene 3D point clouds. Our goal is to detect the changes from multi-temporal point clouds in a complex street environment. We compare three methods on this benchmark, with one handcrafted (PoChaDeHH) and the other two learning-based (HGI-CD and SiamGCN). The results show that the handcrafted algorithm has balanced performance over all classes, while learning-based methods achieve overwhelming performance but suffer from the class-imbalanced problem and may fail on minority classes. The randomized oversampling metric applied in SiamGCN can alleviate this problem. Also, different siamese network architecture in HGI-CD and SiamGCN contribute to the designing of a network for the 3D change detection task.

IMAGE TO POINT CLOUD TRANSLATION USING CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR AIRBORNE LIDAR DATA

Abstract. This study introduces a novel image to a 3D point-cloud translation method with a conditional generative adversarial network that creates a large-scale 3D point cloud. This can generate supervised point clouds observed via airborne LiDAR from aerial images. The network is composed of an encoder to produce latent features of input images, generator to translate latent features to fake point clouds, and discriminator to classify false or real point clouds. The encoder is a pre-trained ResNet; to overcome the difficulty of generating 3D point clouds in an outdoor scene, we use a FoldingNet with features from ResNet. After a fixed number of iterations, our generator can produce fake point clouds that correspond to the input image. Experimental results show that our network can learn and generate certain point clouds using the data from the 2018 IEEE GRSS Data Fusion Contest.