Point Cloud Semantic Segmentation Research Articles

Fine-Grained Point Cloud Semantic Segmentation of Complex Railway Bridge Scenes from UAVs Using Improved DGCNN

Automatic semantic segmentation of point clouds in railway bridge scenes is a crucial step in the digitization process and is required for a variety of subapplications including digital twin reconstruction and component geometric quality verification. This paper details a method for reliably and effectively segmenting point clouds acquired from complex railway bridge scenes by unmanned aerial vehicles (UAVs). The method involves segmenting seven common infrastructure elements in railway bridge point clouds using an improved DGCNN after processing low-quality point clouds from UAVs with a score-based denoising algorithm. The segmentation performance of the network is measured by averaging the intersection to union ratio between the segmentation results and the true labels of different elements, i.e., the mean intersection over union (mIoU). The proposed method is evaluated on three different scenes of railway bridges and achieved mIoU values of 99.18%, 90.76%, and 85.84%, respectively, at three levels of complexity ranging from easy to difficult. The results demonstrate that the proposed method captures the most discriminative features from low-quality point clouds, allowing for the accurate and efficient digital representation of railway bridge scenes.

Santiago urban dataset SUD: Combination of Handheld and Mobile Laser Scanning point clouds

Santiago Urban Dataset SUD is a real dataset that combines Mobile Laser Scanning (MLS) and Handheld Mobile Laser Scanning (HMLS) point clouds. The data is composed by 2 km of streets, sited in Santiago de Compostela (Spain). Point clouds undergo a manual labelling process supported by both heuristic and Deep Learning methods, resulting in the classification of eight specific classes: road, sidewalk, curb, buildings, vehicles, vegetation, poles, and others. Three PointNet++ models were trained; the first one using MLS point clouds, the second one with HMLS point clouds and the third one with both H&MLS point clouds. In order to ascertain the quality and efficacy of each Deep Learning model, various metrics were employed, including confusion matrices, precision, recall, F1-score, and IoU. The results are consistent with other state-of-the-art works and indicate that SUD is valid for comparing point cloud semantic segmentation works. Furthermore, the survey's extensive coverage and the limited occlusions indicate the potential utility of SUD in urban mobility research.

Self-Attentive Local Aggregation Learning With Prototype Guided Regularization for Point Cloud Semantic Segmentation of High-Speed Railways

Point cloud semantic segmentation for railway infrastructures is an essential step towards establishing railway digital twins. Deep learning-based methods have shown great potential in this field compared to traditional methods that rely on hand-crafted features. However, deep learning-based methods for railway point clouds still face typical challenges that need to be addressed. In this regard, we propose a novel learning framework named SALAProNet, which consists of a set of effective and concise modular solutions. The first challenge addressed is the massive data scale of railway point clouds, which makes it difficult to directly process large-scale point clouds due to memory limitations. To solve this problem, we adapt efficient random sampling in the network and propose the Self-Attentive Aggregation (SAA) module based on an attention mechanism to greatly expand the receptive field, which covers the unsampled points and successfully retains information in a high-dimensional feature space. The second challenge is fine-grained segmentation, where we propose the Local Geometry Embedding (LGE) module to embed local geometry. With the help of context information provided by SAA, the network can perform fine-grained segmentation for railway infrastructures. The third challenge is the insufficient generalization ability of the network, where we propose a Prototype Guided Regularization (PGR) method to guide the network to segment the point cloud among railways with different construction standards. This method enhances the network’s interpretability and improves its generalization ability. We have validated our proposed framework through experiments on different datasets, and it outperforms state-of-the-art approaches.

Weakly Supervised Learning for Point Cloud Semantic Segmentation With Dual Teacher

Weakly Supervised Learning for Point Cloud Semantic Segmentation With Dual Teacher

PointMatch: A consistency training framework for weakly supervised semantic segmentation of 3D point clouds

Semantic segmentation of point cloud usually relies on dense annotation that is exhausting and costly, so it attracts wide attention to investigate solutions for the weakly supervised scheme with only sparse points annotated. Existing works start from the given labels and propagate them to highly-related but unlabeled points, with the guidance of data, e.g. intra-point relation. However, it suffers from (i) the inefficient exploitation of data information, and (ii) the strong reliance on labels thus is easily suppressed when given much fewer annotations. Therefore, we propose a novel framework, PointMatch, that stands on both data and label, by applying consistency regularization to sufficiently probe information from data itself and leveraging weak labels as assistance at the same time. By doing so, meaningful information can be learned from both data and label for better representation learning, which also enables the model more robust to the extent of label sparsity. Simple yet effective, the proposed PointMatch achieves the state-of-the-art performance under various weakly-supervised schemes on both ScanNet-v2 and S3DIS datasets, especially on the settings with extremely sparse labels, e.g. surpassing SQN by 21.2% and 17.2% on the 0.01% and 0.1% setting of ScanNet-v2, respectively.

3D Point Cloud Semantic Segmentation Through Functional Data Analysis

AbstractHere, we propose a method for the semantic segmentation of 3D point clouds based on functional data analysis. For each point of a training set, a number of handcrafted features representing the local geometry around it are calculated at different scales, that is, varying the spatial extension of the local analysis. Calculating the scales at small intervals allows each feature to be accurately approximated using a smooth function and, for the problem of semantic segmentation, to be tackled using functional data analysis. We also present a step-wise method to select the optimal features to include in the model based on the calculation of the distance correlation between each feature and the response variable. The algorithm showed promising results when applied to simulated data. When applied to the semantic segmentation of a point cloud of a forested plot, the results proved better than when using a standard multiscale semantic segmentation method. The comparison with two popular deep learning models showed that our proposal requires smaller training samples sizes and that it can compete with these methods in terms of prediction.

One Class One Click: Quasi scene-level weakly supervised point cloud semantic segmentation with active learning

Reliance on vast annotations to achieve leading performance severely restricts the practicality of large-scale point cloud semantic segmentation. For the purpose of reducing data annotation costs, effective labeling schemes are developed and contribute to attaining competitive results under weak supervision strategy. Revisiting current weak label forms, we introduce One Class One Click (OCOC), a low cost yet informative quasi scene-level label, which encapsulates both point-level and scene-level annotations. An active weakly supervised framework is proposed to leverage scarce labels by involving weak supervision from both global and local perspectives. Contextual constraints are imposed by an auxiliary scene classification task, respectively based on global feature embedding and point-wise prediction aggregation, which restricts the model prediction merely to OCOC labels within a sub-cloud. Furthermore, we design a context-aware pseudo labeling strategy, which effectively supplement point-level supervisory signals subject to OCOC labels. Finally, an active learning scheme with a uncertainty measure — temporal output discrepancy is integrated to examine informative samples and provides guidance on sub-clouds query, which is conducive to quickly attaining desirable OCOC annotations and reduces the labeling cost to an extremely low extent. Extensive experimental analysis using three LiDAR benchmarks respectively collected from airborne, mobile and ground platforms demonstrates that our proposed method achieves very promising results though subject to scarce labels. It considerably outperforms genuine scene-level weakly supervised methods by up to 25% in terms of average F1 score and achieves competitive results against full supervision schemes. On terrestrial LiDAR dataset — Semantics3D, using approximately 2‱ of labels, our method achieves an average F1 score of 85.2%, which increases by 11.58% compared to the baseline model. Codes are publicly available at https://github.com/PuzoW/One-Class-One-Click.

Prototype expansion and feature calibration for few-shot point cloud semantic segmentation

Few-shot point cloud semantic segmentation has attracted increasing attention due to its applicability to real-world scenarios in which each novel class contains only a few labelled data. Although existing methods have made great progress, a great challenge still remains: inaccurate decision boundaries are induced by the scarcity of labelled data. In this paper, a Prototype Expansion and Feature Calibration (PEFC) method is proposed, which fully exploits the information contained in query data to increase the amounts of class prototypes and calibrate these prototypes. To this end, a Prototype Expansion Module (PEM) is designed, which clusters query data into pseudo prototypes and expands the set of class prototypes with pseudo prototypes that have a high similarity to the class prototypes. Then, a Feature Calibration Module (FCM) is designed to calibrate these class prototypes, which is achieved by enabling bidirectional information flow between class prototypes and query features in a task-aware manner. Extensive experiments on two benchmark datasets show that the proposed PEFC outperforms the state-of-the-art methods by a large margin in all experimental settings.

Improved point cloud semantic segmentation network based on anisotropic separable set abstraction network

Improved point cloud semantic segmentation network based on anisotropic separable set abstraction network

Deep-Learning-Based Point Cloud Semantic Segmentation: A Survey

With the rapid development of sensor technologies and the widespread use of laser scanning equipment, point clouds, as the main data form and an important information carrier for 3D scene analysis and understanding, play an essential role in the realization of national strategic needs, such as traffic scene perception, natural resource management, and forest biomass carbon stock estimation. As an important research direction in 3D computer vision, point cloud semantic segmentation has attracted more and more researchers’ attention. In this paper, we systematically outline the main research problems and related research methods in point cloud semantic segmentation and summarize the mainstream public datasets and common performance evaluation metrics. Point cloud semantic segmentation methods are classified into rule-based methods and point-based methods according to the representation of the input data. On this basis, the core ideas of each type of segmentation method are introduced, the representative and innovative algorithms of each type of method are elaborated, and the experimental results on the datasets are compared and analyzed. Finally, some promising research directions and potential tendencies are proposed.

CBFLNet: Cross-boundary feature learning for large-scale point cloud segmentation

Large-scale point cloud semantic segmentation presents a crucial yet challenging task. Current point cloud analysis approaches typically partition data into volumetric blocks, independently assigning labels to each point within these blocks. However, this strategy often compromises segmentation accuracy at block boundaries due to the isolated processing of each block, thus hindering the model’s contextual understanding. To address this issue, we present CBFLNet, an innovative semantic segmentation model that enables offset-free feature upsampling and cross-boundary feature learning. CBFLNet facilitates interaction between adjacent blocks, extending its receptive field beyond the input block. As a result, it notably mitigates errors in block boundary segmentation. CBFLNet is composed of three modules: an explicit local representation module, a symmetric sampling module, and a cross-boundary feature fusion module. The explicit local representation module is a plug-and-play module that takes two overlapping point cloud blocks as input to construct an approximate local spatial representation. In the symmetric sampling module, point features are symmetrically downsampled and upsampled, effectively avoiding feature offset caused by interpolation. Lastly, the cross-boundary feature fusion module enables cross-boundary local feature learning and multi-scale feature fusion. CBFLNet demonstrates a significant performance improvement, achieving a 0.9% mIoU increase over state-of-the-art methods on the S3DIS dataset and exhibiting competitive performance on the ScannetV2 dataset.

Semantic Point Cloud Segmentation with Deep-Learning-Based Approaches for the Construction Industry: A Survey

Point cloud learning has recently gained strong attention due to its applications in various fields, like computer vision, robotics, and autonomous driving. Point cloud semantic segmentation (PCSS) enables the automatic extraction of semantic information from 3D point cloud data, which makes it a desirable task for construction-related applications as well. Yet, only a limited number of publications have applied deep-learning-based methods to address point cloud understanding for civil engineering problems, and there is still a lack of comprehensive reviews and evaluations of PCSS methods tailored to such use cases. This paper aims to address this gap by providing a survey of recent advances in deep-learning-based PCSS methods and relating them to the challenges of the construction industry. We introduce its significance for the industry and provide a comprehensive look-up table of publicly available datasets for point cloud understanding, with evaluations based on data scene type, sensors, and point features. We address the problem of class imbalance in 3D data for machine learning, provide a compendium of commonly used evaluation metrics for PCSS, and summarize the most significant deep learning methods developed for PCSS. Finally, we discuss the advantages and disadvantages of the methods for specific industry challenges. Our contribution, to the best of our knowledge, is the first survey paper that comprehensively covers deep-learning-based methods for semantic segmentation tasks tailored to construction applications. This paper serves as a useful reference for prospective research and practitioners seeking to develop more accurate and efficient PCSS methods.

Real-Time Bucket Pose Estimation Based on Deep Neural Network and Registration Using Onboard 3D Sensor.

Real-time and accurate bucket pose estimation plays a vital role in improving the intelligence level of mining excavators, as the bucket is a crucial component of the excavator. Existing methods for bucket pose estimation are realized by installing multiple non-visual sensors. However, these sensors suffer from cumulative errors caused by loose connections and short service lives caused by strong vibrations. In this paper, we propose a method for bucket pose estimation based on deep neural network and registration to solve the large registration error problem caused by occlusion. Specifically, we optimize the Point Transformer network for bucket point cloud semantic segmentation, significantly improving the segmentation accuracy. We employ point cloud preprocessing and continuous frame registration to reduce the registration distance and accelerate the Fast Iterative Closest Point algorithm, enabling real-time pose estimation. By achieving precise semantic segmentation and faster registration, we effectively address the problem of intermittent pose estimation caused by occlusion. We collected our own dataset for training and testing, and the experimental results are compared with other relevant studies, validating the accuracy and effectiveness of the proposed method.

DCNet: Large-Scale Point Cloud Semantic Segmentation With Discriminative and Efficient Feature Aggregation

The point cloud feature aggregation, which learns discriminative features from the disordered points, plays a key role for large-scale point cloud semantic segmentation. Most previous aggregation methods are based on sampling a representative point subset, i.e., by a carefully designed point density metric, facing expensive computation cost especially for large-scale point clouds. Even though speeding up the point sampling process is studied by several recent works, but the component points in the sampled subset are uncertain and may change randomly, thus leading to corrupted geometric structure and discarded edges for representing an object. Therefore, we propose the DCNet, which consists of a fast point random sampling based encoder-decoder structure and several fully connected layers for semantic segmentation. To overcome the key feature loss caused by random down-sampling, the DCNet develops two novel local feature aggregation schemes: Double attention and Consistent constraints, to learn features that are discriminative for the challenging scenarios as above. The former considers both topological and semantic similarity of neighboring points to generate attention features for discriminating classes with similar geometric structures. The latter develops class-consistent constraints between adjacent layers in the decoder stage, to guide each point to aggregate with high-level semantic features of points belonging to the same class from the previous layer, which is beneficial for distinguishing neighboring points of the same class on the boundary. We conduct experiments and compare the proposed DCNet with existing methods on two benchmarks S3DIS and Semantic3D. Experiments show that the mean Intersection-over-Union (mIoU) of our method outperforms state-of-the-art methods by 2-3%, based on the same fast random sampling, and is also comparable to latest sampling-slower but accuracy-higher methods. That is, our method achieves the optimal speed-accuracy trade-off in the field of point cloud segmentation.

Uncertainty-Aware Point-Cloud Semantic Segmentation for Unstructured Roads

Semantic segmentation is one of the fundamental elements for achieving effective and safe autonomous driving. However, due to the irregular boundaries and variable illumination of unstructured roads, applying it in these scenarios is confronted with great challenges. To address these problems, a novel point-cloud semantic segmentation framework for unstructured roads is proposed. It contains three sections: spherical projection, an uncertainty-aware semantic segmentation network, and post-processing. Firstly, point cloud will be projected to the range image which can be processed by the 2D convolution network. Then, the uncertainty-aware semantic segmentation network is constructed. It consists of context-aware attention module and direction attention up-sampling module, which can improve the performance for segmentation of unstructured roads. In addition, a gaussian mixture model is introduced at the end of the network to predict the result with uncertainty, indicating the confidence level of the output. Finally, the segmentation result is refined during the post-processing to help filter the noise points. Experiment data from mine sites was collected to validate the performance for unstructured roads. In addition, the proposed method was also evaluated on the public unstructured dataset RELLIS-3D. The experiments show that the proposed architecture achieved 74.9% mIoU and 40.4% mIoU, which performs better than comparison methods. Additionally, the network is more robust to noisy data by achieving improvements of 4.6%-7.6% under different levels of noise data.

Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network

3D point cloud semantic segmentation aims to group all points into different semantic categories, which benefits important applications such as point cloud scene reconstruction and understanding. Existing supervised point cloud semantic segmentation methods usually require large-scale annotated point clouds for training and cannot handle new categories. While a few-shot learning method was proposed recently to address these two problems, it suffers from high computational complexity caused by graph construction and inability to learn fine-grained relationships among points due to the use of pooling operations. In this paper, we further address these problems by developing a new multi-layer transformer network for few-shot point cloud semantic segmentation. In the proposed network, the query point cloud features are aggregated based on the class-specific support features in different scales. Without using pooling operations, our method makes full use of all pixel-level features from the support samples. By better leveraging the support features for few-shot learning, the proposed method achieves the new state-of-the-art performance, with 15% less inference time, over existing few-shot 3D point cloud segmentation models on the S3DIS dataset and the ScanNet dataset. Our code is available at https://github.com/czzhang179/SCAT.

SYNTHETIC DATA GENERATION AND TESTING FOR THE SEMANTIC SEGMENTATION OF HERITAGE BUILDINGS

Abstract. Over the past decade, the use of machine learning and deep learning algorithms to support 3D semantic segmentation of point clouds has significantly increased, and their impressive results has led to the application of such algorithms for the semantic modeling of heritage buildings. Nevertheless, such applications still face several significant challenges, caused in particular by the high number of training data required during training, by the lack of specific data in the heritage building scenarios, and by the time-consuming operations to data collection and annotation. This paper aims to address these challenges by proposing a workflow for synthetic image data generation in heritage building scenarios. Specifically, the procedure allows for the generation of multiple rendered images from various viewpoints based on a 3D model of a building. Additionally, it enables the generation of per-pixel segmentation maps associated with these images. In the first part, the procedure is tested by generating a synthetic simulation of a real-world scenario using the case study of Spedale del Ceppo. In the second part, several experiments are conducted to assess the impact of synthetic data during training. Specifically, three neural network architectures are trained using the generated synthetic images, and their performance in predicting the corresponding real scenarios is evaluated.

Weakly Supervised 3D Segmentation via Receptive-Driven Pseudo Label Consistency and Structural Consistency

As manual point-wise label is time and labor-intensive for fully supervised large-scale point cloud semantic segmentation, weakly supervised method is increasingly active. However, existing methods fail to generate high-quality pseudo labels effectively, leading to unsatisfactory results. In this paper, we propose a weakly supervised point cloud semantic segmentation framework via receptive-driven pseudo label consistency and structural consistency to mine potential knowledge. Specifically, we propose three consistency contrains: pseudo label consistency among different scales, semantic structure consistency between intra-class features and class-level relation structure consistency between pair-wise categories. Three consistency constraints are jointly used to effectively prepares and utilizes pseudo labels simultaneously for stable training. Finally, extensive experimental results on three challenging datasets demonstrate that our method significantly outperforms state-of-the-art weakly supervised methods and even achieves comparable performance to the fully supervised methods.

MVPNet: A multi-scale voxel-point adaptive fusion network for point cloud semantic segmentation in urban scenes

Point cloud semantic segmentation, which contributes to scene understanding at different scales, is crucial for three-dimensional reconstruction and digital twin cities. However, current semantic segmentation methods mostly extract multi-scale features by down-sampling operations, but the feature maps only have a single receptive field at the same scale, resulting in the misclassification of objects with spatial similarity. To effectively capture the geometric features and the semantic information of different receptive fields, a multi-scale voxel-point adaptive fusion network (MVP-Net) is proposed for point cloud semantic segmentation in urban scenes. First, a multi-scale voxel fusion module with gating mechanism is designed to explore the semantic representation ability of different receptive fields. Then, a geometric self-attention module is constructed to deeply fuse fine-grained point features with coarse-grained voxel features. Finally, a pyramid decoder is introduced to aggregate context information at different scales for enhancing feature representation. The proposed MVP-Net was evaluated on three datasets, Toronto3D, WHU-MLS, and SensatUrban, and achieved superior performance in comparison to the state-of-the-art (SOTA) methods. For the public Toronto3D and SensatUrban datasets, our MVP-Net achieved a mIoU of 84.14% and 59.40%, and an overall accuracy of 98.12% and 93.30%, respectively.

3D data-augmentation methods for semantic segmentation of tomato plant parts.

3D semantic segmentation of plant point clouds is an important step towards automatic plant phenotyping and crop modeling. Since traditional hand-designed methods for point-cloud processing face challenges in generalisation, current methods are based on deep neural network that learn to perform the 3D segmentation based on training data. However, these methods require a large annotated training set to perform well. Especially for 3D semantic segmentation, the collection of training data is highly labour intensitive and time consuming. Data augmentation has been shown to improve training on small training sets. However, it is unclear which data-augmentation methods are effective for 3D plant-part segmentation. In the proposed work, five novel data-augmentation methods (global cropping, brightness adjustment, leaf translation, leaf rotation, and leaf crossover) were proposed and compared to five existing methods (online down sampling, global jittering, global scaling, global rotation, and global translation). The methods were applied to PointNet++ for 3D semantic segmentation of the point clouds of three cultivars of tomato plants (Merlice, Brioso, and Gardener Delight). The point clouds were segmented into soil base, stick, stemwork, and other bio-structures. Among the data augmentation methods being proposed in this paper, leaf crossover indicated the most promising result which outperformed the existing ones. Leaf rotation (around Z axis), leaf translation, and cropping also performed well on the 3D tomato plant point clouds, which outperformed most of the existing work apart from global jittering. The proposed 3D data augmentation approaches significantly improve the overfitting caused by the limited training data. The improved plant-part segmentation further enables a more accurate reconstruction of the plant architecture.