Point Cloud Semantic Segmentation Research Articles

Bilinear Distance Feature Network for Semantic Segmentation in PowerLine Corridor Point Clouds.

Semantic segmentation of target objects in power transmission line corridor point cloud scenes is a crucial step in powerline tree barrier detection. The massive quantity, disordered distribution, and non-uniformity of point clouds in power transmission line corridor scenes pose significant challenges for feature extraction. Previous studies have often overlooked the core utilization of spatial information, limiting the network's ability to understand complex geometric shapes. To overcome this limitation, this paper focuses on enhancing the deep expression of spatial geometric information in segmentation networks and proposes a method called BDF-Net to improve RandLA-Net. For each input 3D point cloud data, BDF-Net first encodes the relative coordinates and relative distance information into spatial geometric feature representations through the Spatial Information Encoding block to capture the local spatial structure of the point cloud data. Subsequently, the Bilinear Pooling block effectively combines the feature information of the point cloud with the spatial geometric representation by leveraging its bilinear interaction capability thus learning more discriminative local feature descriptors. The Global Feature Extraction block captures the global structure information in the point cloud data by using the ratio between the point position and the relative position, so as to enhance the semantic understanding ability of the network. In order to verify the performance of BDF-Net, this paper constructs a dataset, PPCD, for the point cloud scenario of transmission line corridors and conducts detailed experiments on it. The experimental results show that BDF-Net achieves significant performance improvements in various evaluation metrics, specifically achieving an OA of 97.16%, a mIoU of 77.48%, and a mAcc of 87.6%, which are 3.03%, 16.23%, and 18.44% higher than RandLA-Net, respectively. Moreover, comparisons with other state-of-the-art methods also verify the superiority of BDF-Net in point cloud semantic segmentation tasks.

MP-DGCNN for the semantic segmentation of Chinese ancient building point clouds

Point cloud semantic segmentation is a key step in the scan-to-HBIM process. In order to reduce the information in the process of DGCNN, this paper proposes a Mix Pooling Dynamic Graph Convolutional Neural Network (MP-DGCNN) for the segmentation of ancient architecture point clouds. The proposed MP-DGCNN differs from DGCNN mainly in two aspects: (1) to more comprehensively characterize the local topological structure of points, the edge features are redefined, and distance and neighboring points are added to the original edge features; (2) based on a Multilayer Perceptron (MLP), an internal feature adjustment mechanism is established, and a learnable mix pooling operator is designed by fusing adaptive pooling, max pooling, average pooling, and aggregation pooling, to learn local graph features from the point cloud topology. To verify the proposed algorithm, experiments are conducted on the Qutan Temple point cloud dataset, and the results show that compared with PointNet, PointNet++, DGCNN, GACNet and LDGCNN, the MP-DGCNN segmentation network achieves the highest OA, mIOU and mAcc, reaching 90.19%,65.34% and 79.41%, respectively.

Semantic segmentation of point clouds of ancient buildings based on weak supervision

Semantic segmentation of point clouds of ancient buildings plays an important role in Historical Building Information Modelling (HBIM). As the annotation task of point cloud of ancient architecture is characterised by strong professionalism and large workload, which greatly restricts the application of point cloud semantic segmentation technology in the field of ancient architecture, therefore, this paper launches a research on the semantic segmentation method of point cloud of ancient architecture based on weak supervision. Aiming at the problem of small differences between classes of ancient architectural components, this paper introduces a self-attention mechanism, which can effectively distinguish similar components in the neighbourhood. Moreover, this paper explores the insufficiency of positional encoding in baseline and constructs a high-precision point cloud semantic segmentation network model for ancient buildings—Semantic Query Network based on Dual Local Attention (SQN-DLA). Using only 0.1% of the annotations in our homemade dataset and the Architectural Cultural Heritage (ArCH) dataset, the mean Intersection over Union (mIoU) reaches 66.02% and 58.03%, respectively, which is an improvement of 3.51% and 3.91%, respectively, compared to the baseline.

Memory-Augmented 3D Point Cloud Semantic Segmentation Network for Intelligent Mining Shovels.

The semantic segmentation of the 3D operating environment represents the key to intelligent mining shovels' autonomous digging and loading operation. However, the complexity of the operating environment of intelligent mining shovels presents challenges, including the variety of scene targets and the uneven number of samples. This results in low accuracy of 3D semantic segmentation and reduces the autonomous operation accuracy of the intelligent mine shovels. To solve these issues, this paper proposes a 3D point cloud semantic segmentation network based on memory enhancement and lightweight attention mechanisms. This model addresses the challenges of an uneven number of sampled scene targets, insufficient extraction of key features to reduce the semantic segmentation accuracy, and an insufficient lightweight level of the model to reduce deployment capability. Firstly, we investigate the memory enhancement learning mechanism, establishing a memory module for key semantic features of the targets. Furthermore, we address the issue of forgetting non-dominant target point cloud features caused by the unbalanced number of samples and enhance the semantic segmentation accuracy. Subsequently, the channel attention mechanism is studied. An attention module based on the statistical characteristics of the channel is established. The adequacy of the expression of the key features is improved by adjusting the weights of the features. This is done in order to improve the accuracy of semantic segmentation further. Finally, the lightweight mechanism is studied by adopting the deep separable convolution instead of conventional convolution to reduce the number of model parameters. Experiments demonstrate that the proposed method can improve the accuracy of semantic segmentation in the 3D scene and reduce the model's complexity. Semantic segmentation accuracy is improved by 7.15% on average compared with the experimental control methods, which contributes to the improvement of autonomous operation accuracy and safety of intelligent mining shovels.

Domain knowledge-enhanced region growing framework for semantic segmentation of bridge point clouds

Utilising domain knowledge (DK) to semantically segment bridge point clouds has attracted growing research interest. However, current approaches are often tailored to specific bridges, limiting their general applicability. To address this problem, this paper introduces a DK-enhanced Region Growing (DKRG) framework for point cloud semantic segmentation of reinforced concrete (RC) girder bridges. Inspired by the vertical layout characteristics of bridges, the generation of DK-based point features from Finite Element Analysis (FEA) is first proposed. Then, DKRG is employed to segment bridge components from substructures to superstructures by leveraging an “easy-to-difficult” strategy. Validation results demonstrate the effectiveness of our method, achieving the lowest mean Intersection over Union (mIoU) of 95.47% for the entire bridge and 93.44% for different component types. This study provides a new DK-based framework for semantic segmentation of RC girder bridges and sheds new light on using FEA-generated point features.

RailPC: A large‐scale railway point cloud semantic segmentation dataset

AbstractSemantic segmentation in the context of 3D point clouds for the railway environment holds a significant economic value, but its development is severely hindered by the lack of suitable and specific datasets. Additionally, the models trained on existing urban road point cloud datasets demonstrate poor generalisation on railway data due to a large domain gap caused by non‐overlapping special/rare categories, for example, rail track, track bed etc. To harness the potential of supervised learning methods in the domain of 3D railway semantic segmentation, we introduce RailPC, a new point cloud benchmark. RailPC provides a large‐scale dataset with rich annotations for semantic segmentation in the railway environment. Notably, RailPC contains twice the number of annotated points compared to the largest available mobile laser scanning (MLS) point cloud dataset and is the first railway‐specific 3D dataset for semantic segmentation. It covers a total of nearly 25 km railway in two different scenes (urban and mountain), with 3 billion points that are finely labelled as 16 most typical classes with respect to railway, and the data acquisition process is completed in China by MLS systems. Through extensive experimentation, we evaluate the performance of advanced scene understanding methods on the annotated dataset and present a synthetic analysis of semantic segmentation results. Based on our findings, we establish some critical challenges towards railway‐scale point cloud semantic segmentation. The dataset is available at https://github.com/NNU‐GISA/GISA‐RailPC, and we will continuously update it based on community feedback.

DAAL-WS: A weakly-supervised method integrated with data augmentation and active learning strategies for MLS point cloud semantic segmentation

Mobile laser scanning (MLS) point clouds have increasingly been a significant data source for acquiring accurate three-dimensional (3D) semantic information from complex scenes. However, most current point cloud semantic segmentation methods heavily depend on a huge number of manually labeled training samples, which is labor-intensive and time-consuming. To address the above challenge, a weakly-supervised MLS point cloud semantic segmentation method integrated with data augmentation and active learning strategies is proposed (termed DAAL-WS). By taking advantage of our previously presented multi-branch weakly-supervised network (WSPointNet) in weakly supervisory signals and ensemble predictions, the DAAL-WS integrates WSPointNet with two essential components, i.e., an elevation-calibrated Mix3D (EC-Mix3D) data augmentation strategy and a point-level ensemble prediction-based (PLEP) active learning strategy. Specifically, the EC-Mix3D data augmentation strategy leverages elevation information to calibrate sub-point clouds and generates new contextual scenes by mixing the elevation-calibrated sub-point clouds, thereby augmenting the training point cloud distribution. Designed to reduce labeling redundancy, the PLEP active learning strategy selects the most important labeled points for model training. This strategy first measures the uncertainty for each unlabeled point by ensemble predictions and then employs a feature-distance suppression module to select the significant and discriminating unlabeled points for manual labeling. The proposed DAAL-WS method was evaluated on three public MLS datasets, including Toronto3D, Paris-Lille-3D, and WHU-MLS datasets, on which DAAL-WS obtained a competitive performance over fully-supervised baselines using only 0.015 %, 0.03 %, and 0.03 % labeled points, with mean Intersection over Union (mIoU) scores of 81.91 %, 82.59 %, and 60.36 %, respectively.

TCFAP-Net: Transformer-based Cross-feature Fusion and Adaptive Perception Network for large-scale point cloud semantic segmentation

Point cloud semantic segmentation is an ingredient in understanding real-world scenes. Most existing approaches perform poorly on scene boundaries and struggle with recognizing objects of different scales. In this paper, we propose a novel framework that incorporates Transformer into the U-Net architecture for inferring pointwise semantics. Specifically, the Transformer-based cross-feature fusion module is designed first to employ geometric and semantic information to learn feature offsets to overcome the border ambiguity of segmentation results, and then it utilizes the Transformer to learn cross-feature enhanced and fused encoder features. Additionally, to facilitate the overall network’s structure-to-detail perception capabilities, the adaptive perception module is designed, which employs cross-attention to adaptively allocate weights to encoder features at varying resolutions, establishing long-range contextual dependencies. Ablation studies validate the individual contributions of our module design choices. Compared with the existing competitive methods, our approach achieves state-of-the-art performance and exhibits superior results on benchmarks. Code is available at https://github.com/xiluo-cug/TCFAP-Net.

Automated BIM-to-scan point cloud semantic segmentation using a domain adaptation network with hybrid attention and whitening (DawNet)

Deep learning-based point cloud semantic segmentation facilitates scene understanding and BIM modelling, but its success requires vast amount of labelled point clouds, which is laborious and time-consuming. To reduce point cloud annotation cost, researchers attempt to leverage synthetic point clouds, but the domain gap between synthetic and real point clouds deteriorates the segmentation accuracy. To address this issue, this study develops a BIM-to-Scan point cloud semantic segmentation approach to mitigate the domain gap between BIM and real point clouds, improving the segmentation performance on real point clouds. To this end, this study starts by proposing a BIM-based point cloud generation method, which uses FME and BIM models to automatically generate and label synthetic point clouds, decreasing the annotation cost. To fill the domain gap, a DawNet is invented by integrating a domain adaptation network with ZCA whitening operation and hybrid attention mechanism. Specifically, ResPointNet++ is used to extract geometric features and execute the segmentation task, which is then combined with a domain discriminator to perform domain adversarial learning, aligning the data distribution of BIM and real point clouds. To improve the performance of the DawNet, a residual learning block with whitening and a hybrid attention module are designed. These two modules help extract and exploit domain-invariant features to boost the generalisation and segmentation performance of the DawNet. Finally comprehensive experiments show that the proposed BIM-based method spends 0.5 person-hours to generate 0.45 billion labelled BIM point clouds, and that the developed DawNet achieves 17% and 11% more mIoU than ResPointNet++ and DANN. The ablation study also confirms the effectiveness of the hybrid attention module and ZCA whitening operation.

Hierarchical SVM for Semantic Segmentation of 3D Point Clouds for Infrastructure Scenes

The incorporation of building information modeling (BIM) has brought about significant advancements in civil engineering, enhancing efficiency and sustainability across project life cycles. The utilization of advanced 3D point cloud technologies such as laser scanning extends the application of BIM, particularly in operations and maintenance, prompting the exploration of automated solutions for labor-intensive point cloud modeling. This paper presents a demonstration of supervised machine learning—specifically, a support vector machine—for the analysis and segmentation of 3D point clouds, which is a pivotal step in 3D modeling. The point cloud semantic segmentation workflow is extensively reviewed to encompass critical elements such as neighborhood selection, feature extraction, and feature selection, leading to the development of an optimized methodology for this process. Diverse strategies are implemented at each phase to enhance the overall workflow and ensure resilient results. The methodology is then evaluated using diverse datasets from infrastructure scenes of bridges and compared with state-of-the-art deep learning models. The findings highlight the effectiveness of supervised machine learning techniques at accurately segmenting 3D point clouds, outperforming deep learning models such as PointNet and PointNet++ with smaller training datasets. Through the implementation of advanced segmentation techniques, there is a partial reduction in the time required for 3D modeling of point clouds, thereby further enhancing the efficiency and effectiveness of the BIM process.

A point cloud semantic segmentation method with augmented local geometry

With the development of 3D sensors, complex spatial structures can be established from point cloud data. Accurate semantic segmentation of 3D point clouds is beneficial for tasks such as robotic scene understanding. To extract the geometric features of the point cloud, most prior works use the xyz coordinates to potentially learn the shapes of 3D objects, which is not insufficient to handle complex scenes. In this paper, a Local Geometry Encoding (LGE) module is proposed to provide local normal information and express 3D shapes explicitly. Furthermore, considering that most prior methods use a single decoder to recover from abstract features, a Multi-Decoder (MD) ensemble module is proposed to restore multiple segmentation results from various scale feature maps in the encoder, and then a multi-attention fusion mechanism is utilized to produce single segmentation results. Benefiting from the local geometry encoding module and Multi-Decoder ensemble module, our network performs 6-fold cross-validation on the S3DIS dataset and achieves 75% mIoU with fewer parameters.

Unified 3D and 4D Panoptic Segmentation via Dynamic Shifting Networks.

With the rapid advances in autonomous driving, it becomes critical to equip its sensing system with more holistic 3D perception. However, widely explored tasks like 3D detection or point cloud semantic segmentation focus on parsing either the objects or scenes. In this work, we propose to address the challenging task of LiDAR-based Panoptic Segmentation, which aims to parse both objects and scenes in a unified manner. In particular, we propose Dynamic Shifting Network (DS-Net), which serves as an effective panoptic segmentation framework in the point cloud realm. DS-Net features a dynamic shifting module for complex LiDAR point cloud distributions. We present an efficient learnable clustering module, dynamic shifting, which adapts kernel functions for different instances. To further explore the temporal information, we extend the single-scan processing framework to its temporal version, 4D-DS-Net, for the task of 4D Panoptic Segmentation, where the same instance across multiple frames should be given the same ID prediction. Instead of naively appending a tracking module to DS-Net, we propose to solve the 4D panoptic segmentation in a more unified way. Specifically, 4D-DS-Net first constructs 4D data volume by aligning consecutive LiDAR scans, upon which the temporally unified instance clustering is performed to obtain the final results. Extensive experiments on two large-scale autonomous driving LiDAR datasets, SemanticKITTI and Panoptic nuScenes, are conducted to demonstrate the effectiveness and superior performance of the proposed solution.

Semantic segmentation of large-scale point cloud scenes via dual neighborhood feature and global spatial-aware

As a core task in 3D scene information extraction, point cloud semantic segmentation is crucial for understanding 3D scenes and environmental perception. While extracting local geometric structural features from point clouds, existing research often overlooks the long-range dependencies present in the scene, making it challenging to fully uncover the long-range contextual features hidden within point clouds. On this basis, we propose a segmentation algorithm (DG-Net) that integrates dual neighborhood features with global spatial-aware. Initially, the local structure information encoding module is designed to learn about local geometric shapes by encoding spatial position and directional features, thus supplementing structural information. Subsequently, a dual neighborhood features complementary module is introduced to merge the geometric structural and semantic features within local neighborhoods, learning local dependencies and capturing distinguishable local contextual features. Finally, these features are relayed to a global spatial-aware module equipped with a gated unit, which dynamically adjusts the weights of features at different stages, effectively modeling long-range dependencies between local structures and finely extracting long-range contextual features. We conducted experiments on benchmark datasets of point cloud scenes, and both quantitative and qualitative results demonstrate that our algorithm can accurately identify small-scale objects with complex geometric structures within scenes, surpassing other mainstream networks in segmentation performance. The mIoU on the S3DIS, Toronto3D, and SensatUrban datasets are 71.9 %, 82.1 %, and 59.8 %, respectively.

A large-scale point cloud semantic segmentation neural network based on long-range contextual dependencies enhancement

ABSTRACT This letter presents a long-range contextual dependency enhanced network (LCDE-Net) for semantic segmentation of large-scale point cloud, which employs a U-shaped framework. Firstly, point clouds are subsampled with grid sampling and fed into convolutional layers to learn more representative local features of points. Then global and local encoders (GLE) are designed to exploit long-range contextual dependencies and local features simultaneously. The core of GLE consists of two parts: global feature enhancement (GFE) module and feature channel modulation (FCM) module. Secondly, through decoder layers, the encoded features are upsampled through the nearest-neighbour interpolation and aggregated with the intermediate encoded features by skip connections to capture multi-scale discriminative features for semantic segmentation of point cloud. Finally, via Fully Connection layer and Softmax classifier, each point’s label is assigned. Two different benchmark datasets are conducted to evaluate the performance of the proposed method, Experimental results report that the proposed LCDE-Net achieves 78.6% in terms of mean intersection over union (mIoU) on Semantic3D, and 68.2% on S3DIS, which is the highest among the comparison methods. The code of LCDE-Net is available at https://github.com/xrzmyz/LCDE-Net.

Machine and Deep Learning Implementations for Heritage Building Information Modelling: A Critical Review of Theoretical and Applied Research

Research domain and Problem: HBIM modelling from point cloud data has become a crucial research topic in the last decade since it is potentially considered as the central data model paving the way for the digital heritage practice beyond digitization. Reality Capture technologies such as terrestrial laser scanning, drone-mounted LiDAR sensors and photogrammetry enable the reality capture with a sub-millimetre accurate point cloud file that can be used as a reference file for Heritage Building Information Modelling (HBIM). However, HBIM modelling from the point cloud data of heritage buildings is mainly manual, error-prone, and time-consuming. Furthermore, image processing techniques are insufficient for classification and segmentation of point cloud data to speed up and enhance the current workflow for HBIM modelling. Due to the challenges and bottlenecks in the scan-to-HBIM process, which is commonly criticized as complex with its bespoke requirements, semantic segmentation of point clouds is gaining popularity in the literature. Research Aim and Methodology: Therefore, this paper aims to provide a thorough critical review of Machine Learning and Deep Learning methods for point cloud segmentation, classification, and BIM geometry automation for cultural heritage case study applications. Research findings: This paper files the challenges of HBIM practice and the opportunities for semantic point cloud segmentation found across academic literature in the last decade. Beyond definitions and basic occurrence statistics, this paper discusses the success rates and implementation challenges of machine and deep learning classification methods. Research value and contribution: This paper provides a holistic review of point cloud segmentation and its potential for further development and application in the Cultural Heritage sector. The critical analysis provides insight into the current state-of-the-art methods and advises on their suitability for HBIM projects. The review has identified highly original threads of research, which hold the potential to significantly influence practice and further applied research.

Material augmented semantic segmentation of point clouds for building elements

AbstractPoint clouds are utilized to enable automated engineering applications for their ability to represent spatial geometry. However, they inherently lack detailed surface textures, posing challenges in differentiating objects at the texture level. Hence, this study introduces a 2D–3D fusing approach, leveraging material properties recognized from registered images as an augmented feature to enhance deep learning methods for the segmentation of building elements within point clouds. The proposed method was evaluated quantitatively on a 3D indoor data set with an implementation in an office room. The results are promising, showing improvement in recognition performance, particularly for objects with similar geometry but having different material properties. For instance, the segmentation of boards increased by 70.87%, and doors improved by 41.06% using the PointNet architecture. This enhanced segmentation not only reduces the time for interpreting point clouds but also has the potential to benefit downstream applications such as Scan‐to‐building information modeling (BIM), as defining regions for objects is essential.

Learnable scene prior for point cloud semantic segmentation

Learnable scene prior for point cloud semantic segmentation

PSTNet: Transformer for aggregating neighborhood features in 3D point cloud semantic segmentation of eggplant plants

Improving the quality of plant point cloud data and achieving precise segmentation is essential for effective plant phenotyping analysis and plant breeding. To ensure data quality and accuracy of segmentation, we employed a self-developed three-dimensional scanning device based on binocular vision to acquire eggplant plant point clouds with high spatial resolution. And we improved down-sampling algorithm Octree Farthest Point Sampling (OFPS) to process the original point cloud data. Furthermore, we proposed a neural network called Plant Segmentation Transformer Network (PSTNet) to achieve semantic segmentation of high spatial resolution eggplant plant datasets. PSTNet comprises the following components: (i) a Neighborhood Feature Aggregator (NPA) for storing and aggregating local neighborhood features of input points, and (ii) a cascaded Point Self-Attention module (PSA) for capturing contextual information. Experimental results demonstrate the remarkable performance of PSTNet in semantic segmentation tasks, with IoU, Precision, Recall, F1-score, and Accuracy values of 92.20 %, 95.30 %, 95.57 %, 95.43 %, and 95.15 %, respectively. Additionally, compared to the Point Cloud Transformer (PCT), which exhibited the second-best performance, PSTNet achieved improvements of 4.37, 1.47, 2.87, 2.17, and 4.22 percentage points in the abovementioned metrics. This method achieves high-precision segmentation of plant point clouds, expands the semantic segmentation method of plants, and lays a solid foundation for plant phenotypic analysis.

Semantic segmentation of large-scale point clouds by integrating attention mechanisms and transformer models

In the current field of point cloud processing, semantic segmentation of large-scale point clouds remains a challenging problem. Traditional methods often underperform when faced with the complexity and density variations present in large-scale point cloud data. This study introduces an innovative model designed for semantic segmentation of large-scale point clouds. The model integrates a CNN-Transformer-based Context Aggregation Module with a Slot Attention mechanism to enhance the understanding of entity relationships and improve semantic segmentation performance. Furthermore, by flexibly adjusting the weight parameters within the loss function, we have optimized the training process, increasing its flexibility. In our experiments, we compared the proposed model with 55 classical and contemporary point cloud semantic segmentation models, evaluating semantic segmentation performance, computational resource consumption, inference time, and the number of model parameters. The results show a significant improvement in performance on the S3DIS and Semantic3D datasets, specifically achieving an mIoU of 71.53% and an OA of 93.92%. Ablation studies were conducted to further ascertain the contribution of each module to the overall performance of the model. This study presents an innovative approach to overcoming the challenges of large-scale point cloud semantic segmentation, offering significant contributions to the advancement of point cloud processing.

PointMM: Point Cloud Semantic Segmentation CNN under Multi-Spatial Feature Encoding and Multi-Head Attention Pooling

For the actual collected point cloud data, there are widespread challenges such as semantic inconsistency, density variations, and sparse spatial distribution. A network called PointMM is developed in this study to enhance the accuracy of point cloud semantic segmentation in complex scenes. The main contribution of PointMM involves two aspects: (1) Multi-spatial feature encoding. We leverage a novel feature encoding module to learn multi-spatial features from the neighborhood point set obtained by k-nearest neighbors (KNN) in the feature space. This enhances the network’s ability to learn the spatial structures of various samples more finely and completely. (2) Multi-head attention pooling. We leverage a multi-head attention pooling module to address the limitations of symmetric function-based pooling, such as maximum and average pooling, in terms of losing detailed feature information. This is achieved by aggregating multi-spatial and attribute features of point clouds, thereby enhancing the network’s ability to transmit information more comprehensively and accurately. Experiments on publicly available point cloud datasets S3DIS and ISPRS 3D Vaihingen demonstrate that PointMM effectively learns features at different levels, while improving the semantic segmentation accuracy of various objects. Compared to 12 state-of-the-art methods reported in the literature, PointMM outperforms the runner-up by 2.3% in OA on the ISPRS 3D Vaihingen dataset, and achieves the third best performance in both OA and MioU on the S3DIS dataset. Both achieve a satisfactory balance between OA, F1, and MioU.