Point Cloud Semantic Segmentation Research Articles

PointMM: Point Cloud Semantic Segmentation CNN under Multi-Spatial Feature Encoding and Multi-Head Attention Pooling

For the actual collected point cloud data, there are widespread challenges such as semantic inconsistency, density variations, and sparse spatial distribution. A network called PointMM is developed in this study to enhance the accuracy of point cloud semantic segmentation in complex scenes. The main contribution of PointMM involves two aspects: (1) Multi-spatial feature encoding. We leverage a novel feature encoding module to learn multi-spatial features from the neighborhood point set obtained by k-nearest neighbors (KNN) in the feature space. This enhances the network’s ability to learn the spatial structures of various samples more finely and completely. (2) Multi-head attention pooling. We leverage a multi-head attention pooling module to address the limitations of symmetric function-based pooling, such as maximum and average pooling, in terms of losing detailed feature information. This is achieved by aggregating multi-spatial and attribute features of point clouds, thereby enhancing the network’s ability to transmit information more comprehensively and accurately. Experiments on publicly available point cloud datasets S3DIS and ISPRS 3D Vaihingen demonstrate that PointMM effectively learns features at different levels, while improving the semantic segmentation accuracy of various objects. Compared to 12 state-of-the-art methods reported in the literature, PointMM outperforms the runner-up by 2.3% in OA on the ISPRS 3D Vaihingen dataset, and achieves the third best performance in both OA and MioU on the S3DIS dataset. Both achieve a satisfactory balance between OA, F1, and MioU.

Towards accurate correspondence between BIM and construction using high-dimensional point cloud feature tensor

The correspondence between BIM and construction instances is crucial to construction management. However, spatial deviation and geometric heterogeneity between BIM and construction point clouds pose great challenges. This paper establishes high-dimensional point cloud feature tensor to devise a point cloud semantic segmentation network and an incremental point-to-point correspondence estimation strategy against spatial deviation and geometric heterogeneity. In the construction of the stadium for the 31st Summer World University Games, the method achieved semantic segmentation accuracy (OA) of 93.8% - 99.9% and reduced BIM-and-construction correspondence error from 16 cm to 3 cm, and automatically documented four-phase construction progresses of 38 317 instances with progress monitoring error of 1%. These results demonstrate the effectiveness and applicability of the proposed method in BIM-to-construction semantic transfer and BIM-and-construction instance correspondence for large complex buildings. Future research will design transfer learning networks to achieve fully BIM-driven semantic understanding and knowledge mining for intelligent construction.

Retrieval-and-alignment based large-scale indoor point cloud semantic segmentation

Retrieval-and-alignment based large-scale indoor point cloud semantic segmentation

Less Is More: Label Recommendation for Weakly Supervised Point Cloud Semantic Segmentation

Weak supervision has proven to be an effective strategy for reducing the burden of annotating semantic segmentation tasks in 3D space. However, unconstrained or heuristic weakly supervised annotation forms may lead to suboptimal label efficiency. To address this issue, we propose a novel label recommendation framework for weakly supervised point cloud semantic segmentation. Distinct from pre-training and active learning, the label recommendation framework consists of three stages: inductive bias learning, recommendations for points to be labeled, and point cloud semantic segmentation learning. In practice, we first introduce the point cloud upsampling task to induct inductive bias from structural information. During the recommendation stage, we present a cross-scene clustering strategy to generate centers of clustering as recommended points. Then we introduce a recommended point positions attention module LabelAttention to model the long-range dependency under sparse annotations. Additionally, we employ position encoding to enhance the spatial awareness of semantic features. Throughout the framework, the useful information obtained from inductive bias learning is propagated to subsequent semantic segmentation networks in the form of label positions. Experimental results demonstrate that our framework outperforms weakly supervised point cloud semantic segmentation methods and other methods for labeling efficiency on S3DIS and ScanNetV2, even at an extremely low label rate.

Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~ 6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.

Large-scale Point Cloud Segmentation based on Multi-feature Local Enhanced Fusion

This paper introduces a framework for large-scale 3D point cloud semantic segmentation - the MLEF-Net model. The model aims to improve the segmentation accuracy of large-scale point clouds by innovatively combining Manhattan distance-based KNN neighborhood search with feature aggregation techniques. This approach uniquely handles spatial, color, and normal vector attributes, thereby improving the segmentation results. The superiority of the model is validated through comprehensive testing on the SemanticKITTI and nuScenes datasets, demonstrating its potential to enhance point cloud segmentation through advanced feature fusion strategies.

Research on deep learning-based point cloud semantic segmentation for offshore drilling platforms

Offshore drilling platforms (ODP) require effective deformation monitoring and damage detection in the wellhead space for operational maintenance. LiDAR technology, replacing traditional methods, plays a key role in damage detection through 3D reconstruction, and the accurate semantic segmentation of point clouds is fundamental to this process. Following advancements in deep learning, neural network models have emerged as effective tools for the semantic segmentation of point clouds in a range of settings. ODP point cloud data, characterized by large volumes, high density, class imbalance, and complex structures, present unique challenges for feature extraction. To address these challenges, this paper proposed a multi-scale bounding box (MSB-Box) feature extraction strategy based on spatial relation dynamic encoding (SRD-encoding) and a multi-scale point cloud down-sampling (MPDS) strategy to enhance computational efficiency. Utilizing these proposed strategies, we developed an ODP-Net, a deep learning neural network based on the attention mechanism. The ODP-Net's semantic segmentation performance was evaluated using a self-constructed ODP scene point cloud dataset and the S3DIS dataset. The results demonstrate that ODP-Net, based on MPDS and MSB-Box, outperforms other networks in diverse scenes, confirming its high accuracy and wide applicability in the semantic segmentation of point cloud data in ODP scenes.

PVI-Net: Point–Voxel–Image Fusion for Semantic Segmentation of Point Clouds in Large-Scale Autonomous Driving Scenarios

In this study, we introduce a novel framework for the semantic segmentation of point clouds in autonomous driving scenarios, termed PVI-Net. This framework uniquely integrates three different data perspectives—point clouds, voxels, and distance maps—executing feature extraction through three parallel branches. Throughout this process, we ingeniously design a point cloud–voxel cross-attention mechanism and a multi-perspective feature fusion strategy for point images. These strategies facilitate information interaction across different feature dimensions of perspectives, thereby optimizing the fusion of information from various viewpoints and significantly enhancing the overall performance of the model. The network employs a U-Net structure and residual connections, effectively merging and encoding information to improve the precision and efficiency of semantic segmentation. We validated the performance of PVI-Net on the SemanticKITTI and nuScenes datasets. The results demonstrate that PVI-Net surpasses most of the previous methods in various performance metrics.

Indoor point cloud semantic segmentation based on direction perception and hole sampling

AbstractMost existing point cloud segmentation methods ignore directional information when extracting neighbourhood features. Those methods are ineffective in extracting point cloud neighbourhood features because the point cloud data is not uniformly distributed and is restricted by the size of the convolution kernel. Therefore, we take into account both multiple directions and hole sampling (MDHS). First, we execute spherically sparse sampling with directional encoding in the surrounding domain for every point inside the data to increase the local perceptual field. The data input is the basic geometric features. We use the graph convolutional neural network to conduct the maximisation of point cloud characteristics in a local neighbourhood. Then the more representative local point features are automatically weighted and fused by an attention pooling layer. Finally, spatial attention is added to increase the connection between remote points, and then the segmentation accuracy is improved. Experimental results show that the OA and mIoU are 1.3% and 4.0% higher than the method PointWeb and 0.6% and 0.7% higher than the baseline method RandLA‐Net. For the indoor point cloud semantic segmentation, the segmentation effect of the proposed network is superior to other methods.

Semantic segmentation of raw multispectral laser scanning data from urban environments with deep neural networks

Real-time semantic segmentation of point clouds has increasing importance in applications related to 3D city modelling and mapping, automated inventory of forests, autonomous driving and mobile robotics. Current state-of-the-art point cloud semantic segmentation methods rely heavily on the availability of 3D laser scanning data. This is problematic in regards of low-latency, real-time applications that use data from high-precision mobile laser scanners, as those are typically 2D line scanning devices. In this study, we experiment with real-time semantic segmentation of high-density multispectral point clouds collected from 2D line scanners in urban environments using encoder - decoder convolutional neural network architectures. We introduce a rasterized multi-scan input format that can be constructed exclusively from the raw (non-georeferenced profiles) 2D laser scanner measurement stream without odometry information. In addition, we investigate the impact of multispectral data on the segmentation accuracy. The dataset used for training, validation and testing was collected with multispectral FGI AkhkaR4-DW backpack laser scanning system operating at the wavelengths of 905 nm and 1550 nm, and consists in total of 228 million points (39 583 scans). The data was divided into 13 classes that represent various targets in urban environments. The results show that the increased spatial context of the multi-scan format improves the segmentation performance on the single-wavelength lidar dataset from 45.4 mIoU (a single scan) to 62.1 mIoU (24 consecutive scans). In the multispectral point cloud experiments we achieved a 71 % and 28 % relative increase in the segmentation mIoU (43.5 mIoU) as compared to the purely single-wavelength reference experiments, in which we achieved 25.4 mIoU (905 nm) and 34.1 mIoU (1550 nm). Our findings show that it is possible to semantically segment 2D line scanner data with good results by combining consecutive scans without the need for odometry information. The results also serve as motivation for developing multispectral mobile laser scanning systems that can be used in challenging urban surveys.

Integrating Contextual Information and Attention Mechanisms with Sparse Convolution for the Extraction of Internal Objects within Buildings from Three-Dimensional Point Clouds

Deep learning-based point cloud semantic segmentation has gained popularity over time, with sparse convolution being the most prominent example. Although sparse convolution is more efficient than regular convolution, it comes with the drawback of sacrificing global context information. To solve this problem, this paper proposes the OcspareNet network, which uses sparse convolution as the backbone and captures global contextual information using the offset attention module and context aggregation module. The offset attention module improves the network’s capacity to obtain global contextual information about the point cloud. The context aggregation module utilizes contextual information in the training and testing phases, which increases the network’s capacity to discern the overall structure and successfully improves the network’s capacity and the accuracy of the difficult-scene segmentation category. Compared to the state-of-the-art (SOTA) models, our model has a smaller parameter count and achieves higher accuracy on challenging segmentation categories such as ‘pictures’, ‘counters’, and ‘desks’ in the ScanNetV2 dataset, with IoU scores of 41.1%, 70.3%, and 72.5%, respectively. Furthermore, ablation experiments confirmed the efficacy of our designed modules.

Dual fusion network for semantic segmentation of point clouds

The current networks for point cloud segmentation fail to fully consider the connections between points in each neighborhood, which makes the final segmentation capability underperform. To obtain richer local relationships between points, this paper proposes a network called GDFNet (Graph Convolution-based Dual Fusion Network), which dynamically mines the correlations between points in each layer of the network. GDFNet is divided into two parts: a density-sensitive assignment module and a spatial dynamic convolution module. The density-sensitive assignment module treats the traditional convolution kernel as a nonlinear function consisting of a weight and a density function applicable to the local coordinates of 3D points. For a given point, the weight function is learned by a multilayer perceptron network, and the density function is learned by kernel density estimation. In this way, the density features of each point are fitted, thus improving the inherent defect of the original point cloud with non-uniform density. The spatial dynamic convolution module groups the original feature channels and then uses gated processing units to enrich the feature representation of the input points. At the same time, dynamic convolution is embedded in the Euclidean space to obtain the structural information of each point and further supplement the local detail features. Extensive experiments demonstrate the effectiveness and superiority of our method on S3DIS and ScanNetv2 datasets.

Transformer based 3D semantic segmentation of urban bicycle infrastructure

ABSTRACT This work investigates Point Cloud Semantic Segmentation (PCSS) to detect bicycle paths from 3D point cloud data of bicycle-mounted LiDARs. For the task of PCSS, an existing Convolutional Neural Network architecture (CNN) is first pre-trained on the Semantic KITTI data set, an important data set for PCSS in LiDAR scans of driving situations. Furthermore, a new, semantically-labelled 3D LiDAR data set, the Salzburg Bicycle LiDAR Data Set (SBLD), is presented, which consists of 16,008 point clouds recorded by a ROS2-enabled sensor bicycle with five 3D LiDARs covering every direction. After fine-tuning the CNN on the SBLD train set, the segmentation performance is evaluated on the SBLD test set. The CNN shows promising results in recognising bike paths, vegetation, terrain and buildings in the SBLD. To further push the segmentation performance, the existing CNN is enhanced with self-attention blocks. The evaluation of this enhanced architecture shows a performance gain of 2.79% points compared to the original architecture. The proposed segmentation approach shows practical potential for detecting bike paths. The SBLD is provided to the scientific community to further drive research in the field of sensor bicycles.

HAVANA: Hard Negative Sample-Aware Self-Supervised Contrastive Learning for Airborne Laser Scanning Point Cloud Semantic Segmentation

Deep Neural Network (DNN)-based point cloud semantic segmentation has presented significant breakthrough using large-scale labeled aerial laser point cloud datasets. However, annotating such large-scaled point clouds is time-consuming. Self-Supervised Learning (SSL) is a promising approach to this problem by pre-training a DNN model utilizing unlabeled samples followed by a fine-tuned downstream task involving very limited labels. The traditional contrastive learning for point clouds selects the hardest negative samples by solely relying on the distance between the embedded features derived from the learning process, potentially evolving some negative samples from the same classes to reduce the contrastive learning effectiveness. This work proposes a hard-negative sample-aware self-supervised contrastive learning algorithm to pre-train the model for semantic segmentation. We designed a k-means clustering-based Absolute Positive And Negative samples (AbsPAN) strategy to filter the possible false-negative samples. Experiments on two typical ALS benchmark datasets demonstrate that the proposed method is more appealing than supervised training schemes without pre-training. Especially when the labels are severely inadequate (10% of the ISPRS training set), the results obtained by the proposed HAVANA method still exceed 94% of the supervised paradigm performance with full training set.

Weakly supervised point cloud semantic segmentation with the fusion of heterogeneous network features

Weakly supervised point cloud segmentation has emerged as a prominent research area to address the problem of manual annotation costs. A crucial challenge in weakly supervised point cloud segmentation is the implicit augmentation of the total amount of supervision signals. In this article, we propose a novel method that utilizes the fusion of features from different networks to enhance the supervision signals. Specifically, we utilize a deep Encoder-Decoder network to capture high-level semantic features of labeled points, while a shallow Encoder network captures multi-scale detail features of labeled data. By combining these two heterogeneous networks, we acquire richer feature representations that implicitly enhance the supervision signal.Furthermore, we introduce scene-level and instance-level contrast to enhance feature representations in both coarse-grained and fine-grained manners, thus further boosting the supervisory signal. To validate the effectiveness of our approach, we conducted experiments on the large-scale indoor scene dataset, S3DIS, and the outdoor datasets, Toronto3D and Semantic3D, achieving convincing results.

Enhancing point cloud semantic segmentation in the data‐scarce domain of industrial plants through synthetic data

AbstractDigitizing existing structures is essential for applying digital methods in architecture, engineering, and construction. However, the adoption of data‐driven techniques for transforming point cloud data into useful digital models faces challenges, particularly in the industrial domain, where ground truth datasets for training are scarce. This paper investigates a solution leveraging synthetic data to train data‐driven models effectively. In the investigated industrial domain, the complex geometry of building elements often leads to occlusions, limiting the effectiveness of conventional sampling‐based synthetic data generation methods. Our approach proposes the automatic generation of realistic and semantically enriched ground truth data using surface‐based sampling methods and laser scan simulation on industry‐standard 3D models. In the presented experiments, we use a neural network for point cloud semantic segmentation to demonstrate that compared to sampling‐based alternatives, simulation‐based synthetic data significantly improve mean class intersection over union performance on real point cloud data, achieving up to 7% absolute increase.

Label-Efficient Point Cloud Semantic Segmentation: A Holistic Active Learning Approach

Deep learning models are the state of the art for semantic segmentation of point clouds, the success of which relies on the availability of large-scale annotated datasets. However, it can be prohibitively costly to prepare such datasets. In this work, we propose a holistic active learning (AL) approach to maximize model performance given limited annotation budgets. We investigate the appropriate sample granularity for active selection under the realistic “click” measurement of annotation cost, and demonstrate that superpoint-based selection allows for most efficient usage of the limited budget, when compared with point-level, polygon-level and instance/shape-level selection. We further propose new objective for AL acquisition function and exploit local consistency constraints to boost the performance of our superpoint-based approach. We evaluate our methods on three benchmark datasets, ShapeNet and PartNet and S3DIS. The results demonstrate that AL is an effective strategy to address the high annotation costs in semantic point cloud segmentation.

Bilateral cross enhancement with self-attention compensation for semantic segmentation of point clouds

Bilateral cross enhancement with self-attention compensation for semantic segmentation of point clouds

A Large-Scale Network Construction and Lightweighting Method for Point Cloud Semantic Segmentation.

To significantly enhance the performance of point cloud semantic segmentation, this manuscript presents a novel method for constructing large-scale networks and offers an effective lightweighting technique. First, a latent point feature processing (LPFP) module is utilized to interconnect base networks such as PointNet++ and Point Transformer. This intermediate module serves both as a feature information transfer and a ground truth supervision function. Furthermore, in order to alleviate the increase in computational costs brought by constructing large-scale networks and better adapt to the demand for terminal deployment, a novel point cloud lightweighting method for semantic segmentation network (PCLN) is proposed to compress the network by transferring multidimensional feature information of large-scale networks. Specifically, at different stages of the large-scale network, the structure and attention information of the point features are selectively transferred to guide the compressed network to train in the direction of the large-scale network. This paper also solves the problem of representing global structure information of large-scale point clouds through feature sampling and aggregation. Extensive experiments on public datasets and real-world data demonstrate that the proposed method can significantly improve the performance of different base networks and outperform the state-of-the-art.

Point Cloud Semantic Segmentation by Adaptively Fusing Information With Varying Distances

Point Cloud Semantic Segmentation by Adaptively Fusing Information With Varying Distances