Semantic Segmentation Of 3D Point Clouds Research Articles

PSTNet: Transformer for aggregating neighborhood features in 3D point cloud semantic segmentation of eggplant plants

Improving the quality of plant point cloud data and achieving precise segmentation is essential for effective plant phenotyping analysis and plant breeding. To ensure data quality and accuracy of segmentation, we employed a self-developed three-dimensional scanning device based on binocular vision to acquire eggplant plant point clouds with high spatial resolution. And we improved down-sampling algorithm Octree Farthest Point Sampling (OFPS) to process the original point cloud data. Furthermore, we proposed a neural network called Plant Segmentation Transformer Network (PSTNet) to achieve semantic segmentation of high spatial resolution eggplant plant datasets. PSTNet comprises the following components: (i) a Neighborhood Feature Aggregator (NPA) for storing and aggregating local neighborhood features of input points, and (ii) a cascaded Point Self-Attention module (PSA) for capturing contextual information. Experimental results demonstrate the remarkable performance of PSTNet in semantic segmentation tasks, with IoU, Precision, Recall, F1-score, and Accuracy values of 92.20 %, 95.30 %, 95.57 %, 95.43 %, and 95.15 %, respectively. Additionally, compared to the Point Cloud Transformer (PCT), which exhibited the second-best performance, PSTNet achieved improvements of 4.37, 1.47, 2.87, 2.17, and 4.22 percentage points in the abovementioned metrics. This method achieves high-precision segmentation of plant point clouds, expands the semantic segmentation method of plants, and lays a solid foundation for plant phenotypic analysis.

Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~ 6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.

Large-scale Point Cloud Segmentation based on Multi-feature Local Enhanced Fusion

This paper introduces a framework for large-scale 3D point cloud semantic segmentation - the MLEF-Net model. The model aims to improve the segmentation accuracy of large-scale point clouds by innovatively combining Manhattan distance-based KNN neighborhood search with feature aggregation techniques. This approach uniquely handles spatial, color, and normal vector attributes, thereby improving the segmentation results. The superiority of the model is validated through comprehensive testing on the SemanticKITTI and nuScenes datasets, demonstrating its potential to enhance point cloud segmentation through advanced feature fusion strategies.

YUTO SEMANTIC: A LARGE SCALE AERIAL LIDAR DATASET FOR SEMANTIC SEGMENTATION

Abstract. Creating virtual duplicates of the real world has garnered significant attention due to its applications in areas such as autonomous driving, urban planning, and urban mapping. One of the critical tasks in the computer vision community is semantic segmentation of outdoor collected point clouds. The development and research of robust semantic segmentation algorithms heavily rely on precise and comprehensive benchmark datasets. In this paper, we present the York University Teledyne Optech 3D Semantic Segmentation Dataset (YUTO Semantic), a multi-mission large-scale aerial LiDAR dataset specifically designed for 3D point cloud semantic segmentation. The dataset comprises approximately 738 million points, covering an area of 9.46 square kilometers, which results in a high point density of 100 points per square meter. Each point in the dataset is annotated with one of nine semantic classes. Additionally, we conducted performance tests of state-of-the-art algorithms to evaluate their effectiveness in semantic segmentation tasks. The YUTO Semantic dataset serves as a valuable resource for advancing research in 3D point cloud semantic segmentation and contributes to the development of more accurate and robust algorithms for real-world applications. The dataset is available at https://github.com/Yacovitch/YUTO_Semantic.

Compositional Semantic Mix for Domain Adaptation in Point Cloud Segmentation.

Deep-learning models for 3D point cloud semantic segmentation exhibit limited generalization capabilities when trained and tested on data captured with different sensors or in varying environments due to domain shift. Domain adaptation methods can be employed to mitigate this domain shift, for instance, by simulating sensor noise, developing domain-agnostic generators, or training point cloud completion networks. Often, these methods are tailored for range view maps or necessitate multi-modal input. In contrast, domain adaptation in the image domain can be executed through sample mixing, which emphasizes input data manipulation rather than employing distinct adaptation modules. In this study, we introduce compositional semantic mixing for point cloud domain adaptation, representing the first unsupervised domain adaptation technique for point cloud segmentation based on semantic and geometric sample mixing. We present a two-branch symmetric network architecture capable of concurrently processing point clouds from a source domain (e.g.synthetic) and point clouds from a target domain (e.g.real-world). Each branch operates within one domain by integrating selected data fragments from the other domain and utilizing semantic information derived from source labels and target (pseudo) labels. Additionally, our method can leverage a limited number of human point-level annotations (semi-supervised) to further enhance performance. We assess our approach in both synthetic-to-real and real-to-real scenarios using LiDAR datasets and demonstrate that it significantly outperforms state-of-the-art methods in both unsupervised and semi-supervised settings.

Analysis of Current Advancement in 3D Point Cloud Semantic Segmentation

INTRODUCTION: The division of a 3D point cloud into various meaningful regions or objects is known as point cloud segmentation.
 OBJECTIVES: The paper discusses the challenges faced in 3D point cloud segmentation, such as the high dimensionality of point cloud data, noise, and varying point densities.
 METHODS: The paper compares several commonly used datasets in the field, including the ModelNet, ScanNet, S3DIS, and Semantic 3D datasets, ApploloCar3D, and provides an analysis of the strengths and weaknesses of each dataset. Also provides an overview of the papers that uses Traditional clustering techniques, deep learning-based methods, and hybrid approaches in point cloud semantic segmentation. The report also discusses the benefits and drawbacks of each approach.
 CONCLUSION: This study sheds light on the state of the art in semantic segmentation of 3D point clouds.

A survey on weakly supervised 3D point cloud semantic segmentation

AbstractWith the popularity and advancement of 3D point cloud data acquisition technologies and sensors, research into 3D point clouds has made considerable strides based on deep learning. The semantic segmentation of point clouds, a crucial step in comprehending 3D scenes, has drawn much attention. The accuracy and effectiveness of fully supervised semantic segmentation tasks have greatly improved with the increase in the number of accessible datasets. However, these achievements rely on time‐consuming and expensive full labelling. In solve of these existential issues, research on weakly supervised learning has recently exploded. These methods train neural networks to tackle 3D semantic segmentation tasks with fewer point labels. In addition to providing a thorough overview of the history and current state of the art in weakly supervised semantic segmentation of 3D point clouds, a detailed description of the most widely used data acquisition sensors, a list of publicly accessible benchmark datasets, and a look ahead to potential future development directions is provided.

Few-Shot 3D Point Cloud Semantic Segmentation via Stratified Class-Specific Attention Based Transformer Network

3D point cloud semantic segmentation aims to group all points into different semantic categories, which benefits important applications such as point cloud scene reconstruction and understanding. Existing supervised point cloud semantic segmentation methods usually require large-scale annotated point clouds for training and cannot handle new categories. While a few-shot learning method was proposed recently to address these two problems, it suffers from high computational complexity caused by graph construction and inability to learn fine-grained relationships among points due to the use of pooling operations. In this paper, we further address these problems by developing a new multi-layer transformer network for few-shot point cloud semantic segmentation. In the proposed network, the query point cloud features are aggregated based on the class-specific support features in different scales. Without using pooling operations, our method makes full use of all pixel-level features from the support samples. By better leveraging the support features for few-shot learning, the proposed method achieves the new state-of-the-art performance, with 15% less inference time, over existing few-shot 3D point cloud segmentation models on the S3DIS dataset and the ScanNet dataset. Our code is available at https://github.com/czzhang179/SCAT.

Fighting against COVID‐19: Innovations and applications

Fighting against <scp>COVID</scp>‐19: Innovations and applications

Three-Dimensional Point Cloud Semantic Segmentation for Cultural Heritage: A Comprehensive Review

In the cultural heritage field, point clouds, as important raw data of geomatics, are not only three-dimensional (3D) spatial presentations of 3D objects but they also have the potential to gradually advance towards an intelligent data structure with scene understanding, autonomous cognition, and a decision-making ability. The approach of point cloud semantic segmentation as a preliminary stage can help to realize this advancement. With the demand for semantic comprehensibility of point cloud data and the widespread application of machine learning and deep learning approaches in point cloud semantic segmentation, there is a need for a comprehensive literature review covering the topics from the point cloud data acquisition to semantic segmentation algorithms with application strategies in cultural heritage. This paper first reviews the current trends of acquiring point cloud data of cultural heritage from a single platform with multiple sensors and multi-platform collaborative data fusion. Then, the point cloud semantic segmentation algorithms are discussed with their advantages, disadvantages, and specific applications in the cultural heritage field. These algorithms include region growing, model fitting, unsupervised clustering, supervised machine learning, and deep learning. In addition, we summarized the public benchmark point cloud datasets related to cultural heritage. Finally, the problems and constructive development trends of 3D point cloud semantic segmentation in the cultural heritage field are presented.

Prototype Adaption and Projection for Few- and Zero-Shot 3D Point Cloud Semantic Segmentation.

In this work, we address the challenging task of few-shot and zero-shot 3D point cloud semantic segmentation. The success of few-shot semantic segmentation in 2D computer vision is mainly driven by the pre-training on large-scale datasets like imagenet. The feature extractor pre-trained on large-scale 2D datasets greatly helps the 2D few-shot learning. However, the development of 3D deep learning is hindered by the limited volume and instance modality of datasets due to the significant cost of 3D data collection and annotation. This results in less representative features and large intra-class feature variation for few-shot 3D point cloud segmentation. As a consequence, directly extending existing popular prototypical methods of 2D few-shot classification/segmentation into 3D point cloud segmentation won't work as well as in 2D domain. To address this issue, we propose a Query-Guided Prototype Adaption (QGPA) module to adapt the prototype from support point clouds feature space to query point clouds feature space. With such prototype adaption, we greatly alleviate the issue of large feature intra-class variation in point cloud and significantly improve the performance of few-shot 3D segmentation. Besides, to enhance the representation of prototypes, we introduce a Self-Reconstruction (SR) module that enables prototype to reconstruct the support mask as well as possible. Moreover, we further consider zero-shot 3D point cloud semantic segmentation where there is no support sample. To this end, we introduce category words as semantic information and propose a semantic-visual projection model to bridge the semantic and visual spaces. Our proposed method surpasses state-of-the-art algorithms by a considerable 7.90% and 14.82% under the 2-way 1-shot setting on S3DIS and ScanNet benchmarks, respectively.

3D Point Cloud Semantic Segmentation System Based on Lightweight FPConv

In this paper, we proposed a 3D point cloud semantic segmentation system based on lightweight FPConv. In 3D point cloud mapping, data is depicted in a 3D space to represent 3D imagery data. These maps are collected through direct measurements; all points in a 3D point cloud map corresponds to a measurement point and, therefore, contains a large amount of data. Data in 3D point cloud maps are stored in point clouds, and they are extracted using 3D image processing or deep learning. However, because of the non-structured and high-dimensional properties of point clouds, the development of 3-D image recognition applications in the field of computer vision warrants further exploration. Large-scale neural networks are highly accurate, but they have the disadvantages of high computation complexity and low portability. Therefore, the present study proposed a 3D point cloud semantic segmentation system based on lightweight FPConv. The proposed network combines depth-wise separate convolution, quantization, and Winograd convolution technology to lighten and accelerate neural network computation. The performance of the presented network was verified using the Stanford 3D Large-Scale Indoor Spaces (S3DIS) large scene database provided by Stanford 3D AI Lab. The results reveals the excellent performance of the proposed model.

Exploring Semantic Information Extraction From Different Data Forms in 3D Point Cloud Semantic Segmentation

As a critical step in 3D scene understanding, semantic segmentation of point clouds has broad application scenarios, including intelligent driving, augmented reality, smart factories, etc. Point cloud data is complex and irregular, and traditional machine learning methods are difficult to achieve ideal segmentation results. Deep learning techniques have yielded remarkable outcomes for researchers, leading to a surge in interest in investigating the semantic segmentation of point clouds. This article begins by examining the difficulties involved in segmenting point clouds by analyzing the inherent structural characteristics of point clouds. Then, commonly used datasets for point cloud semantic segmentation and evaluation metrics for assessing segmentation performance were introduced. Subsequently, an exploration was carried out on extracting semantic information from different data forms in point cloud semantic segmentation. Based on these findings, the experimental results of these methods on publicly available datasets are compared quantitatively. Lastly, several outlooks are presented regarding the future development of semantic segmentation techniques for 3D point clouds. The point cloud semantic segmentation techniques summarized in this paper are mainly from the state-of-the-art methods presented at top international conferences. The goal is to provide a comprehensive overview of this field’s state of the art and can be used as a reference for researchers and beginners.

Geometry-Injected Image-Based Point Cloud Semantic Segmentation

Image-based methods have replicated the success from 2D domain to 3D point cloud semantic segmentation. However, when we directly apply 2D techniques to the projected pseudo-image, inherent differences between the point cloud and the image cause geometric distortion. This paper analyzes the geometric distortion between the point cloud and the pseudo-image, including truncation, dislocation, and hole. To ensure geometric fidelity, we propose the Geometry-injected Image-based point cloud semantic segmentation Network (GINet). We design a Cyclic Convolution to optimize the convolution operation, dealing with truncation. For dislocation and hole, we propose Dual Geometric Constraints, including Local Spatial Attention and Local Affinity Regularization, to incorporate the geometric information into semantic feature learning. Local Spatial Attention generates an attention map from the point coordinates to modulate the feature map before convolution. Local Affinity Regularization supervises the semantic similarity of pixels in the convolution kernel range. GINet rectifies the geometric distortion with these mechanisms while taking advantage of the successful 2D semantic segmentation methods. Quantitative and qualitative experiments on SemanticKITTI and SemanticPOSS demonstrate the effectiveness of GINet.

A Study on the Effect of Multispectral LiDAR Data on Automated Semantic Segmentation of 3D-Point Clouds

Mobile mapping is an application field of ever-increasing relevance. Data of the surrounding environment is typically captured using combinations of LiDAR systems and cameras. The large amounts of measurement data are then processed and interpreted, which is often done automated using neural networks. For the evaluation the data of the LiDAR and the cameras needs to be fused, which requires a reliable calibration of the sensors. Segmentation solemnly on the LiDAR data drastically decreases the amount of data and makes the complex data fusion process obsolete but on the other hand often performs poorly due to the lack of information about the surface remission properties. The work at hand evaluates the effect of a novel multispectral LiDAR system on automated semantic segmentation of 3D-point clouds to overcome this downside. Besides the presentation of the multispectral LiDAR system and its implementation on a mobile mapping vehicle, the point cloud processing and the training of the CNN are described in detail. The results show a significant increase in the mIoU when using the additional information from the multispectral channel compared to just 3D and intensity information. The impact on the IoU was found to be strongly dependent on the class.

Semantic Segmentation of Substation Site Cloud Based on Seg-PointNet

3D point cloud semantic segmentation has been widely used in industrial scenes and has attracted continuous attention as a critical technology for understanding the intelligent robot scene. However, extracting visual semantics in complex environments remains a challenge. We propose the Seg-PointNet model based on multi-layer residual structure and feature pyramid for the LiDAR point cloud data semantic segmentation task in the complex substations scene. The model is based on the PointNet network and introduces a multi-scale residual structure. The residual structure multilayer perception (RES-MLP) model is proposed to fully excavate features at different levels and improve the characterization capabilities of complex features. Moreover, the 3D point cloud feature pyramid module is proposed to characterize the substation scene’s semantic features. We tested and verified the Seg-PointNet model on a self-built substation cloud point (SCP) dataset. The results show that the proposed Seg-PointNet model effectively improves the point cloud data segmentation accuracy, with an accuracy of 89.23% and mean intersection over union (mIoU) of 63.57%. This shows that the model can be applied to substation scenarios and provide technical support to intelligent robots in complex substation environments.

SVASeg: Sparse Voxel-Based Attention for 3D LiDAR Point Cloud Semantic Segmentation

3D LiDAR has become an indispensable sensor in autonomous driving vehicles. In LiDAR-based 3D point cloud semantic segmentation, most voxel-based 3D segmentors cannot efficiently capture large amounts of context information, resulting in limited receptive fields and limiting their performance. To address this problem, a sparse voxel-based attention network is introduced for 3D LiDAR point cloud semantic segmentation, termed SVASeg, which captures large amounts of context information between voxels through sparse voxel-based multi-head attention (SMHA). The traditional multi-head attention cannot directly be applied to the non-empty sparse voxels. To this end, a hash table is built according to the incrementation of voxel coordinates to lookup the non-empty neighboring voxels of each sparse voxel. Then, the sparse voxels are grouped into different groups, and each group corresponds to a local region. Afterwards, position embedding, multi-head attention and feature fusion are performed for each group to capture and aggregate the context information. Based on the SMHA module, the SVASeg can directly operate on the non-empty voxels, maintaining a comparable computational overhead to the convolutional method. Extensive experimental results on the SemanticKITTI and nuScenes datasets show the superiority of SVASeg.

LLGF-Net: Learning Local and Global Feature Fusion for 3D Point Cloud Semantic Segmentation

Three-dimensional (3D) point cloud semantic segmentation is fundamental in complex scene perception. Currently, although various efficient 3D semantic segmentation networks have been proposed, the overall effect has a certain gap to 2D image segmentation. Recently, some transformer-based methods have opened a new stage in computer vision, which also has accelerated the effective development of methods in 3D point cloud segmentation. In this paper, we propose a novel semantic segmentation network named LLGF-Net that can aggregate features from both local and global levels of point clouds, effectively improving the ability to extract feature information from point clouds. Specifically, we adopt the multi-head attention mechanism in the original Transformer model to obtain the local features of point clouds and then use the position-distance information of point clouds in 3D space to obtain the global features. Finally, the local features and global features are fused and embedded into the encoder–decoder network to generate our method. Our extensive experimental results on the 3D point cloud dataset demonstrate the effectiveness and superiority of our method.

Improving performance of deep learning models for 3D point cloud semantic segmentation via attention mechanisms

3D Semantic segmentation is a key element for a variety of applications in robotics and autonomous vehicles. For such applications, 3D data are usually acquired by LiDAR sensors resulting in a point cloud, which is a set of points characterized by its unstructured form and inherent sparsity. For the task of 3D semantic segmentation where the corresponding point clouds should be labeled with semantics, the current tendency is the use of deep learning neural network architectures for effective representation learning. On the other hand, various 2D and 3D computer vision tasks have used attention mechanisms which result in an effective re-weighting of the already learned features. In this work, we aim to investigate the role of attention mechanisms for the task of 3D semantic segmentation for autonomous driving, by identifying the significance of different attention mechanisms when adopted in existing deep learning networks. Our study is further supported by an extensive experimentation on two standard datasets for autonomous driving, namely Street3D and SemanticKITTI, that permit to draw conclusions at both a quantitative and qualitative level. Our experimental findings show that there is a clear advantage when attention mechanisms have been adopted, resulting in a superior performance. In particular, we show that the adoption of a Point Transformer in a SPVCNN network, results in an architecture which outperforms the state of the art on the Street3D dataset.

RAILWAY LIDAR SEMANTIC SEGMENTATION WITH AXIALLY SYMMETRICAL CONVOLUTIONAL LEARNING

Abstract. This paper presents a new deep-learning-based method for 3D Point Cloud Semantic Segmentation specifically designed for processing real-world LIDAR railway scenes. The new approach relies on the use of spatial local point cloud transformations for convolutional learning. These transformations allow an increased robustness to varying point cloud densities while preserving metric information and a sufficient descriptive ability. The resulting performances are illustrated with results on railway data from two distinct LIDAR point cloud datasets acquired in industrial settings. The quality of the extraction of useful information for maintenance operations and topological analysis is pointed together with a noticeable robustness to point cloud variations in distribution and point redundancy.