3D Point Cloud Segmentation Research Articles

Transformer-Based Visual Segmentation: A Survey.

Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several specific subfields, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue. In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

3D dataset generation using virtual reality for forest biodiversity

ABSTRACT While forest biodiversity faces a concerning decline, modern technology presents promising avenues for mitigation. However, a critical gap persists in reconciling ecological knowledge with the technical expertise required to use state-of-the-art technologies in 3D data classification. Currently, one main issue is the scarcity of 3D datasets for biodiversity, particularly within the context of machine learning applications. Unlike the straightforward classification of human-made structures, forest environments are uniquely intricate and nuanced due to its inherently complex nature. This study addresses this challenge by introducing a fully automated pipeline for tree stem 3D point cloud segmentation, focussing on a biodiversity indicator: tree-related microhabitats (TreMs). Furthermore, our research advances the field by demonstrating that machine learning models trained with labels generated by our proposed virtual reality (VR) method, Labelling Flora, yield predictions statistically similar to the traditional desktop-based labelling methods. This implies that existing 3D datasets could be augmented using the more rapid approach of VR labelling. Additionally, the findings of this paper demonstrate the potential integration of VR and immersive technology into the 3D labelling workflow, facilitating a quicker and more intuitive labelling process. This could empower users, who are non-familiar with 3D modelling, to contribute their expertise to the segmentation process.

Automatic measurement of rebar spacing based on 3D point cloud segmentation using Rebar-YOLOv8-seg and depth data

In power transmission and transformation expansion projects, construction personnel must measure rebar spacing at construction sites to ensure the quality of concrete structures. Due to electrified equipment, high-precision steel rulers are prohibited. We introduced a new method for automatically measuring rebar spacing to improve construction safety and work efficiency. We developed the Rebar-YOLOv8-seg model to accurately extract the rebar image mask from complex backgrounds. Subsequently, the rebar mask was aligned with depth data to create a point cloud, which performed statistical filtering and principal component analysis. Finally, we used the Random Sample Consensus method to fit the point cloud centerline, then extracted key points and calculated rebar spacing. Experiments on 4 × 4 rebar measurements using the proposed method showed that, within a specific range of camera heights and shooting angles, the measured values of rebar spacing meet the engineering measurement requirements. The method achieves an average absolute error of 1.98 mm and an average relative error of 1.76 %. Additionally, multiple adjacent rebar spacings can be measured simultaneously and completed within 5 s, providing a new feasible approach for rebar spacing measurement.

HBIM MODELLING PROCESS FROM 3D POINT CLOUDS BY APPLYING ARTIFICIAL INTELLIGENCE ALGORITHMS IN CULTURAL HERITAGE

In the context of Cultural Heritage (CH), the widespread adoption of 3D point cloud technology, coupled with Artificial Intelligence (AI) algorithms, plays a pivotal role. These technologies facilitate the creation of as-built models by integrating Building Information Modelling (BIM) strategies, enhancing collaboration within the Architecture, Engineering, and Construction (AEC) sector. Leveraging computer vision, robotics, and remote sensing, 3D points clouds provide rich data. However, manual segmentation and classification are labor-intensive and error prone. Consequently, researchers increasingly turn to machine learning (ML) and deep learning (DL) techniques for automating these tasks. The transition from manual reconstruction to automated procedures is crucial. Despite progress, gaps remain, particularly in incorporating 3D point cloud segmentation into Historical Building Information Modelling (HBIM). The lack of conclusive evidence regarding automated derivation of parametric attributes from segmentation outcomes underscores the need for further exploration. Addressing this gap is essential for cultural asset documentation, conservation, and upkeep. By automating the segmentation and classification of 3D point clouds, efficient communication via a shared database becomes feasible. The article aims to review studies on semantically parsing and classifying 3D point clouds using AI algorithms, particularly within complex cultural heritage geometries, shedding light on potential benefits and barriers.

Multi-channel depth segmentation network based on 3D graph convolution algorithm and its application in point cloud segmentation

At the current stage, 3D graph convolutional algorithms face the following issues: (1) The problem of determining the neighborhood in graph convolution. (2) The issue of fusing features at different depths in deep graph convolutional algorithms. (3) The challenge of feature fusion and recognition in multi-path deep graph convolutional algorithms. Based on these, the paper "Multi-Path Deep Segmentation Network Based on 3D Graph Convolutional Algorithm and Its Application in Point Cloud Segmentation" is proposed. Inspired by visual computing theory, the neighborhood for 3D graph convolution is determined based on the receptive field theory, and a 3D graph convolutional algorithm based on visual selectivity is proposed; A single-link deep feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, it achieve the fusion of features at different depths learned by the graph convolutional algorithm by a pyramid learning method; A multi-link feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, and achieve multi-link feature fusion and segmentation to utilize the RPN calculation method. compared with algorithms such as PointNet, PointNet++, KPConv deform, 3D GCN, and My Algorithm, The segmentation performance and geometric invariance of the proposed algorithm are validated on the ShapeNetPart dataset and the custom Mortise_and_Tenon_DB dataset, and Experiments demonstrate that the proposed algorithm is correct and feasible, offers superior 3D point cloud segmentation performance, and exhibits strong geometric invariance.

SemiGMMPoint: Semi-supervised point cloud segmentation based on Gaussian mixture models

Existing semi-supervised point cloud segmentation methods emphasize on discriminative learning, which overlooks the underlying class-conditional distributions and distribution similarities. In this paper, we propose SemiGMMPoint, the first generative framework for semi-supervised 3D point cloud segmentation in real-world and large-scale settings. Specifically, we propose a point dense generative classifier based on Gaussian mixture models (GMMs) to explicitly estimate class-conditional distributions. On top of it, we incorporate a novel similarity-minimization algorithm into the Expectation–Maximization (EM) based GMM parameter estimation, which minimizes the inter-class distribution similarity in the representation space. Moreover, we utilize the well-calibrated posterior to develop a modified point contrastive loss to mitigate sampling bias in semi-supervised settings. Extensive experiments show that SemiGMMPoint significantly boosts performance for semi-supervised point cloud segmentation on many state-of-the-art backbones without requiring architectural changes. Codes are available at https://github.com/jojodidli/SemiGMMPoint.

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

A domain adaptation framework for cross-modality SAR 3D reconstruction point clouds segmentation utilizing LiDAR data

Synthetic aperture radar (SAR) 3D point clouds reconstruction can eliminate the problems of layover in 2D SAR image projections, the recognition of the reconstructed point cloud can significantly enhance target identification and information extraction. Current SAR 3D point cloud segmentation methods, based on traditional machine learning clustering approaches, suffer from poor automation and accuracy. However, the absence of publicly labeled SAR datasets impedes the progress of deep learning-based SAR 3D point cloud segmentation methods. To tackle the aforementioned challenges, we introduce, for the first time, an alternative training approach for SAR 3D point cloud segmentation using LiDAR annotated data, which offers a more abundant sample pool, thus alleviating the lack of training sets. Nevertheless, a segmentation model trained on LiDAR point clouds exhibits a significant decline in performance when directly applied to SAR 3D reconstructed point clouds due to the cross-domain discrepancy. This research presents a pioneering domain adaptation 3D semantic segmentation framework to implement cross-modal learning for SAR point clouds. In our scheme, a simple yet effective technique is developed to achieve segmentation of SAR double-bounce scattering regions called SARDBS-Mix, which employs a mixing strategy, to capture the distinctive reflection characteristics of SAR data. Furthermore, we implement approaches including center alignment and normalization, local augmentation, and weighted cross entropy to mitigate LiDAR and SAR domain gap and class imbalances. The experimental results validate the feasibility and effectiveness of the proposed method for SAR 3D reconstructed point cloud segmentation.

Segmentation of 3D Point Clouds of Heritage Buildings Using Edge Detection and Supervoxel-Based Topology.

This paper presents a novel segmentation algorithm specially developed for applications in 3D point clouds with high variability and noise, particularly suitable for heritage building 3D data. The method can be categorized within the segmentation procedures based on edge detection. In addition, it uses a graph-based topological structure generated from the supervoxelization of the 3D point clouds, which is used to make the closure of the edge points and to define the different segments. The algorithm provides a valuable tool for generating results that can be used in subsequent classification tasks and broader computer applications dealing with 3D point clouds. One of the characteristics of this segmentation method is that it is unsupervised, which makes it particularly advantageous for heritage applications where labelled data is scarce. It is also easily adaptable to different edge point detection and supervoxelization algorithms. Finally, the results show that the 3D data can be segmented into different architectural elements, which is important for further classification or recognition. Extensive testing on real data from historic buildings demonstrated the effectiveness of the method. The results show superior performance compared to three other segmentation methods, both globally and in the segmentation of planar and curved zones of historic buildings.

Language-Level Semantics-Conditioned 3D Point Cloud Segmentation

In this work, a language-level Semantics-Conditioned framework for 3D Point cloud segmentation, called SeCondPoint, is proposed, where language-level semantics are introduced to condition the modeling of the point feature distribution, as well as the pseudo-feature generation, and a feature–geometry-based Mixup approach is further proposed to facilitate the distribution learning. Since a large number of point features could be generated from the learned distribution thanks to the semantics-conditioned modeling, any existing segmentation network could be embedded into the proposed framework to boost its performance. In addition, the proposed framework has the inherent advantage of dealing with novel classes, which seems an impossible feat for the current segmentation networks. Extensive experimental results on two public datasets demonstrate that three typical segmentation networks could achieve significant improvements over their original performances after enhancement by the proposed framework in the conventional 3D segmentation task. Two benchmarks are also introduced for a newly introduced zero-shot 3D segmentation task, and the results also validate the proposed framework.

Weakly Supervised Learning Method for Semantic Segmentation of Large-Scale 3D Point Cloud Based on Transformers

Abstract. Nowadays, semantic segmentation results of 3D point cloud have been widely applied in the fields of robotics, autonomous driving, and augmented reality etc. Thanks to the development of relevant deep learning models (such as PointNet), supervised training methods have become hotspot, in which two common limitations exists: inferior feature representation of 3D points and massive annotations. To improve 3D point feature, inspired by the idea of transformer, we employ a so-call LCP network that extracts better feature by investigating attentions between target 3D points and its corresponding local neighbors via local context propagation. Training transformer-based network needs amount of training samples, which itself is a labor-intensive, costly and error-prone work, therefore, this work proposes a weakly supervised framework, in particular, pseudo-labels are estimated based on the feature distances between unlabeled points and prototypes, which are calculated based on labeled data. The extensive experimental results show that, the proposed PL-LCP can yield considerable results (67.6% mIOU for indoor and 67.3% for outdoor) even if only using 1% real labels, and comparing to several state-of-the-art method using all labels, we achieve superior results in mIOU, OA for indoor (65.9%, 89.2%).

Hierarchical SVM for Semantic Segmentation of 3D Point Clouds for Infrastructure Scenes

The incorporation of building information modeling (BIM) has brought about significant advancements in civil engineering, enhancing efficiency and sustainability across project life cycles. The utilization of advanced 3D point cloud technologies such as laser scanning extends the application of BIM, particularly in operations and maintenance, prompting the exploration of automated solutions for labor-intensive point cloud modeling. This paper presents a demonstration of supervised machine learning—specifically, a support vector machine—for the analysis and segmentation of 3D point clouds, which is a pivotal step in 3D modeling. The point cloud semantic segmentation workflow is extensively reviewed to encompass critical elements such as neighborhood selection, feature extraction, and feature selection, leading to the development of an optimized methodology for this process. Diverse strategies are implemented at each phase to enhance the overall workflow and ensure resilient results. The methodology is then evaluated using diverse datasets from infrastructure scenes of bridges and compared with state-of-the-art deep learning models. The findings highlight the effectiveness of supervised machine learning techniques at accurately segmenting 3D point clouds, outperforming deep learning models such as PointNet and PointNet++ with smaller training datasets. Through the implementation of advanced segmentation techniques, there is a partial reduction in the time required for 3D modeling of point clouds, thereby further enhancing the efficiency and effectiveness of the BIM process.

A point cloud semantic segmentation method with augmented local geometry

With the development of 3D sensors, complex spatial structures can be established from point cloud data. Accurate semantic segmentation of 3D point clouds is beneficial for tasks such as robotic scene understanding. To extract the geometric features of the point cloud, most prior works use the xyz coordinates to potentially learn the shapes of 3D objects, which is not insufficient to handle complex scenes. In this paper, a Local Geometry Encoding (LGE) module is proposed to provide local normal information and express 3D shapes explicitly. Furthermore, considering that most prior methods use a single decoder to recover from abstract features, a Multi-Decoder (MD) ensemble module is proposed to restore multiple segmentation results from various scale feature maps in the encoder, and then a multi-attention fusion mechanism is utilized to produce single segmentation results. Benefiting from the local geometry encoding module and Multi-Decoder ensemble module, our network performs 6-fold cross-validation on the S3DIS dataset and achieves 75% mIoU with fewer parameters.

A lightweight Transformer‐based neural network for large‐scale masonry arch bridge point cloud segmentation

AbstractTransformer architecture based on the attention mechanism achieves impressive results in natural language processing (NLP) tasks. This paper transfers the successful experience to a 3D point cloud segmentation task. Inspired by newly proposed 3D Transformer neural networks, this paper introduces a new Transformer‐based module, which is called Local Geo‐Transformer. To alleviate the heavy memory consumption of the original Transformer, Local Geo‐Transformer only performs the attention mechanism in local regions. It is designed to mitigate the information loss caused by the subsampling of point clouds for segmentation. Global Geo‐Transformer is proposed to exploit the global relationships between features with the lowest resolution. The new architecture is validated on a masonry bridge dataset developed by the authors for their earlier work on a previous segmentation network called BridgeNet. The new version of the network with Transformer architecture, BridgeNetv2, outperforms BridgeNet in all metrics. BridgeNetv2 is also shown to be lightweight and memory efficient, well‐adapted to large‐scale point cloud segmentation tasks in civil engineering.

Panoptic segmentation of 3D point clouds with Gaussian mixture model in outdoor scenes

Accurate panoptic segmentation of 3D point clouds in outdoor scenes is critical for the success of applications such as autonomous driving and robot navigation. Existing methods in this area typically assume that the differences between instances are greater than the differences between points belonging to the same instance and use heuristic techniques for segmentation. However, this assumption may not hold in real scenes with occlusion and noise. In addition, most of the previous methods formulate point-wise embedding learning and instance clustering as two decoupled steps for separate optimization, making it a challenging task to learn discriminative embeddings. To address these issues, we introduce a framework for modeling points belonging to the same instance using learnable Gaussian distributions and formulate the point cloud as a Gaussian mixture model. Based on this formulation, we introduce a unified loss function that links the embedding learning and instance clustering in an end-to-end manner. Our framework is generic and can be seamlessly incorporated with existing panoptic segmentation networks. By explicitly modeling intra-instance variance and leveraging end-to-end optimization, our framework improves the discrimination capability of point embeddings with higher accuracy and robustness. Extensive experiments on two large-scale benchmarks demonstrate the effectiveness of the proposed method.

Building a Fully-Automatized Active Learning Framework for the Semantic Segmentation of Geospatial 3D Point Clouds

In recent years, significant progress has been made in developing supervised Machine Learning (ML) systems like Convolutional Neural Networks. However, it’s crucial to recognize that the performance of these systems heavily relies on the quality of labeled training data. To address this, we propose a shift in focus towards developing sustainable methods of acquiring such data instead of solely building new classifiers in the ever-evolving ML field. Specifically, in the geospatial domain, the process of generating training data for ML systems has been largely neglected in research. Traditionally, experts have been burdened with the laborious task of labeling, which is not only time-consuming but also inefficient. In our system for the semantic interpretation of Airborne Laser Scanning point clouds, we break with this convention and completely remove labeling obligations from domain experts who have completed special training in geosciences and instead adopt a hybrid intelligence approach. This involves active and iterative collaboration between the ML model and humans through Active Learning, which identifies the most critical samples justifying manual inspection. Only these samples (typically ≪1%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ll 1{\\%}$$\\end{document} of Passive Learning training points) are subject to human annotation. To carry out this annotation, we choose to outsource the task to a large group of non-specialists, referred to as the crowd, which comes with the inherent challenge of guiding those inexperienced annotators (i.e., “short-term employees”) to still produce labels of sufficient quality. However, we acknowledge that attracting enough volunteers for crowdsourcing campaigns can be challenging due to the tedious nature of labeling tasks. To address this, we propose employing paid crowdsourcing and providing monetary incentives to crowdworkers. This approach ensures access to a vast pool of prospective workers through respective platforms, ensuring timely completion of jobs. Effectively, crowdworkers become human processing units in our hybrid intelligence system mirroring the functionality of electronic processing units.

Deep Ordinal Classification in Forest Areas Using Light Detection and Ranging Point Clouds.

Recent advances in Deep Learning and aerial Light Detection And Ranging (LiDAR) have offered the possibility of refining the classification and segmentation of 3D point clouds to contribute to the monitoring of complex environments. In this context, the present study focuses on developing an ordinal classification model in forest areas where LiDAR point clouds can be classified into four distinct ordinal classes: ground, low vegetation, medium vegetation, and high vegetation. To do so, an effective soft labeling technique based on a novel proposed generalized exponential function (CE-GE) is applied to the PointNet network architecture. Statistical analyses based on Kolmogorov-Smirnov and Student's t-test reveal that the CE-GE method achieves the best results for all the evaluation metrics compared to other methodologies. Regarding the confusion matrices of the best alternative conceived and the standard categorical cross-entropy method, the smoothed ordinal classification obtains a more consistent classification compared to the nominal approach. Thus, the proposed methodology significantly improves the point-by-point classification of PointNet, reducing the errors in distinguishing between the middle classes (low vegetation and medium vegetation).

A Rapid Segmentation Method of Highway Surface Point Cloud Data Based on a Supervoxel and Improved Region Growing Algorithm

Mobile laser scanning (MLS) systems have become an important technology for collecting and measuring road information for highway maintenance and reconstruction services. However, the efficient and accurate extraction of unstructured road surfaces from MLS point cloud data collected on highways is challenging. Specifically, the complex and unstructured characteristics of road surveying point cloud data lead to traditional 3D point cloud segmentation. When traditional 3D point cloud algorithms extract unstructured road surfaces, over-segmentation and under-segmentation often occur, which affects efficiency and accuracy. To solve these problems, this study introduces an enhanced road extraction method that integrates supervoxel and trajectory information into a traditional region growing algorithm. The method involves two main steps: first, a supervoxel data structure is applied to reconstruct the original MLS point cloud data, which diminishes the calculation time of the point cloud feature vector and accelerates the merging speed of a similar region; second, the trajectory information of the vehicle is used to optimize the seed selection strategy of the regio growing algorithm, which improves the accuracy of road surface extraction. Finally, two typical highway section tests (flat road and slope road) were conducted to validate the positioning performance of the proposed algorithm in an MLS point cloud. The results show that, compared with three kinds of traditional road surface segmentation algorithms, our method achieves an average extraction recall and precision of 99.1% and 96.0%, and by calculating the recall and precision, an F1 score of 97.5% can be obtained to evaluate the performance of the proposed method, for both datasets. Additionally, our method exhibits an average road surface extraction time that is 45.0%, 50.3%, and 55.8% faster than those of the other three automated segmentation algorithms.

Semantic Segmentation of 3D Point Clouds in Outdoor Environments Based on Local Dual-Enhancement

Semantic segmentation of 3D point clouds in drivable areas is very important for unmanned vehicles. Due to the imbalance between the size of various outdoor scene objects and the sample size, the object boundaries are not clear, and small sample features cannot be extracted. As a result, the semantic segmentation accuracy of 3D point clouds in outdoor environment is not high. To solve these problems, we propose a local dual-enhancement network (LDE-Net) for semantic segmentation of 3D point clouds in outdoor environments for unmanned vehicles. The network is composed of local-global feature extraction modules, and a local feature aggregation classifier. The local-global feature extraction module captures both local and global features, which can improve the accuracy and robustness of semantic segmentation. The local feature aggregation classifier considers the feature information of neighboring points to ensure clarity of object boundaries and the high overall accuracy of semantic segmentation. Experimental results show that provides clearer boundaries between various objects, and has higher identification accuracy for small sample objects. The LDE-Net has good performance for semantic segmentation of 3D point clouds in outdoor environments.