Point Cloud Semantic Segmentation Research Articles

Real-Time Free Space Semantic Segmentation for Detection of Traversable Space for an Intelligent Wheelchair.

The last decades saw a great innovation in computer vision. Recently, the field has been fundamental in the development of autonomous navigation systems. Modern assistive technologies, like smart wheelchairs, could employ autonomous navigation to assist users during operation. A prerequisite for such systems is to recognise the navigable space in real-time. The current research features an off-the-shelf powered wheelchair customised into an intelligent robot, which perceives the environment using Point Cloud Semantic Segmentation (PCSS). The implemented algorithm is used to distinguish between two conditions, traversable and non-traversable space, in real-time, using the aforementioned conditions as the two labelled classes. The accuracy of traversable space detection resulted as 99.64% while the accuracy of non-traversable space detection was 91.79%. The performance of the suggested method was invariant to changes in wheelchair velocity indicating that the latency of the suggested algorithm is within the tolerable limits for real-time operation.

Tackling background ambiguities in multi-class few-shot point cloud semantic segmentation

Few-shot point cloud semantic segmentation learns to segment novel classes with scarce labeled samples. Within an episode, a novel target class is defined by a few support samples with corresponding binary masks, where only the points of this class are labeled as foreground and others are regarded as background. In the tasks involving multiple target classes, since the meanings of background are diverse for different target classes, background ambiguities appear: Some points labeled as background in one support sample may be of other target classes. It will result in incorrect guidance and damage model’s segmentation performance. However, previous methods in the literature do not consider this problem. In this paper, we propose a simple yet effective approach to tackle background ambiguities, which adopts the entropy of predictions on query samples to the training objective function as an additional regularization. Besides, we design a feature transformation operation to reduce the feature differences between support and query samples. With our proposed approach, fine-tuning, a weak baseline method for few-shot segmentation, gains significant performance improvement (e.g., 7.48% and 7.04% in 2-way-1-shot and 3-way-1-shot tasks of S3DIS, respectively) and outperforms current state-of-the-art methods in all the task settings of S3DIS and ScanNet benchmark datasets.

LLGF-Net: Learning Local and Global Feature Fusion for 3D Point Cloud Semantic Segmentation

Three-dimensional (3D) point cloud semantic segmentation is fundamental in complex scene perception. Currently, although various efficient 3D semantic segmentation networks have been proposed, the overall effect has a certain gap to 2D image segmentation. Recently, some transformer-based methods have opened a new stage in computer vision, which also has accelerated the effective development of methods in 3D point cloud segmentation. In this paper, we propose a novel semantic segmentation network named LLGF-Net that can aggregate features from both local and global levels of point clouds, effectively improving the ability to extract feature information from point clouds. Specifically, we adopt the multi-head attention mechanism in the original Transformer model to obtain the local features of point clouds and then use the position-distance information of point clouds in 3D space to obtain the global features. Finally, the local features and global features are fused and embedded into the encoder–decoder network to generate our method. Our extensive experimental results on the 3D point cloud dataset demonstrate the effectiveness and superiority of our method.

Point cloud semantic segmentation based on pillarpointnet

Aiming at the problem that the deep neural network pointnet does not introduce local features, the segmentation accuracy is not high and the segmentation efficiency of pointnet++ is low. On the basis of pointnet, a two-way feature extraction method, pillarpointnet, is proposed. Our network is divided into upper and lower parallel paths. The upper path is a simplified version of pointnet, which is used to extract global features. In the lower path, we use pillars instead of voxels to reduce the problem of wasting computing power due to inconsistency in density. The features of each voxel are then assigned to the points within the grid to represent the domain features of the points. Finally, the local features, global features and the points are concatenated and put into the segmentation network. The final experimental results show that compared to some current state-of-the-art segmentation networks, pillarpointnet maintains a good balance between speed and accuracy.

Improving performance of deep learning models for 3D point cloud semantic segmentation via attention mechanisms

3D Semantic segmentation is a key element for a variety of applications in robotics and autonomous vehicles. For such applications, 3D data are usually acquired by LiDAR sensors resulting in a point cloud, which is a set of points characterized by its unstructured form and inherent sparsity. For the task of 3D semantic segmentation where the corresponding point clouds should be labeled with semantics, the current tendency is the use of deep learning neural network architectures for effective representation learning. On the other hand, various 2D and 3D computer vision tasks have used attention mechanisms which result in an effective re-weighting of the already learned features. In this work, we aim to investigate the role of attention mechanisms for the task of 3D semantic segmentation for autonomous driving, by identifying the significance of different attention mechanisms when adopted in existing deep learning networks. Our study is further supported by an extensive experimentation on two standard datasets for autonomous driving, namely Street3D and SemanticKITTI, that permit to draw conclusions at both a quantitative and qualitative level. Our experimental findings show that there is a clear advantage when attention mechanisms have been adopted, resulting in a superior performance. In particular, we show that the adoption of a Point Transformer in a SPVCNN network, results in an architecture which outperforms the state of the art on the Street3D dataset.

Point cloud semantic segmentation of complex railway environments using deep learning

Safety of transportation networks is of utmost importance for our society. With the emergency of digitalization, the railway sector is accelerating the automation in inventory and inspection procedures. Mobile mapping systems allow capturing three-dimensional point clouds of the infrastructure in short periods of time. In this paper, a deep learning methodology for semantic segmentation of railway infrastructures is presented. The methodology segments both linear and punctual elements from railway infrastructure, and it is tested in four scenarios: i) 90 km-long railway; ii) 2 km-long low-quality point clouds; iii) 400 m-long high-quality point clouds; iv) 1.4 km-long railway recoded with aerial mapping system. The longest one is used for training and testing, obtaining mean accuracy greater than 90%. The other scenarios are used only for testing, and qualitative results are discussed, proving that the method can be applied to new scenarios that significantly differ in terms of data quality and resolution.

Understanding the Imperfection of 3D point Cloud and Semantic Segmentation algorithms for 3D Models of Indoor Environment

Abstract. Point clouds data provides new potentials for automated construction of more geometrically accurate and semantically rich 3D models for indoor environments. Recent advances in deep learning methods on point cloud semantic segmentation demonstrated impressive accuracy in labeling points of 3D surfaces with object classes. However, it remains challenging to reconstruct the shape of semantic objects from semantically-labeled 3D points, due to imperfection of such data and the under-determination of object construction algorithms. We have little empirical knowledge about how data imperfections affect the reconstruction of 3D indoor room objects. This paper contributes to understanding the nature of such imperfection of 3D point cloud data and semantic segmentation algorithms by analyzing the reconstructability of indoor room objects from semantically-labeled point cloud. 181 rooms from Stanford Large-Scale 3D Indoor Spaces Dataset (S3DIS) were used in our experiment. After generating semantic labels on point-clouds using PointNet++ segmentic segmentation algorithm, we use human coders to judge the reconstructability of indoor objects, following a qualitative coding scheme. Human exploration of object shape imperfection was assisted by a visual analytic tool in making their judgement. We found that high point-level accuracy achieved through semantic segmentation of point cloud data does not guarantee high object-level accuracy. The extent of this problem varies widely among different spatial settings and configurations. We discuss the significance of these findings on the choice of 3D reconstruction methods.

MULTI-SOURCE POINT CLOUD SEMANTIC SEGMENTATION USING NEURAL NETWORK

Abstract. The purpose of this study is to enhance point cloud semantic segmentation by using point clouds from multiple distinct technologies on the same capture location and to determine whether employing various technologies throughout the acquisition process yields better performance during classification. The different point clouds were captured in the same geographical location and have previously been aligned and classified by professionals of the field. Three locations have been scanned with airborne lidar, terrestrial lidar and photogrammetry using UAV or helicopter. The use of various sources of capture on the same location opens the door to creating new features, such as the proportion of each source involved in the semantic segmentation of point clouds. This plurality of sources also enables us to spread various features, such as RGB colors, that have been propagated to other sources via the neighborhood. The initial results lean towards capture using different technologies as the overall accuracy increase by two to four points and the mean Matthews correlation coefficient increase by four to seven points. The main drawbacks are the cost of some technologies, as well as the processing time, which is greater than with a single technology.

2D TO 3D LABEL PROPAGATION FOR THE SEMANTIC SEGMENTATION OF HERITAGE BUILDING POINT CLOUDS

Abstract. During the last decade, the use of semantic models of 3D buildings and structures kept growing, fostered in particular by the spread of Building Information Models (BIMs), becoming quite popular in several civil engineering and geomatics applications. Nevertheless, semantic model production usually requires quite a lot of human interaction, which may result in quite long and annoying procedures for human operators. The production of 3D semantic models of buildings often takes advantage of already available 3D reconstructions of the considered objects. Given the ever increasing resolution of 3D reconstructions, obtained thanks to the recently developed laser scanners and photogrammetric software, the availability of tools for supporting the automatic or semi-automatic generation of semantic models represents a key step for easing and speeding up the process of semantic model production. In particular, the correct semantic interpretation of the different parts of a 3D point cloud, can be seen as the basic step for the production of a BIM model. The most frequently used methods for point cloud semantic segmentation can be separated in two categories: those directly segmenting the point clouds and those based on the ancillary semantic segmentation of images representing the object of interest, then transferring back the segmentation results to the point cloud. This work focuses on the latter method, considering more specifically the application of heritage building semantic segmentation. To be more specific, this paper investigates the semantic segmentation performance on a set of four heritage buildings, obtained first applying deep-learning based image semantic segmentation and then propagating back the semantic information to the point cloud by means of a voting strategy. The obtained results are quite encouraging, motivating future investigations on improvements of this strategy, in particular when including more buildings in the considered dataset.

SEMANTIC SEGMENTATION OF URBAN TEXTURED MESHES THROUGH POINT SAMPLING

Abstract. Textured meshes are becoming an increasingly popular representation combining the 3D geometry and radiometry of real scenes. However, semantic segmentation algorithms for urban mesh have been little investigated and do not exploit all radiometric information. To address this problem, we adopt an approach consisting in sampling a point cloud from the textured mesh, then using a point cloud semantic segmentation algorithm on this cloud, and finally using the obtained semantic to segment the initial mesh. In this paper, we study the influence of different parameters such as the sampling method, the density of the extracted cloud, the features selected (color, normal, elevation) as well as the number of points used at each training period. Our result outperforms the state-of-the-art on the SUM dataset, earning about 4 points in OA and 18 points in mIoU.

RAILWAY LIDAR SEMANTIC SEGMENTATION WITH AXIALLY SYMMETRICAL CONVOLUTIONAL LEARNING

Abstract. This paper presents a new deep-learning-based method for 3D Point Cloud Semantic Segmentation specifically designed for processing real-world LIDAR railway scenes. The new approach relies on the use of spatial local point cloud transformations for convolutional learning. These transformations allow an increased robustness to varying point cloud densities while preserving metric information and a sufficient descriptive ability. The resulting performances are illustrated with results on railway data from two distinct LIDAR point cloud datasets acquired in industrial settings. The quality of the extraction of useful information for maintenance operations and topological analysis is pointed together with a noticeable robustness to point cloud variations in distribution and point redundancy.

EXPLORING LABEL INITIALIZATION FOR WEAKLY SUPERVISED ALS POINT CLOUD SEMANTIC SEGMENTATION

Abstract. Although a number of emerging point-cloud semantic segmentation methods achieve state-of-the-art results, acquiring fully interpreted training data is a time-consuming and labor-intensive task. To reduce the burden of data annotation in training, semiand weakly supervised methods are proposed to address the situation of limited supervisory sources, achieving competitive results compared to full supervision schemes. However, given a fixed budget, the effective annotation of a few points is typically ignored, which is referred to as weak-label initialization in this study. In practice, random selection is typically adopted by default. Because weakly supervised methods largely rely on semantic information supplied by initial weak labels, this studies explores the influence of different weak-label initialization strategies. In addition to random initialization, we propose a feature-constrained framework to guide the selection of initial weak labels. A feature space of point clouds is first constructed by feature extraction and embedding. Then, we develop a density-biased strategy to annotate points by locating highly dense clustered regions, as significant information distinguishing semantic classes is often concentrated in such areas. Our method outperforms random initialization on ISPRS Vaihingen 3D data when only using sparse weak labels, achieving an overall accuracy of 78.06% using 1‰ of labels. However, only a minor increase is observed on the LASDU dataset. Additionally, the results show that initialization with category-wise uniformly distributed weak labels is more effective when incorporated using a weakly supervised method.

Color guided convolutional network for point cloud semantic segmentation

Point cloud semantic segmentation based on deep learning methods is still a challenge due to the irregularity of structures and uncertainty of sampling. Color information often contains a lot of prior information, whereas the existing methods do not attach more importance to it. To deal with this problem, we propose a novel hard attention mechanism, named color-guided convolution. This convolution operator learns the correlation between geometric and color information by reordering the local points with color-indicated vectors. In addition, the global feature fusion is proposed to rectify features selected by the feature selecting unit. Experimental results and comparisons with recent methods demonstrate the superiority of our approach.

A new weakly supervised approach for ALS point cloud semantic segmentation

Although novel point cloud semantic segmentation schemes that continuously surpass state-of-the-art results exist, the success of learning an effective model typically relies on the availability of abundant labeled data. However, data annotation is a time-consumng and labor-intensive task, particularly for large-scale airborne laser scanning (ALS) point clouds involving multiple classes in urban areas. Therefore, simultaneously obtaining promising results while significantly reducing labeling is crucial. In this study, we propose a deep-learning-based weakly supervised framework for the semantic segmentation of ALS point clouds. This is to exploit implicit information from unlabeled data subject to incomplete and sparse labels. Entropy regularization is introduced to penalize class overlap in the predictive probability. Additionally, a consistency constraint is designed to improve the robustness of the predictions by minimizing the difference between the current and ensemble predictions. Finally, we propose an online soft pseudo-labeling strategy to create additional supervisory sources in an efficient and nonparametric manner. Extensive experimental analysis using three benchmark datasets demonstrates that our proposed method significantly boosts the classification performance without compromising the computational efficiency, considering the sparse point annotations. It outperforms the current weakly supervised methods and achieves a result comparable to that of full supervision competitors. Considering the ISPRS Vaihingen 3D data, using only 1‰ labels, our method achieved an overall accuracy of 83.0% and an average F1 score of 70.0%. These increased by 6.9% and 12.8%, respectively, compared to the model trained only using sparse label information.

Local Transformer Network on 3D Point Cloud Semantic Segmentation

Semantic segmentation is an important component in understanding the 3D point cloud scene. Whether we can effectively obtain local and global contextual information from points is of great significance in improving the performance of 3D point cloud semantic segmentation. In this paper, we propose a self-attention feature extraction module: the local transformer structure. By stacking the encoder layer composed of this structure, we can extract local features while preserving global connectivity. The structure can automatically learn each point feature from its neighborhoods and is invariant to different point orders. We designed two unique key matrices, each of which focuses on the feature similarities and geometric structure relationships between the points to generate attention weight matrices. Additionally, the cross-skip selection of neighbors is used to obtain larger receptive fields for each point without increasing the number of calculations required, and can therefore better deal with the junction between multiple objects. When the new network was verified on the S3DIS, the mean intersection over union was 69.1%, and the segmentation accuracies on the complex outdoor scene datasets Semantic3D and SemanticKITTI were 94.3% and 87.8%, respectively, which demonstrate the effectiveness of the proposed methods.

A Low-Power Graph Convolutional Network Processor With Sparse Grouping for 3D Point Cloud Semantic Segmentation in Mobile Devices

A low-power graph convolutional network (GCN) processor is proposed for accelerating 3D point cloud semantic segmentation (PCSS) in real-time on mobile devices. Three key features enable the low-power GCN-based 3D PCSS. First, the new hardware-friendly GCN algorithm, sparse grouping-based dilated graph convolution (SG-DGC) is proposed. SG-DGC reduces 71.7% of the overall computation and 76.9% of EMA through the sparse grouping of the point cloud. Second, the two-level pipeline (TLP) consisting of the point-level pipeline (PLP) and group-level pipelining (GLP) was proposed to improve low utilization by the imbalanced workload of GCN. The PLP enables point-level module-wise fusion (PMF) which reduces 47.4% of EMA for low power consumption. Also, center point feature reuse (CPFR) reuses computation results of the redundant operation and reduces 11.4% of computation. Finally, the GLP increased the core utilization by 21.1% by balancing the workload of graph generation and graph convolution and enable <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.1\times $ </tex-math></inline-formula> higher throughput. The processor is implemented with 65nm CMOS technology, and the 4.0mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> 3D PCSS processor show 95mW power consumption while operating in real-time of 30.8 fps in the 3D PCSS of the indoor scene with 4k points.

The Point Cloud Semantic Segmentation Method for the Ming and Qing Dynasties’ Official-Style Architecture Roof Considering the Construction Regulations

Point cloud semantic segmentation has played an important role in the scan-to-BIM process of the Ming and Qing Dynasties’ official-style architecture roof. To overcome the complexity of roof components’ shape and the scale differences between different roof component types, a point cloud semantic segmentation method for the MQDOAs roof considering the construction regulations is proposed in this paper. This method is composed of two stages. In the first stage, the features from the construction rules of MQDOAs, including the normalized symmetrical distance (NSD), relative height (RH) and local height difference (LHD), are extracted alongside the regular geometric features. To lower the influence of scale differences, a multi-scale feature connection strategy is also applied to construct the feature classification vector. In the second stage, RF method is applied to classify the point cloud. To verify the efficiency of the proposed method, we took the Hall of Complete Harmony as the study case. The experiments showed that our method achieved segmentation result in overall classification accuracy and reached 96.8%.

A training dataset for semantic segmentation of urban point cloud map for intelligent vehicles

This paper presents a training dataset of point-wisely labeled global point cloud map (GPCM) in urban area. The labeled GPCM dataset can be used to train machine learning models for point cloud semantic segmentation (PCSS). The trained PCSS machine learning model can classify the semantic class of urban GPCM point-wisely to build a semantic GPCM (SGPCM). The SGPCM can be used to enhance the functionality of many intelligent and autonomous vehicle applications, such as localization, map-based perception, and high-definition (HD) map processing. To maximize the utility of the training dataset for the applications, the semantic classes are designed taking into account the requirements of intelligent and autonomous vehicles in urban area. Therefore, the classes contain urban-related objects, such as road surface markers, curba, and in-tunnel objects, which were not included in other previous datasets. First step of construction process for the PCSS training dataset is a generation of the GPCM in the urban area by using a mobile mapping system (MMS). The points in the generated GPCM with MMS contain information about the position (X, Y, Z) and the surface (R, G, B, I). After building the unlabeled GPCM, each point is labeled with the designed semantic classes thorough semi-automatic and manual labeling. The labeled SGPCM was used to train various state-of-the-art PCSS models to verify the effectiveness of SGPCM train-set for the intelligent and autonomous vehicles.

IAGC: Interactive Attention Graph Convolution Network for Semantic Segmentation of Point Clouds in Building Indoor Environment

Point-based networks have been widely used in the semantic segmentation of point clouds owing to the powerful 3D convolution neural network (CNN) baseline. Most of the current methods resort to intermediate regular representations for reorganizing the structure of point clouds for 3D CNN networks, but they may neglect the inherent contextual information. In our work, we focus on capturing discriminative features with the interactive attention mechanism and propose a novel method consisting of the regional simplified dual attention network and global graph convolution network. Firstly, we cluster homogeneous points into superpoints and construct a superpoint graph to effectively reduce the computation complexity and greatly maintain spatial topological relations among superpoints. Secondly, we integrate cross-position attention and cross-channel attention into a single head attention module and design a novel interactive attention gating (IAG)-based multilayer perceptron (MLP) network (IAG–MLP), which is utilized for the expansion of the receptive field and augmentation of discriminative features in local embeddings. Afterwards, the combination of stacked IAG–MLP blocks and the global graph convolution network, called IAGC, is proposed to learn high-dimensional local features in superpoints and progressively update these local embeddings with the recurrent neural network (RNN) network. Our proposed framework is evaluated on three indoor open benchmarks, and the 6-fold cross-validation results of the S3DIS dataset show that the local IAG–MLP network brings about 1% and 6.1% improvement in overall accuracy (OA) and mean class intersection-over-union (mIoU), respectively, compared with the PointNet local network. Furthermore, our IAGC network outperforms other CNN-based approaches in the ScanNet V2 dataset by at least 7.9% in mIoU. The experimental results indicate that the proposed method can better capture contextual information and achieve competitive overall performance in the semantic segmentation task.

DeepGCNs-Att: Point cloud semantic segmentation with contextual point representations

Graph Convolutional Networks are able to characterize non-Euclidean spaces effectively compared with traditional Convolutional Neural Networks, which can extract the local features of the point cloud using deep neural networks, but it cannot make full use of the global features of the point cloud for semantic segmentation. To solve this problem, this paper proposes a novel network structure called DeepGCNs-Att that enables deep Graph Convolutional Network to aggregate global context features efficiently. Moreover, to speed up the computation, we add an Attention layer after the Graph Convolutional Network Backbone Block to mutually enhance the connection between the distant points of the non-Euclidean space. Our model is tested on the standard benchmark S3DIS. By comparing with other deep Graph Convolutional Networks, our DeepGCNs-Att’s mIoU has at least two percent higher than that of all other models and even shows excellent results in space complexity and computational complexity under the same number of Graph Convolutional Network layers.