3D Point Cloud Segmentation Research Articles

An Inverse Node Graph-Based Method for the Urban Scene Segmentation of 3D Point Clouds

Urban object segmentation and classification tasks are critical data processing steps in scene understanding, intelligent vehicles and 3D high-precision maps. Semantic segmentation of 3D point clouds is the foundational step in object recognition. To identify the intersecting objects and improve the accuracy of classification, this paper proposes a segment-based classification method for 3D point clouds. This method firstly divides points into multi-scale supervoxels and groups them by proposed inverse node graph (IN-Graph) construction, which does not need to define prior information about the node, it divides supervoxels by judging the connection state of edges between them. This method reaches minimum global energy by graph cutting, obtains the structural segments as completely as possible, and retains boundaries at the same time. Then, the random forest classifier is utilized for supervised classification. To deal with the mislabeling of scattered fragments, higher-order CRF with small-label cluster optimization is proposed to refine the classification results. Experiments were carried out on mobile laser scan (MLS) point dataset and terrestrial laser scan (TLS) points dataset, and the results show that overall accuracies of 97.57% and 96.39% were obtained in the two datasets. The boundaries of objects were retained well, and the method achieved a good result in the classification of cars and motorcycles. More experimental analyses have verified the advantages of the proposed method and proved the practicability and versatility of the method.

The Hessigheim 3D (H3D) benchmark on semantic segmentation of high-resolution 3D point clouds and textured meshes from UAV LiDAR and Multi-View-Stereo

Automated semantic segmentation and object detection are of great importance in geospatial data analysis. However, supervised machine learning systems such as convolutional neural networks require large corpora of annotated training data. Especially in the geospatial domain, such datasets are quite scarce. Within this paper, we aim to alleviate this issue by introducing a new annotated 3D dataset that is unique in three ways: i) The dataset consists of both an Unmanned Aerial Vehicle (UAV) laser scanning point cloud and a 3D textured mesh. ii) The point cloud features a mean point density of about 800 ​pts/m2 and the oblique imagery used for 3D mesh texturing realizes a ground sampling distance of about 2–3 ​cm. This enables the identification of fine-grained structures and represents the state of the art in UAV-based mapping. iii) Both data modalities will be published for a total of three epochs allowing applications such as change detection. The dataset depicts the village of Hessigheim (Germany), henceforth referred to as H3D - either represented as 3D point cloud H3D(PC) or 3D mesh H3D(Mesh). It is designed to promote research in the field of 3D data analysis on one hand and to evaluate and rank existing and emerging approaches for semantic segmentation of both data modalities on the other hand. Ultimately, we hope that H3D will become a widely used benchmark dataset in company with the well-established ISPRS Vaihingen 3D Semantic Labeling Challenge benchmark (V3D). The dataset can be downloaded from https://ifpwww.ifp.uni-stuttgart.de/benchmark/hessigheim/default.aspx.

3D URBAN CHANGE DETECTION WITH POINT CLOUD SIAMESE NETWORKS

Abstract. As the majority of the earth population is living in urban environments, cities are continuously evolving and efficient monitoring tools are needed to retrieve and classify their evolution. In this context, analysing changes between two dates is a crucial point. In urban environments, most changes occur along the vertical axis (with new construction or demolition of buildings) and the use of 3D data is therefore mandatory. Among them, LiDAR constitutes a valuable source of information. However, With the difficulty of processing sparse and unordered 3D point clouds, most of existing methods start by rasterizing point clouds (for example to Digital Surface Models) before using more conventional image processing tools. This implies a significant loss of information. Among existing studies dealing directly with point clouds, and to the best of our knowledge, no deep neural network-based method has been explored yet. Thus, in order to fill this gap and to test the ability of deep methods to deal with change detection and characterization of 3D point clouds, we propose a Siamese network with Kernel Point Convolution inspired by Siamese architectures that have already shown their performances on change detection in 2D images and on KPConv network which achieves high-quality results for semantic segmentation of raw 3D point clouds. We show quantitatively and qualitatively that our method outperforms by more than 25% (in terms of average Intersection over Union for classes of change) existing machine learning methods based on hand-crafted features.

Revisiting 2D Convolutional Neural Networks for Graph-Based Applications.

Graph convolutional networks (GCNs) are widely used in graph-based applications such as graph classification and segmentation. However, current GCNs have limitations on implementation such as network architectures due to their irregular inputs. In contrast, convolutional neural networks (CNNs) are capable of extracting rich features from large-scale input data, but they do not support general graph inputs. To bridge the gap between GCNs and CNNs, in this paper we study the problem of how to effectively and efficiently map general graphs to 2D grids that CNNs can be directly applied to, while preserving graph topology as much as possible. We therefore propose two novel graph-to-grid mapping schemes, namely, graph-preserving grid layout (GPGL) and its extension Hierarchical GPGL (H-GPGL) for computational efficiency. We formulate the GPGL problem as integer programming and further propose an approximate yet efficient solver based on a penalized Kamada-Kawai method, a well-known optimization algorithm in 2D graph drawing. We propose a novel vertex separation penalty that encourages graph vertices to lay on the grid without any overlap. Along with this image representation, even extra 2D maxpooling layers contribute to the PointNet, a widely applied point-based neural network. We demonstrate the empirical success of GPGL on general graph classification with small graphs and H-GPGL on 3D point cloud segmentation with large graphs, based on 2D CNNs including VGG16, ResNet50 and multi-scale maxout (MSM) CNN.

Boundary-Aware Geometric Encoding for Semantic Segmentation of Point Clouds

Boundary information plays a significant role in 2D image segmentation, while usually being ignored in 3D point cloud segmentation where ambiguous features might be generated in feature extraction, leading to misclassification in the transition area between two objects. In this paper, firstly, we propose a Boundary Prediction Module (BPM) to predict boundary points. Based on the predicted boundary, a boundary-aware Geometric Encoding Module (GEM) is designed to encode geometric information and aggregate features with discrimination in a neighborhood, so that the local features belonging to different categories will not be polluted by each other. To provide extra geometric information for boundary-aware GEM, we also propose a light-weight Geometric Convolution Operation (GCO), making the extracted features more distinguishing. Built upon the boundary-aware GEM, we build our network and test it on benchmarks like ScanNet v2, S3DIS. Results show our methods can significantly improve the baseline and achieve state-of-the-art performance.

LRGNet: Learnable Region Growing for Class-Agnostic Point Cloud Segmentation

3D point cloud segmentation is an important function that helps robots understand the layout of their surrounding environment and perform tasks such as grasping objects, avoiding obstacles, and finding landmarks. Current segmentation methods are mostly class-specific, many of which are tuned to work with specific object categories and may not be generalizable to different types of scenes. This research proposes a learnable region growing method for class-agnostic point cloud segmentation, specifically for the task of instance label prediction. The proposed method is able to segment any class of objects using a single deep neural network without any assumptions about their shapes and sizes. The deep neural network is trained to predict how to add or remove points from a point cloud region to morph it into incrementally more complete regions of an object instance. Segmentation results on the S3DIS and ScanNet datasets show that the proposed method outperforms competing methods by 1%-9% on 6 different evaluation metrics.

Object-level classification of vegetable crops in 3D LiDAR point cloud using deep learning convolutional neural networks

Crop discrimination at the plant or patch level is vital for modern technology-enabled agriculture. Multispectral and hyperspectral remote sensing data have been widely used for crop classification. Even though spectral data are successful in classifying row-crops and orchards, they are limited in discriminating vegetable and cereal crops at plant or patch level. Terrestrial laser scanning is a potential remote sensing approach that offers distinct structural features useful for classification of crops at plant or patch level. The objective of this research is the improvement and application of an advanced deep learning framework for object-based classification of three vegetable crops: cabbage, tomato, and eggplant using high-resolution LiDAR point cloud. Point clouds from a terrestrial laser scanner (TLS) were acquired over experimental plots of the University of Agricultural Sciences, Bengaluru, India. As part of the methodology, a deep convolution neural network (CNN) model named CropPointNet is devised for the semantic segmentation of crops from a 3D perspective. The CropPointNet is an adaptation of the PointNet deep CNN model developed for the segmentation of indoor objects in a typical computer vision scenario. Apart from adapting to 3D point cloud segmentation of crops, the significant methodological improvements made in the CropPointNet are a random sampling scheme for training point cloud, and optimization of the network architecture to enable structural attribute-based segmentation of point clouds of unstructured objects such as TLS point clouds crops. The performance of the 3D crop classification has been validated and compared against two popular deep learning architectures: PointNet, and the Dynamic Graph-based Convolutional Neural Network (DGCNN). Results indicate consistent plant level object-based classification of crop point cloud with overall accuracies of 81% or better for all the three crops. The CropPointNet architecture proposed in this research can be generalized for segmentation and classification of other row crops and natural vegetation types.

HCFS3D: Hierarchical coupled feature selection network for 3D semantic and instance segmentation

Semantic segmentation and instance segmentation based on 3D point clouds involve significant challenges, specifically in the task of joint semantic and instance segmentation. The efficient and effective mutual assistance between semantic and instance segmentation is rarely considered and still remains an unaddressed research problem. To address this, herein, a novel and robust 3D point cloud segmentation framework employing hierarchical coupled feature selection, named HCFS3D, is proposed; this framework can jointly and reciprocally perform semantic and instance segmentation. The framework is designed to promote these two tasks to exploit beneficial information from each other, on a shallow as well as a deep level. Moreover, to prevent the network from overfitting and to improve performance, we designed a loss function called the Adaptive Smooth Loss, which can adaptively assign different weights to samples that are difficult to segment. Furthermore, joint semantic and instance conditional random fields are included in the proposed framework to further improve its performance. Extensive experiments based on different datasets and various backbone networks demonstrate that HCFS3D outperforms other state-of-the-art methods.

PIG-Net: Inception based deep learning architecture for 3D point cloud segmentation

Point clouds, being the simple and compact representation of surface geometry of 3D objects, have gained increasing popularity with the evolution of deep learning networks for classification and segmentation tasks. Unlike human, teaching the machine to analyze the segments of an object is a challenging task and quite essential in various machine vision applications. In this paper, we address the problem of segmentation and labelling of the 3D point clouds by proposing a inception based deep network architecture called PIG-Net, that effectively characterizes the local and global geometric details of the point clouds. In PIG-Net, the local features are extracted from the transformed input points using the proposed inception layers and then aligned by feature transform. These local features are aggregated using the global average pooling layer to obtain the global features. Finally, feed the concatenated local and global features to the convolution layers for segmenting the 3D point clouds. We perform an exhaustive experimental analysis of the PIG-Net architecture on two state-of-the-art datasets, namely, ShapeNet [1] and PartNet [2]. We evaluate the effectiveness of our network by performing ablation study.

Annotation Tool and Urban Dataset for 3D Point Cloud Semantic Segmentation

Accurate semantic segmentation of unstructured 3D point clouds requires large amount of annotated training data for deep learning. However, there is currently no free specialized software available that can efficiently annotate large 3D point clouds. We fill this gap by introducing PC-Annotate - a public annotation tool for 3D point cloud research. The proposed tool not only enables systematic annotation with a variety of fundamental volumetric shapes, but also provides useful functionalities of point cloud registration and the generation of volumetric samples that can be readily consumed by contemporary deep learning point cloud models. We also introduce a large outdoor public dataset for 3D semantic segmentation. The proposed dataset, PC-Urban is collected in a civic setup with Ouster LiDAR and labeled with PC-Annotate. It has over 4.3 billion points covering 66K frames and 25 annotated classes. Finally, we provide baseline semantic segmentation results on PC-Urban for popular recent techniques.

Learning from Synthetic Point Cloud Data for Historical Buildings Semantic Segmentation

Historical heritage is demanding robust pipelines for obtaining Heritage Building Information Modeling models that are fully interoperable and rich in their informative content. The definition of efficient Scan-to-BIM workflows represent a very important step toward a more efficient management of the historical real estate, as creating structured three-dimensional (3D) models from point clouds is complex and time-consuming. In this scenario, semantic segmentation of 3D Point Clouds is gaining more and more attention, since it might help to automatically recognize historical architectural elements. The way paved by recent Deep Learning approaches proved to provide reliable and affordable degrees of automation in other contexts, as road scenes understanding. However, semantic segmentation is particularly challenging in historical and classical architecture, due to the shapes complexity and the limited repeatability of elements across different buildings, which makes it difficult to define common patterns within the same class of elements. Furthermore, as Deep Learning models requires a considerably large amount of annotated data to be trained and tuned to properly handle unseen scenes, the lack of (big) publicly available annotated point clouds in the historical building domain is a huge problem, which in fact blocks the research in this direction. However, creating a critical mass of annotated point clouds by manual annotation is very time-consuming and impractical. To tackle this issue, in this work we explore the idea of leveraging synthetic point cloud data to train Deep Learning models to perform semantic segmentation of point clouds obtained via Terrestrial Laser Scanning. The aim is to provide a first assessment of the use of synthetic data to drive Deep Learning--based semantic segmentation in the context of historical buildings. To achieve this purpose, we present an improved version of the Dynamic Graph CNN (DGCNN) named RadDGCNN. The main improvement consists on exploiting the radius distance. In our experiments, we evaluate the trained models on synthetic dataset (publicly available) about two different historical buildings: the Ducal Palace in Urbino, Italy, and Palazzo Ferretti in Ancona, Italy. RadDGCNN yields good results, demonstrating improved segmentation performances on the TLS real datasets.

Coarse-to-Fine Segmentation on LiDAR Point Clouds in Spherical Coordinate and Beyond

Active sensors such as LiDARs (light detection and ranging) are popular in autonomous driving systems for perception and localization. Existing perception approaches process the rich 3D LiDAR point clouds for object detection, tracking and recognition. These methods generally require an initial segmentation procedure containing two steps: (1) filter points as ground and non-ground points, and (2) cluster the non-ground points into objects. Leveraging a range image created in Spherical coordinates, this paper proposes a field-tested coarse-to-fine 3D point cloud segmentation framework to achieve both speed and accuracy. Under this framework, a basic version and an advanced version of segmentation algorithms are presented. In the basic version, we demonstrate how a coarse-to-fine scheme is applied in a range image for ground filtering and object clustering. In the advanced version, we move forward to reduce the processing time and correct the motion distortion by directly dealing with the data packets instead of the range image. Tests in the well-known KITTI dataset and field experiments in public roads have shown that the method significantly improves the speed of 3D point cloud segmentation whilst maintains good accuracy.

DCARN: Deep Context Aware Recurrent Neural Network for Semantic Segmentation of Large Scale Unstructured 3D Point Cloud

Semantic segmentation of large unstructured 3D point clouds is important problem for 3D object recognition which in turn is essential to solving more complex tasks such as scene understanding. The problem is highly challenging owing to large scale of data, varying point density and localization errors of 3D points. Nevertheless, with recent successes of deep neural network architectures to solve complex 2D perceptual problems, several researchers have shown interest to translate the developed 2D networks to 3D point cloud segmentation by a prior voxelization step for an explicit neighborhood representation. However, such a 3D grid representation loses the fine details and inherent structure due to quantization artifacts. For this purpose, this paper proposes an approach to performing semantic segmentation of 3D point clouds by exploiting the idea of super-point based graph construction. The proposed architecture is composed of two cascaded modules including a light-weight representation learning module which uses unsupervised geometric grouping to partition the large-scale unstructured 3D point cloud and a deep context aware sequential network based on long short memory units and graph convolutions with embedding residual learning for semantic segmentation. The proposed model is evaluated on two standard benchmark datasets and achieves competitive performance with the existing state-of-the-art datasets. The code and the obtained results have been made public at https://github.com/saba155/DCARN .

Segmentation of unbalanced and in-homogeneous point clouds and its application to 3D scanned trees

Segmentation of 3D point clouds is still an open issue in the case of unbalanced and in-homogeneous data-sets. In the application context of the modeling of botanical trees, a fundamental challenge consists in separating the leaves from the wood. Based on deep learning and a class decision process, we propose an innovative method designed to separate leaf points from wood points in terrestrial LiDAR point clouds of trees. Although simple, our approach learns trees characteristic point patterns efficiently and robustly. To train our 3D deep learning model, we constructed a 3D labeled point cloud data-set of different tree species. Experiments show that our 3D deep representation together with our geometric approach leads to significant improvement over the state-of-the-art methods in segmentation task.

Comparing Machine and Deep Learning Methods for Large 3D Heritage Semantic Segmentation

In recent years semantic segmentation of 3D point clouds has been an argument that involves different fields of application. Cultural heritage scenarios have become the subject of this study mainly thanks to the development of photogrammetry and laser scanning techniques. Classification algorithms based on machine and deep learning methods allow to process huge amounts of data as 3D point clouds. In this context, the aim of this paper is to make a comparison between machine and deep learning methods for large 3D cultural heritage classification. Then, considering the best performances of both techniques, it proposes an architecture named DGCNN-Mod+3Dfeat that combines the positive aspects and advantages of these two methodologies for semantic segmentation of cultural heritage point clouds. To demonstrate the validity of our idea, several experiments from the ArCH benchmark are reported and commented.

A 1.15 TOPS/W Energy-Efficient Capsule Network Accelerator for Real-Time 3D Point Cloud Segmentation in Mobile Environment

An energy-efficient capsule network accelerator is proposed for real-time 3D point cloud segmentation in mobile devices. The proposed accelerator adopts the pipelined heterogeneous core architecture to achieve $1.55\times $ throughput enhancement. Furthermore, the proposed dynamic route skipping controller predicts unimportant operations and skips them to reduce the external memory access by 39.1%. At last, the new squash activation function unit exploits the look-up table (LUT) based computing with L2-norm approximation to minimize the power and area overhead. The proposed architecture is implemented with the FPGA, Altera Cyclone V 5CEBA9F31C7 and we test capsule network-based 3D point cloud segmentation application. It consumes 2.15 W power and shows 0.05 TOPS/W energy-efficiency. Also, the architecture is simulated with the 65nm CMOS technology, showing 94.3mW power consumption and 1.15 TOPS/W energy-efficiency.

SUPERVOXEL-BASED MULTI-SCALE POINT CLOUD SEGMENTATION USING FNEA FOR OBJECT-ORIENTED ROCK SLOPE CLASSIFICATION USING TLS

Abstract. Computer vision applications have been increasingly gaining space in the field of remote sensing and geosciences for automated terrain classification and semantic labelling purposes. The continuous and rapid development of monitoring techniques and enhancements in the spatial resolution of sensors have increased the demand for new remote sensing data analysis approaches. For semantic labelling of 2D (or 2.5D) image terrain representations for rock slopes, it has been shown that Object-Based Image Analysis (OBIA) results in high efficiency and accurate identification of landslide hazards. However, the application of such object-based approaches in 3D point cloud analysis is still under development for geospatial data analysis. In the field of engineering geology, which deals with complex rural landscapes, frequently the analysis needs to be conducted based solely on 3D geometrical information accounting for multiple scales simultaneously. In this study, the primary segmentation step of the object-based model is applied to a TLS-derived point cloud collected at a landslide-active rock slope. The 3D point cloud segmentation methodology proposed here builds on the principles of the Fractal Net Evolution Approach (FNEA). The objective is to provide a geometry-based point cloud segmentation framework that preserves the 3D character of the data throughout the process and favours the multi-scale analysis. The segmentation is performed on the basis of supervoxels based on purely geometrical local descriptors derived directly from the TLS point clouds and comprises the basis for the subsequent steps towards the development of an efficient Object-Based Point cloud Analysis (OBPA) framework in rock slope stability assessment by adding semantic meaning to the data through a homogenization process.

Semantic segmentation of 3D point cloud based on contextual attention CNN

Semantic segmentation of 3D point cloud based on contextual attention CNN

PGCNet: patch graph convolutional network for point cloud segmentation of indoor scenes

Semantic segmentation of 3D point clouds is a crucial task in scene understanding and is also fundamental to indoor scene applications such as indoor navigation, mobile robotics, augmented reality. Recently, deep learning frameworks have been successfully adopted to point clouds but are limited by the size of data. While most existing works focus on individual sampling points, we use surface patches as a more efficient representation and propose a novel indoor scene segmentation framework called patch graph convolution network (PGCNet). This framework treats patches as input graph nodes and subsequently aggregates neighboring node features by dynamic graph U-Net (DGU) module, which consists of dynamic edge convolution operation inside U-shaped encoder–decoder architecture. The DGU module dynamically update graph structures at each level to encode hierarchical edge features. Incorporating PGCNet, we can segment the input scene into two types, i.e., room layout and indoor objects, which is afterward utilized to carry out final rich semantic labeling of various indoor scenes. With considerable speedup training, the proposed framework achieves effective performance equivalent to state-of-the-art for segmenting standard indoor scene dataset.

Review of built heritage modelling: Integration of HBIM and other information techniques

Abstract Built heritage documentation involves the 3D modelling of the geometry (typically using 3D computer graphics, photogrammetry and laser scanning techniques) and information management of semantic knowledge (i.e., using Geographic Information System (GIS) and ontology tools). The recent developed Building Information Modelling (BIM) technique combines 3D modelling and information management. One of its modern application is heritage documentation and has generated a new concept of Historic/Heritage Building Information Modelling (HBIM). This paper summarises the applications of these information techniques on the built heritage documentation. We utilise Web of Science Collection to monitor the publications on built heritage documentation. We analyse the research trend in heritage modelling by comparing the attention paid by researchers before and during the 2010s. The results show that photogrammetry is always the most popular method in heritage modelling. More and more works in heritage modelling have begun to use laser scanning, computer science, GIS and especially BIM techniques. Ontologies and 3D computer graphics are traditional ways for heritage documentation. Moreover, we pay attention to the roles of BIM on heritage documentation and conduct a detailed discussion on how to extend the HBIM capabilities by integrating with other techniques. The integration provides possible enhanced functions in HBIM, including accurate parametric modelling from computer graphics, automatic semantic segmentation of 3D point cloud from reality-based modelling, spatial information management and analysis by GIS, and knowledge modelling by ontology.