Cloud Segmentation Research Articles

Building Extraction from LiDAR Point Clouds Based on Revised RandLA-Net

Abstract. 3D building models is crucial for applications in smart cities. Automatic reconstruction of 3D buildings has been investigated based on various data sources. Point clouds from airborne LiDAR scanners can be used to extract buildings data due to its high accuracy and point density. In this paper, we present a methodology to segment buildings and corresponding rooftop structure from point clouds. First, RandLA-Net, which is an efficient and lightweight neural network for semantic segmentation of large-scale point clouds, is revised and adopted for building segmentation. By implementing local feature aggregation of each point, RandLA-Net can effectively preserve geometric details in point clouds. Besides 3D coordinates of point clouds, we incorporated point attributes including pulse intensity and return numbers into the network as additional features. Feature normalizations are applied to the input features. To achieve a better result of the local feature aggregation, hyperparameters of the network are fine-tuned according to the density of points and building size. Based on the classified building point clouds, DBSCAN clustering algorithm is implemented for segmenting individual buildings. Elevation histogram analysis is conducted to determine optimal threshold values for delineating candidate rooftop point clouds of individual buildings. For the buildings with multiple rooftops, multiple elevation threshold values are necessary to extract corresponding rooftops or walls. Then DBSCAN is employed again for segmentation of individual rooftops and denoising of point clouds of each building. Finally, Alpha-shape analysis is applied based on adaptive threshold values to build the envelope of each rooftop. Experiments show that our implementation of building segmentation using RandLA-net achieves higher mean IoU (Intersection over Union) and better classification performance in building segmentation. ISPRS benchmark data was used in our experiment and our methodology produce results with accuracy of 90.79%.

Weakly Supervised Learning Method for Semantic Segmentation of Large-Scale 3D Point Cloud Based on Transformers

Abstract. Nowadays, semantic segmentation results of 3D point cloud have been widely applied in the fields of robotics, autonomous driving, and augmented reality etc. Thanks to the development of relevant deep learning models (such as PointNet), supervised training methods have become hotspot, in which two common limitations exists: inferior feature representation of 3D points and massive annotations. To improve 3D point feature, inspired by the idea of transformer, we employ a so-call LCP network that extracts better feature by investigating attentions between target 3D points and its corresponding local neighbors via local context propagation. Training transformer-based network needs amount of training samples, which itself is a labor-intensive, costly and error-prone work, therefore, this work proposes a weakly supervised framework, in particular, pseudo-labels are estimated based on the feature distances between unlabeled points and prototypes, which are calculated based on labeled data. The extensive experimental results show that, the proposed PL-LCP can yield considerable results (67.6% mIOU for indoor and 67.3% for outdoor) even if only using 1% real labels, and comparing to several state-of-the-art method using all labels, we achieve superior results in mIOU, OA for indoor (65.9%, 89.2%).

RANSAC-Based Planar Point Cloud Segmentation Enhanced by Normal Vector and Maximum Principal Curvature Clustering

Abstract. Planar feature segmentation is an essential task for 3D point cloud processing, finding many applications in various fields such as robotics and computer vision. The Random Sample Consensus (RANSAC) is one of the most common algorithms for the segmentation, but its performance, as given by the original form, is usually limited due to the use of a single threshold and interruption of similar planar features presented close to each other. To address these issues, we present a novel point cloud processing workflow which aims at developing an initial segmentation stage before the basic RANSAC is performed. Initially, normal vectors and maximum principal curvatures for each point of a given point cloud are analyzed and integrated. Subsequently, a subset of normal vectors and curvature is utilized to cluster planes with similar geometry based on the region growing algorithm, serving as a coarse but fast segmentation process. The segmentation is therefore refined with the RANSAC algorithm which can be now performed with higher accuracy and speed due to the reduced interference. After the RANSAC process is applied, resultant planar point clouds are built from the sparse ones via a point aggregation process based on geometric constraints. Four datasets (three real and one simulated) were used to verify the method. Compared to the classic segmentation method, our method achieves higher accuracy, with an RMSE from fitting equal to 0.0521 m, along with a higher recall of 93.31% and a higher F1-score of 95.38%.

Autonomous ultrasound scanning robotic system based on human posture recognition and image servo control: an application for cardiac imaging.

In traditional cardiac ultrasound diagnostics, the process of planning scanning paths and adjusting the ultrasound window relies solely on the experience and intuition of the physician, a method that not only affects the efficiency and quality of cardiac imaging but also increases the workload for physicians. To overcome these challenges, this study introduces a robotic system designed for autonomous cardiac ultrasound scanning, with the goal of advancing both the degree of automation and the quality of imaging in cardiac ultrasound examinations. The system achieves autonomous functionality through two key stages: initially, in the autonomous path planning stage, it utilizes a camera posture adjustment method based on the human body's central region and its planar normal vectors to achieve automatic adjustment of the camera's positioning angle; precise segmentation of the human body point cloud is accomplished through efficient point cloud processing techniques, and precise localization of the region of interest (ROI) based on keypoints of the human body. Furthermore, by applying isometric path slicing and B-spline curve fitting techniques, it independently plans the scanning path and the initial position of the probe. Subsequently, in the autonomous scanning stage, an innovative servo control strategy based on cardiac image edge correction is introduced to optimize the quality of the cardiac ultrasound window, integrating position compensation through admittance control to enhance the stability of autonomous cardiac ultrasound imaging, thereby obtaining a detailed view of the heart's structure and function. A series of experimental validations on human and cardiac models have assessed the system's effectiveness and precision in the correction of camera pose, planning of scanning paths, and control of cardiac ultrasound imaging quality, demonstrating its significant potential for clinical ultrasound scanning applications.

Hierarchical SVM for Semantic Segmentation of 3D Point Clouds for Infrastructure Scenes

The incorporation of building information modeling (BIM) has brought about significant advancements in civil engineering, enhancing efficiency and sustainability across project life cycles. The utilization of advanced 3D point cloud technologies such as laser scanning extends the application of BIM, particularly in operations and maintenance, prompting the exploration of automated solutions for labor-intensive point cloud modeling. This paper presents a demonstration of supervised machine learning—specifically, a support vector machine—for the analysis and segmentation of 3D point clouds, which is a pivotal step in 3D modeling. The point cloud semantic segmentation workflow is extensively reviewed to encompass critical elements such as neighborhood selection, feature extraction, and feature selection, leading to the development of an optimized methodology for this process. Diverse strategies are implemented at each phase to enhance the overall workflow and ensure resilient results. The methodology is then evaluated using diverse datasets from infrastructure scenes of bridges and compared with state-of-the-art deep learning models. The findings highlight the effectiveness of supervised machine learning techniques at accurately segmenting 3D point clouds, outperforming deep learning models such as PointNet and PointNet++ with smaller training datasets. Through the implementation of advanced segmentation techniques, there is a partial reduction in the time required for 3D modeling of point clouds, thereby further enhancing the efficiency and effectiveness of the BIM process.

A Point Cloud Segmentation Method for Pigs from Complex Point Cloud Environments Based on the Improved PointNet++

In animal husbandry applications, segmenting live pigs in complex farming environments faces many challenges, such as when pigs lick railings and defecate within the acquisition environment. The pig’s behavior makes point cloud segmentation more complex because dynamic animal behaviors and environmental changes must be considered. This further requires point cloud segmentation algorithms to improve the feature capture capability. In order to tackle the challenges associated with accurately segmenting point cloud data collected in complex real-world scenarios, such as pig occlusion and posture changes, this study utilizes PointNet++. The SoftPool pooling method is employed to implement a PointNet++ model that can achieve accurate point cloud segmentation for live pigs in complex environments. Firstly, the PointNet++ model is modified to make it more suitable for pigs by adjusting its parameters related to feature extraction and sensory fields. Then, the model’s ability to capture the details of point cloud features is further improved by using SoftPool as the point cloud feature pooling method. Finally, registration, filtering, and extraction are used to preprocess the point clouds before integrating them into a dataset for manual annotation. The improved PointNet++ model’s segmentation ability was validated and redefined with the pig point cloud dataset. Through experiments, it was shown that the improved model has better learning ability across 529 pig point cloud data sets. The optimal mean Intersection over Union (mIoU) was recorded at 96.52% and the accuracy at 98.33%. This study has achieved the automatic segmentation of highly overlapping pigs and pen point clouds. This advancement enables future animal husbandry applications, such as estimating body weight and size based on 3D point clouds.

Semantic segmentation of large-scale segmental lining point clouds using 3D deep learning

It is highly challenging to comprehensively measure the deformations of large-scale tunnel structures because of the numerous individual segments of tunnel linings. Laser scanning and 3D deep learning (DL) can provide abundant geometric data and the ability to interpret such data automatically. To fill the gap in applying 3D DL to the segment-wise segmentation of large-scale tunnel point clouds, a new 3D dataset and an optimized DL network are presented in this paper. The point cloud dataset was manually annotated by labeling every individual segment and element. Subsequently, a 3D DL network capable of handling large-scale point clouds (approximately 106 points) was developed to perform the data experiments. The influences of network parameters, namely, the number of neighbor points, layer number, down-sampling ratio, and channel number, were investigated in the data experiments. The results indicate that the number of neighboring points and channel number have a notable influence on the segmentation performance, with the best mean interaction over union reaching 85.5%. This study provides preliminary insights into the 3D DL network design for large-scale tunnel point clouds.

MSFA-Net: A Multiscale Feature Aggregation Network for Semantic Segmentation of Historical Building Point Clouds

In recent years, research on the preservation of historical architecture has gained significant attention, where the effectiveness of semantic segmentation is particularly crucial for subsequent repair, protection, and 3D reconstruction. Given the sparse and uneven nature of large-scale historical building point cloud scenes, most semantic segmentation methods opt to sample representative subsets of points, often leading to the loss of key features and insufficient segmentation accuracy of architectural components. Moreover, the geometric feature information at the junctions of components is cluttered and dense, resulting in poor edge segmentation. Based on this, this paper proposes a unique semantic segmentation network design called MSFA-Net. To obtain multiscale features and suppress irrelevant information, a double attention aggregation module is first introduced. Then, to enhance the model’s robustness and generalization capabilities, a contextual feature enhancement and edge interactive classifier module are proposed to train edge features and fuse the context data. Finally, to evaluate the performance of the proposed model, experiments were conducted on a self-curated ancient building dataset and the S3DIS dataset, achieving OA values of 95.2% and 88.7%, as well as mIoU values of 86.2% and 71.6%, respectively, further confirming the effectiveness and superiority of the proposed method.

A point cloud semantic segmentation method with augmented local geometry

With the development of 3D sensors, complex spatial structures can be established from point cloud data. Accurate semantic segmentation of 3D point clouds is beneficial for tasks such as robotic scene understanding. To extract the geometric features of the point cloud, most prior works use the xyz coordinates to potentially learn the shapes of 3D objects, which is not insufficient to handle complex scenes. In this paper, a Local Geometry Encoding (LGE) module is proposed to provide local normal information and express 3D shapes explicitly. Furthermore, considering that most prior methods use a single decoder to recover from abstract features, a Multi-Decoder (MD) ensemble module is proposed to restore multiple segmentation results from various scale feature maps in the encoder, and then a multi-attention fusion mechanism is utilized to produce single segmentation results. Benefiting from the local geometry encoding module and Multi-Decoder ensemble module, our network performs 6-fold cross-validation on the S3DIS dataset and achieves 75% mIoU with fewer parameters.

Design and application of data security privacy protection technology in dance action sequence arrangement and display system

In view of the excessive consumption of difficult resources in traditional dance choreography, insufficient data in choreography and data leakage crisis in the system, this paper puts forward artificial intelligence technology to study the protection of dance choreography and system data security and privacy. In this paper, the directed graph neural network is used to intelligently arrange the dance movements, and the Hidden Markov Model (HMM) is used to process the dance scene data so that the output data is highly similar to the original data. In the protection of data security and privacy, the method of data encryption is used, and the encrypted data needs to be decrypted by algorithm for reading. The experimental results show that the more complete the dance movement display, the higher the dance recognition rate, and the recognition rate of the dynamic light projection algorithm is greater than 90 % regardless of several sets of dance movements, while the recognition rate of the traditional light projection algorithm is not up to the standard regardless of several sets of movements, and the recognition rate of the point cloud segmentation method is higher than 90 % only in 3–4 sets of dance and 5–6 sets of dance movements.

Concrete Armour Unit Breakwater Physical Model Monitoring With 3D Modeling Tools

SEABIM® is a patented scan to BIM process that can generate a reliable and complete 3D model of a rubble-mound breakwater precast protective layer (Xbloc®, Accropode™, Core-loc™ etc.). Using computer vision techniques, the known 3D shape of the Concrete Armour Units (CAU) is detected in a high-resolution point cloud, which allows to obtain the position and orientation of each unit relative to the cloud. By superimposing 3D models produced from different scans at different dates, the movement of each block can then be quantified and represented as a vector. For natural riprap armours, a point cloud segmentation algorithm is applied. Each rock is identified, and then geometric characteristics of the armour can be computed (apparent rock diameters, placement density), and the movement between each scan can be monitored. This tool has been applied to numerous real-scale projects since 2019, both in the monitoring of newly built infrastructures and in the asset management phase (Aberdeen South Harbour, Calais port 2015, Nouvelle Route du Littoral etc...). Breakwater physical models in laboratories, in wave flumes or basins, also require monitoring of the movement of reduced-scale blocks between each wave series. With SEABIM® this movement can be computed with accuracy for each CAU, allowing for a consistent evaluation of the armour layer response to wave loads. The automatic block detection process can be launched on the point cloud from a survey of the initial physical model, obtained either by photogrammetry or LiDAR. Then a new survey is done after each wave series in order to create a sequence of 3D models. Those models can be compared by computing the displacement vectors of the centers of gravity of the units. The vectors can be visualized using color graded arrows within the 3D model. All the information can then be exported to a spreadsheet for further analysis. Using this technique, settlement, rocking and sliding effects can be identified. In this paper we will discuss in detail the adaptation of the survey techniques from real scale to laboratory projects (difference in scanning technique, scale effect on point cloud density etc.). Then we will explain the 3D modeling process and the applications of the 3D models for engineering analysis purposes.

SRX Net: A point cloud segmentation framework based on surface representation and X-Net

We propose a novel unified point cloud segmentation framework called SRX Net. This framework is composed of our proposed SurRep Group module and X-Net architecture, reconstructing point cloud segmentation models from two aspects: point cloud surface representation and network architecture. The SurRep Group can model the local surface structure of the point cloud through structures such as the Ellipsoid Surface Representation and Umbrella Surface Representation, providing rich geometric information to the network. The XNet establishes an inverse encoding–decoding path to deepen the modeling of shallow semantics, solving the interference of shallow semantics on deep semantics in prediction in the U-Net architecture. It also introduces a GLFEnhancement module to enhance each point’s perception of global information. As a unified framework, SRX Net can improve the semantic segmentation effects of various existing models. Extensive experimental results show that our model produces more accurate segmentation edges and significantly reduces discrete predicted points. Based on Point Transformer, our SRX-PT Net achieves state-of-the-art performance of 72.1 mIoU on S3DIS, Scannet V2, and WCS3D datasets, respectively.

A large-scale point cloud semantic segmentation neural network based on long-range contextual dependencies enhancement

ABSTRACT This letter presents a long-range contextual dependency enhanced network (LCDE-Net) for semantic segmentation of large-scale point cloud, which employs a U-shaped framework. Firstly, point clouds are subsampled with grid sampling and fed into convolutional layers to learn more representative local features of points. Then global and local encoders (GLE) are designed to exploit long-range contextual dependencies and local features simultaneously. The core of GLE consists of two parts: global feature enhancement (GFE) module and feature channel modulation (FCM) module. Secondly, through decoder layers, the encoded features are upsampled through the nearest-neighbour interpolation and aggregated with the intermediate encoded features by skip connections to capture multi-scale discriminative features for semantic segmentation of point cloud. Finally, via Fully Connection layer and Softmax classifier, each point’s label is assigned. Two different benchmark datasets are conducted to evaluate the performance of the proposed method, Experimental results report that the proposed LCDE-Net achieves 78.6% in terms of mean intersection over union (mIoU) on Semantic3D, and 68.2% on S3DIS, which is the highest among the comparison methods. The code of LCDE-Net is available at https://github.com/xrzmyz/LCDE-Net.

Machine and Deep Learning Implementations for Heritage Building Information Modelling: A Critical Review of Theoretical and Applied Research

Research domain and Problem: HBIM modelling from point cloud data has become a crucial research topic in the last decade since it is potentially considered as the central data model paving the way for the digital heritage practice beyond digitization. Reality Capture technologies such as terrestrial laser scanning, drone-mounted LiDAR sensors and photogrammetry enable the reality capture with a sub-millimetre accurate point cloud file that can be used as a reference file for Heritage Building Information Modelling (HBIM). However, HBIM modelling from the point cloud data of heritage buildings is mainly manual, error-prone, and time-consuming. Furthermore, image processing techniques are insufficient for classification and segmentation of point cloud data to speed up and enhance the current workflow for HBIM modelling. Due to the challenges and bottlenecks in the scan-to-HBIM process, which is commonly criticized as complex with its bespoke requirements, semantic segmentation of point clouds is gaining popularity in the literature. Research Aim and Methodology: Therefore, this paper aims to provide a thorough critical review of Machine Learning and Deep Learning methods for point cloud segmentation, classification, and BIM geometry automation for cultural heritage case study applications. Research findings: This paper files the challenges of HBIM practice and the opportunities for semantic point cloud segmentation found across academic literature in the last decade. Beyond definitions and basic occurrence statistics, this paper discusses the success rates and implementation challenges of machine and deep learning classification methods. Research value and contribution: This paper provides a holistic review of point cloud segmentation and its potential for further development and application in the Cultural Heritage sector. The critical analysis provides insight into the current state-of-the-art methods and advises on their suitability for HBIM projects. The review has identified highly original threads of research, which hold the potential to significantly influence practice and further applied research.

Hypergraph Position Attention Convolution Networks for 3D Point Cloud Segmentation

Point cloud segmentation, as the basis for 3D scene understanding and analysis, has made significant progress in recent years. Graph-based modeling and learning methods have played an important role in point cloud segmentation. However, due to the inherent complexity of point cloud data, it is difficult to capture higher-order and complex features of 3D data using graph learning methods. In addition, how to quickly and efficiently extract important features from point clouds also poses a great challenge to the current research. To address these challenges, we propose a new framework, called hypergraph position attention convolution networks (HGPAT), for point cloud segmentation. Firstly, we use hypergraph to model the higher-order relationships among point clouds. Secondly, in order to effectively learn the feature information of point cloud data, a hyperedge position attention convolution module is proposed, which utilizes the hyperedge–hyperedge propagation pattern to extract and aggregate more important features. Finally, we design a ResNet-like module to reduce the computational complexity of the network and improve its efficiency. We have conducted point cloud segmentation experiments on the ShapeNet Part and S3IDS datasets, and the experimental results demonstrate the effectiveness of the proposed method compared with the state-of-the-art ones.

An image segmentation and point cloud registration combined scheme for sensing of obscured tree branches

Automated robots are emerging as a solution for labor-intensive fruit orchard management. Three-dimensional (3D) reconstruction of tree branches is a fundamental requirement for robots to perform tasks like pruning and fruit harvesting. Current branch sensing methods often rely on planar segmentation with limited 3D information or computationally expensive point cloud segmentation, which may not be suitable for natural orchards with obscured tree branches. This study proposes a novel scheme that reconstructs occluded branches from RGB-D (Red-Green-Blue-Depth) images by integrating the point clouds converted from planar segmentation masks and depth images. The proposed approach extends the existing 2D branch sensing techniques to 3D, leveraging multi-view information. The deep learning models DeeplabV3+ and Pix2pix are employed to generate the segmentation masks, separately. And the Fast Global Registration (FGR) is used to register the multi-view point clouds. The results demonstrate that the output point clouds have at least a 24 % increase in the number of corresponding points after FGR. Furthermore, the time cost per hundred corresponding points is reduced by 85 % and 69 % when using the DeepLabV3 + and Pix2pix-based schemes, respectively, compared to the PointNet++ approach. These findings indicate that the proposed scheme significantly improves the sensing of occluded branches in terms of output richness and computational efficiency, making it applicable to natural orchard working spaces.

MRFA-Net: Multi-Scale Receptive Feature Aggregation Network for Cloud and Shadow Detection

The effective segmentation of clouds and cloud shadows is crucial for surface feature extraction, climate monitoring, and atmospheric correction, but it remains a critical challenge in remote sensing image processing. Cloud features are intricate, with varied distributions and unclear boundaries, making accurate extraction difficult, with only a few networks addressing this challenge. To tackle these issues, we introduce a multi-scale receptive field aggregation network (MRFA-Net). The MRFA-Net comprises an MRFA-Encoder and MRFA-Decoder. Within the encoder, the net includes the asymmetric feature extractor module (AFEM) and multi-scale attention, which capture diverse local features and enhance contextual semantic understanding, respectively. The MRFA-Decoder includes the multi-path decoder module (MDM) for blending features and the global feature refinement module (GFRM) for optimizing information via learnable matrix decomposition. Experimental results demonstrate that our model excelled in generalization and segmentation performance when addressing various complex backgrounds and different category detections, exhibiting advantages in terms of parameter efficiency and computational complexity, with the MRFA-Net achieving a mean intersection over union (MIoU) of 94.12% on our custom Cloud and Shadow dataset, and 87.54% on the open-source HRC_WHU dataset, outperforming other models by at least 0.53% and 0.62%. The proposed model demonstrates applicability in practical scenarios where features are difficult to distinguish.

Instance Segmentation of Sparse Point Clouds with Spatio-Temporal Coding for Autonomous Robot

Instance Segmentation of Sparse Point Clouds with Spatio-Temporal Coding for Autonomous Robot

Material augmented semantic segmentation of point clouds for building elements

AbstractPoint clouds are utilized to enable automated engineering applications for their ability to represent spatial geometry. However, they inherently lack detailed surface textures, posing challenges in differentiating objects at the texture level. Hence, this study introduces a 2D–3D fusing approach, leveraging material properties recognized from registered images as an augmented feature to enhance deep learning methods for the segmentation of building elements within point clouds. The proposed method was evaluated quantitatively on a 3D indoor data set with an implementation in an office room. The results are promising, showing improvement in recognition performance, particularly for objects with similar geometry but having different material properties. For instance, the segmentation of boards increased by 70.87%, and doors improved by 41.06% using the PointNet architecture. This enhanced segmentation not only reduces the time for interpreting point clouds but also has the potential to benefit downstream applications such as Scan‐to‐building information modeling (BIM), as defining regions for objects is essential.

Calculating Volume of Pig Point Cloud Based on Improved Poisson Reconstruction.

Pig point cloud data can be used to digitally reconstruct surface features, calculate pig body volume and estimate pig body weight. Volume, as a pig novel phenotype feature, has the following functions: (a) It can be used to estimate livestock weight based on its high correlation with body weight. (b) The volume proportion of various body parts (such as head, legs, etc.) can be obtained through point cloud segmentation, and the new phenotype information can be utilized for breeding pigs with smaller head volumes and stouter legs. However, as the pig point cloud has an irregular shape and may be partially missing, it is difficult to form a closed loop surface for volume calculation. Considering the better water tightness of Poisson reconstruction, this article adopts an improved Poisson reconstruction algorithm to reconstruct pig body point clouds, making the reconstruction results smoother, more continuous, and more complete. In the present study, standard shape point clouds, a known-volume Stanford rabbit standard model, a measured volume piglet model, and 479 sets of pig point cloud data with known body weight were adopted to confirm the accuracy and reliability of the improved Poisson reconstruction and volume calculation algorithm. Among them, the relative error was 4% in the piglet model volume result. The average absolute error was 2.664 kg in the weight estimation obtained from pig volume by collecting pig point clouds, and the average relative error was 2.478%. Concurrently, it was determined that the correlation coefficient between pig body volume and pig body weight was 0.95.