Point Cloud Segmentation Research Articles

Enhancing BDS-3 PPP-B2b real-time positioning performance by tightly integrating MEMS IMU and LiDAR in GNSS-Degraded environment

Currently, BDS-3 can provide satellite-based real-time precise point positioning (PPP) service via its PPP-B2b signal. However, in a complex Global Navigation Satellite System (GNSS)-degraded environment, the BDS-3 PPP-B2b positioning performance is severely influenced. A tightly coupled PPP/microelectromechanical system inertial measurement unit (MEMS IMU)/Light Detection and Ranging (LiDAR) combination is proposed, in which the original pseudorange and carrier observations, IMU measurements, and LiDAR planar features are fused via an extended Multi-State Constraint Kalman Filter (MSCKF). Moreover, the Galileo satellites are further utilized using the broadcast ephemerides to increase the number of available satellites. And to improve the real-time computing efficiency, a planar feature tracking model based on ground segmentation is adopted. The model consists of point cloud segmentation, planar feature extraction and fusion, and data correlation. To evaluate the effectiveness of the proposed method, the experiment is conducted at both a campus and a park. The result shows that approximately sub-meter-level positioning accuracy can be achieved using tightly coupled BDS-3 PPP-B2b, IMU, and LiDAR, while the positioning errors of BDS-3 PPP-B2b and BDS-3 PPP-B2b/IMU are both larger than 5 m. Furthermore, the MEMS IMU and LiDAR measurements show the capability of bridging the gaps in the GNSS data, which enables continuous and stable positioning. Moreover, compared to the state-of-the-art A-LOAM, LEGO-LOAM, and LIO-SAM frameworks, absolute trajectory errors of less than 1.0 and 2.5 m are achieved in the campus and park, respectively, using the proposed algorithm, with improvements of more than 3.40 and 1.58 times.

Multi-channel depth segmentation network based on 3D graph convolution algorithm and its application in point cloud segmentation

At the current stage, 3D graph convolutional algorithms face the following issues: (1) The problem of determining the neighborhood in graph convolution. (2) The issue of fusing features at different depths in deep graph convolutional algorithms. (3) The challenge of feature fusion and recognition in multi-path deep graph convolutional algorithms. Based on these, the paper "Multi-Path Deep Segmentation Network Based on 3D Graph Convolutional Algorithm and Its Application in Point Cloud Segmentation" is proposed. Inspired by visual computing theory, the neighborhood for 3D graph convolution is determined based on the receptive field theory, and a 3D graph convolutional algorithm based on visual selectivity is proposed; A single-link deep feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, it achieve the fusion of features at different depths learned by the graph convolutional algorithm by a pyramid learning method; A multi-link feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, and achieve multi-link feature fusion and segmentation to utilize the RPN calculation method. compared with algorithms such as PointNet, PointNet++, KPConv deform, 3D GCN, and My Algorithm, The segmentation performance and geometric invariance of the proposed algorithm are validated on the ShapeNetPart dataset and the custom Mortise_and_Tenon_DB dataset, and Experiments demonstrate that the proposed algorithm is correct and feasible, offers superior 3D point cloud segmentation performance, and exhibits strong geometric invariance.

Seabed Segmentation of Airborne Bathymetric Light Detection and Ranging Point Cloud Using Window-based Attention and Orthogonal Regularized PointNet

Seabed Segmentation of Airborne Bathymetric Light Detection and Ranging Point Cloud Using Window-based Attention and Orthogonal Regularized PointNet

Field-grown tomato yield estimation using point cloud segmentation with 3D shaping and RGB pictures from a field robot and digital single lens reflex cameras

The aim of this study was to estimate field-grown tomato yield (weight) and quantity of tomatoes using a self-developed robot and digital single lens reflex (DSLR) camera pictures. The authors suggest a new approach to predicting tomato yield that is based on images taken in the field, 3D scanning, and shape. Field pictures were used for tomato segmentation to determine the ripeness of the crop. A convolution neural network (CNN) model using TensorFlow library was devised for the segmentation of tomato berries along with a small robot, which had a 59.3 % F1 score. To enhance the accurate tomato crop model and to estimate the yield later, point cloud imaging was applied using a Ciclops 3D scanner. The best fitting sphere model was generated using the 3D model. The most optimal model was the 3D model, which gave the best representation and provided the weight of the tomatoes with a relative error of 21.90 % and a standard deviation of 17.9665 %. The results indicate a consistent object-based classification of the tomato crop above the plant/row level with an accuracy of 55.33 %, which is better than in-row sampling (images taken by the robot). By comparing the measured and estimated yield, the average difference for DSLR camera images was more favorable at 3.42 kg.

3D Point Cloud Semantic Segmentation based on Multi-scale Dense Nested Networks

Aiming at the problem that the relationship between geometric features and semantic features is ignored in the point cloud data downsampling process, which leads to inaccurate segmentation of object boundaries and structural details, this paper proposes a 3D point cloud semantic segmentation network based on multi-scale dense nested type. Firstly, a dense nested network architecture is constructed by nesting multiple multi-scale feature fusion modules to fuse multi-scale features of different directions between encoder-decoder paths, so as to effectively propagate local ge-ometric context information and enhance the ability of cross-scale information interaction. Secondly, a local feature aggregation unit is constructed in the multi-scale feature fusion module, which strengthens the structural awareness within the local point set based on graph convolution and attention mechanism, and promotes the complementarity of local geometric features and abstract semantic information. Then, the cross-layer multi-loss supervision module is combined to further optimize the multi-scale feature propagation, which makes the network training more stable and improves the point cloud segmentation accuracy. Finally, this paper verified the proposed network on the S3DIS dataset. The experimental results show that the proposed network has the mean intersection over Union of 71.2% and the overall accuracy of 88.7%, which is 1.2 and 0.7 percentage points higher than RandLA-Net, respectively, which proves that the proposed network can effectively improve the accuracy of 3D point cloud semantic segmentation.

Automated recognition and rebar dimensional assessment of prefabricated bridge components from low-cost 3D laser scanner

Dimension quality inspections for rebar spacing and length of prefabricated bridge components are critical for subsequent efficient assembly on-site. Currently, the traditional manual inspection method is time-consuming, labor-intensive, and error-prone. The existing commercial 3D LiDAR-based methods are difficult in making balances between inspection cost, efficiency, and accuracy. Thus, the automated recognition and segmentation method of 3D point clouds acquired by a self-developed low-cost LiDAR is proposed to evaluate the rebar spacing and length of bridge prefabricated components. Due to the high-cost commercial 3D laser scanner, a spinning 2D LiDAR-based low-cost 3D laser scanner device and geometric calibration model were developed to acquire the 3D spatial point cloud. Then, two-stage automated point cloud segmentation framework is proposed with coarse extraction of the point cloud in the interest region and fine recognition of point cloud using machine learning methods, including plane segmentation and circle fitting employing random sample consensus (RANSAC) algorithm, and rebars segmentation utilizing Gaussian mixture model (GMM) and density-based spatial clustering of applications with noise (DBSCAN) algorithm. The proposed system was evaluated in two prefabricated concrete columns and shows the potential for improving the quality inspection efficiency and accuracy.

An end-to-end radar pulse deinterleaving structure based on point cloud mapping

Radar pulse deinterleaving is a critical technology of electronic reconnaissance equipment. This paper proposes an end-to-end radar pulses deinterleaving structure based on point cloud mapping. The core idea is mapping radar pulse description word (PDW) to a point cloud for mimetic vision, which converts the radar pulse deinterleaving task into a point cloud segmentation task. This structure is characterized by lightweight and strong generalization compared to the image segmentation-based deinterleaving structure. Then this paper proposes a multi-stage graph convolution network (MSGCN) based on graph convolution for point cloud segmentation, which utilises the message passing mechanism of the graph structure to effectively extract, pass and fuse the features of different pulses, thus achieving better segmentation performance. The simulation experimental results show that the proposed method can effectively realize the deinterleaving of densely interleaved and overlapped pulses, and the method has an excellent robustness in pulse missing and spurious pulse interference scenarios.

3D reconstruction and volume measurement of irregular objects based on RGB-D camera

Abstract To address the challenge of measuring volumes of irregular objects, this paper proposes a volume measurement method based on 3D point cloud reconstruction. The point clouds of the object with multiple angles are obtained from an RGB-D camera mounted on a robotic arm, and then are reconstructed to form a whole complete point cloud to calculate the volume of the object. Firstly, the robotic arm is controlled to move to four angles for capturing the original point clouds of the target. Then, by using the rotation and translation matrices obtained from the calibration block pre-registration, the point clouds data from the four angles are fused and reconstructed. Subsequently, the issue of missing bottom point cloud data is addressed using a bottom-filling algorithm. Following this, the efficiency of the point cloud volume calculation algorithm is enhanced through the application of AABB filtering. Finally, the reconstructed point cloud volume is calculated using a slicing algorithm that integrates 2D point cloud segmentation and point cloud sorting. Experimental results show that this method achieves a volume measurement accuracy of over 95% for irregular objects and exhibits good robustness.

Enhancing explainability of deep learning models for point cloud analysis: a focus on semantic segmentation

ABSTRACT Semantic segmentation of point clouds plays a critical role in various applications, such as urban planning, infrastructure management, environmental analyses and autonomous navigation. Understanding the behaviour of deep neural networks (DNNs) in analysing point cloud data is essential for improving segmentation accuracy and developing effective network architectures and acquisition strategies. In this paper, we investigate the traits of some state-of-the-art neural networks using indoor and urban outdoor point cloud datasets. We compare PointNet, DGCNN, and BAAF-Net on specifically selected datasets, including synthetic and real-world environments. The chosen datasets are S3DIS, SynthCity, Semantic3D, and KITTI. We analyse the impact of different factors such as dataset type (synthetic vs. real), scene type (indoor vs. outdoor), and acquisition system (static vs. mobile sensors). Through detailed analyses and comparisons, we provide insights into the strengths and limitations not only of different network architectures in handling urban point clouds but also of their data structure. This study contributes to going beyond the mere and unconditional use of AI algorithms, trying to explain DNNs behaviour in point cloud analysis and paving the way for future research to enhance segmentation accuracy and develop possible guidelines both for network design and data acquisition in the geomatics field.

The Improvement of Density Peaks Clustering Algorithm and Its Application to Point Cloud Segmentation of LiDAR.

This work focuses on the improvement of the density peaks clustering (DPC) algorithm and its application to point cloud segmentation in LiDAR. The improvement of DPC focuses on avoiding the manual determination of the cut-off distance and the manual selection of cluster centers. And the clustering process of the improved DPC is automatic without manual intervention. The cut-off distance is avoided by forming a voxel structure and using the number of points in the voxel as the local density of the voxel. The automatic selection of cluster centers is realized by selecting the voxels whose gamma values are greater than the gamma value of the inflection point of the fitted γ curve as cluster centers. Finally, a new merging strategy is introduced to overcome the over-segmentation problem and obtain the final clustering result. To verify the effectiveness of the improved DPC, experiments on point cloud segmentation of LiDAR under different scenes were conducted. The basic DPC, K-means, and DBSCAN were introduced for comparison. The experimental results showed that the improved DPC is effective and its application to point cloud segmentation of LiDAR is successful. Compared with the basic DPC, K-means, the improved DPC has better clustering accuracy. And, compared with DBSCAN, the improved DPC has comparable or slightly better clustering accuracy without nontrivial parameters.

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

Towards automatic urban tree inventory: Enhancing tree instance segmentation via moving object removal and a chord length-based DBH estimation approach

To enhance urban forestry efficacy in Hong Kong, implementing a paradigm shift towards an automated urban tree inventory that utilizes advanced sensing technologies and artificial intelligence is essential for streamlined data collection and analysis. This study advances this objective by creating a comprehensive framework for estimating diameter at breast height (DBH) and extracting tree images. This framework encompasses five key stages: (1) data acquisition utilizing StructXray, a mobile mapping system equipped with a 360° camera and a multi-beam flash LiDAR sensor; (2) vegetation point clouds extraction using deep learning techniques; (3) individual tree segmentation through machine learning algorithms; (4) DBH estimation; and (5) tree image extraction. Six datasets were collected, yielding tree detection precision, recall and F1 score of 0.88, 0.95 and 0.91 respectively. The presence of moving objects within the 3D point cloud map, exhibiting diverse geometric structures, hinders precise vegetation point cloud segmentation by the pointwise neural network. To tackle this challenge, SalsaNext was employed to rectify the predictions of a pointwise neural network, specifically RandLA-Net in this study, eliminating 91 % of misclassified moving object point clouds and completely removing them from 47 % of affected individual tree point clouds. Additionally, a chord length-based method was proposed to enhance DBH estimation accuracy by dividing the point cloud slice into sectors and summing the chord lengths to estimate the tree trunk perimeter. Compared to the ellipse least squares fitting method, this approach reduced the root-mean-square error of the estimated DBH by 1.31 cm.

LESA-Net: Semantic segmentation of multi-type road point clouds in complex agroforestry environment

Point-cloud semantic segmentation is a visual task essential for agricultural robots to comprehend natural agroforestry environments. However, owing to the extremely large amount of point-cloud data in agroforestry environments, learning effective features for semantic segmentation from large-scale point clouds is challenging. Therefore, to address this issue and achieve accurate semantic segmentation of different types of road-surface point clouds in large-scale agroforestry environments, this study proposes a point-cloud semantic segmentation network framework based on double-distance self-attention. First, a point-cloud local feature enhancement module is proposed. This module primarily extends the receptive field and enhances the generalizability of multidimensional features by incorporating reflection intensity information and a spatial feature-encoding block that is enhanced with contextual semantic information. Second, we introduce a dual-distance attention pooling (DDAPS) block based on the self-attention mechanism. This block initially learns the feature representation of the local neighborhood of each point through the self-attention mechanism. Then, it uses the DDAPS block to aggregate more discriminative local neighborhood point features. Finally, extensive experimental results on large-scale point-cloud datasets, SemanticKITTI and RELLIS-3D, demonstrate that our algorithm outperforms similar algorithms in large-scale agroforestry environments.

A domain adaptation framework for cross-modality SAR 3D reconstruction point clouds segmentation utilizing LiDAR data

Synthetic aperture radar (SAR) 3D point clouds reconstruction can eliminate the problems of layover in 2D SAR image projections, the recognition of the reconstructed point cloud can significantly enhance target identification and information extraction. Current SAR 3D point cloud segmentation methods, based on traditional machine learning clustering approaches, suffer from poor automation and accuracy. However, the absence of publicly labeled SAR datasets impedes the progress of deep learning-based SAR 3D point cloud segmentation methods. To tackle the aforementioned challenges, we introduce, for the first time, an alternative training approach for SAR 3D point cloud segmentation using LiDAR annotated data, which offers a more abundant sample pool, thus alleviating the lack of training sets. Nevertheless, a segmentation model trained on LiDAR point clouds exhibits a significant decline in performance when directly applied to SAR 3D reconstructed point clouds due to the cross-domain discrepancy. This research presents a pioneering domain adaptation 3D semantic segmentation framework to implement cross-modal learning for SAR point clouds. In our scheme, a simple yet effective technique is developed to achieve segmentation of SAR double-bounce scattering regions called SARDBS-Mix, which employs a mixing strategy, to capture the distinctive reflection characteristics of SAR data. Furthermore, we implement approaches including center alignment and normalization, local augmentation, and weighted cross entropy to mitigate LiDAR and SAR domain gap and class imbalances. The experimental results validate the feasibility and effectiveness of the proposed method for SAR 3D reconstructed point cloud segmentation.

A Hierarchical Neural Network for Point Cloud Segmentation and Geometric Primitive Fitting.

Automated generation of geometric models from point cloud data holds significant importance in the field of computer vision and has expansive applications, such as shape modeling and object recognition. However, prevalent methods exhibit accuracy issues. In this study, we introduce a novel hierarchical neural network that utilizes recursive PointConv operations on nested subdivisions of point sets. This network effectively extracts features, segments point clouds, and accurately identifies and computes parameters of regular geometric primitives with notable resilience to noise. On fine-grained primitive detection, our approach outperforms Supervised Primitive Fitting Network (SPFN) by 18.5% and Cascaded Primitive Fitting Network (CPFN) by 11.2%. Additionally, our approach consistently maintains low absolute errors in parameter prediction across varying noise levels in the point cloud data. Our experiments validate the robustness of our proposed method and establish its superiority relative to other methodologies in the extant literature.

3D surface change detection of slope using 3D point cloud based on construction of double surface

Aiming at the complex geological conditions of slopes and the difficulty of manual investigation, this paper proposes a method of constructing double surface based on 3D point cloud data to estimate the depth of slope deformation and disaster analysis. Terrestrial Laser Scanner is utilized to collect the slope point cloud data. Through the construction of double surface to simulate the original slope, the deformation depth is estimated and the depth data are fused into the point cloud to realize the automatic identification of slope deformation disaster and the calculation of deformation depth, and provide the deformation depth map. The shortcomings of the traditional Multiscale Model and Model Point Cloud Comparison (M3C2) algorithm, which needs to capture multiple time periods of point cloud data for calculation can be overcome. And there is no need to recognize the segmentation of the slope point cloud. The study results show that the maximum relative error of slope deformation is 10.82% in slope deformation. The proposed method is suitable for detecting slope hazards in various scenarios, providing a basis for further analysis and the development of effective erosion control measures. It introduces a new technology for daily monitoring of slope deformations and can be extended to drone-based LiDAR applications.

LSGRNet: Local Spatial Latent Geometric Relation Learning Network for 3D point cloud semantic segmentation

In recent years, remarkable ability has been demonstrated by the Transformer model in capturing remote dependencies and improving point cloud segmentation performance. However, localized regions separated from conventional sampling architectures have resulted in the destruction of structural information of instances and a lack of exploration of potential geometric relationships between localized regions. To address this issue, a Local Spatial Latent Geometric Relation Learning Network (LSGRNet) is proposed in this paper, with the geometric properties of point clouds serving as a reference. Specifically, spatial transformation and gradient computation are performed on the local point cloud to uncover potential geometric relationships within the local neighborhood. Furthermore, a local relationship aggregator based on semantic and geometric relationships is constructed to enable the interaction of spatial geometric structure and information within the local neighborhood. Simultaneously, boundary interaction feature learning module is employed to learn the boundary information of the point cloud, aiming to better describe the local structure. The experimental results indicate that excellent segmentation performance is exhibited by the proposed LSGRNet in benchmark tests on the indoor datasets S3DIS and ScanNetV2, as well as the outdoor datasets SemanticKITTI and Semantic3D.

A transfer learning-based network model integrating kernel convolution with graph attention mechanism for point cloud segmentation of livestock

Non-contact body size measurement has become a hot research topic in intelligent livestock farming. In regard to body size measurement of large livestock, such as cattle, collecting a substantial number of point clouds is frequently involved. The direct calculation of all point clouds for body size measurement can be impacted as point clouds of different body parts may interfere with each other, which poses huge challenges for the positioning of key points and induces inaccurate positioning, resulting in measurement errors. The accuracy of body size measurement can be improved by segmenting point clouds of different body parts from each other, key measurement points can be precisely located, thus enhancing the accuracy of body size measurement. In this paper, we propose a network model initially trained for pig point cloud segmentation based on the Kernel Convolution integrated with Graph Attention Mechanism (KCGATNet for short), which, through transfer learning techniques, can also be used to achieve successful segmentation of various cattle point clouds using only 7 training samples. The model utilizes two core modules, Kernel Convolution (KC) and Point-based Graph Attention Mechanism (P-GAT), to extract local neighborhood features of point clouds. When using pig body point clouds as training data, it achieved precise segmentation of the head, ears, limbs, torso, and tail of pigs through a downsampling-upsampling architecture. On the test set of pig point clouds, Overall Accuracy (OA) reached 98.1% and mean Intersection over Union (mIoU) was up to 90.5%. Furthermore, when this model served as a pre-trained model and underwent transfer learning using 7 sets of annotated data of Simmental cattle, it achieved a mIoU of 90.1% on a test set of 93 Simmental cattle, 89.6% on a test set of 439 dairy buffalo, 90.2% on a test set of 103 Hereford cattle, and 90.0% on a test set of 119 Black Angus cattle. The experimental outcomes fully demonstrate the robustness of the proposed livestock point cloud segmentation model, KCGATNet. With transfer learning of a small sample size, it can reliably perform point cloud segmentation on a wide range of different breeds of quadrupedal livestock, saving a significant amount of time spent on manual annotation and improving the efficiency of livestock point cloud segmentation models.

A Precise Segmentation Algorithm of Pumpkin Seedling Point Cloud Stem Based on CPHNet.

Accurate segmentation of the stem of pumpkin seedlings has a great influence on the modernization of pumpkin cultivation, and can provide detailed data support for the growth of pumpkin plants. We collected and constructed a pumpkin seedling point cloud dataset for the first time. Potting soil and wall background in point cloud data often interfere with the accuracy of partial cutting of pumpkin seedling stems. The stem shape of pumpkin seedlings varies due to other environmental factors during the growing stage. The stem of the pumpkin seedling is closely connected with the potting soil and leaves, and the boundary of the stem is easily blurred. These problems bring challenges to the accurate segmentation of pumpkin seedling point cloud stems. In this paper, an accurate segmentation algorithm for pumpkin seedling point cloud stems based on CPHNet is proposed. First, a channel residual attention multilayer perceptron (CRA-MLP) module is proposed, which suppresses background interference such as soil. Second, a position-enhanced self-attention (PESA) mechanism is proposed, enabling the model to adapt to diverse morphologies of pumpkin seedling point cloud data stems. Finally, a hybrid loss function of cross entropy loss and dice loss (HCE-Dice Loss) is proposed to address the issue of fuzzy stem boundaries. The experimental results show that CPHNet achieves a 90.4% average cross-to-merge ratio (mIoU), 93.1% average accuracy (mP), 95.6% average recall rate (mR), 94.4% F1 score (mF1) and 0.03 plants/second (speed) on the self-built dataset. Compared with other popular segmentation models, this model is more accurate and stable for cutting the stem part of the pumpkin seedling point cloud.

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion