Cloud Segmentation Research Articles

LSGRNet: Local Spatial Latent Geometric Relation Learning Network for 3D point cloud semantic segmentation

In recent years, remarkable ability has been demonstrated by the Transformer model in capturing remote dependencies and improving point cloud segmentation performance. However, localized regions separated from conventional sampling architectures have resulted in the destruction of structural information of instances and a lack of exploration of potential geometric relationships between localized regions. To address this issue, a Local Spatial Latent Geometric Relation Learning Network (LSGRNet) is proposed in this paper, with the geometric properties of point clouds serving as a reference. Specifically, spatial transformation and gradient computation are performed on the local point cloud to uncover potential geometric relationships within the local neighborhood. Furthermore, a local relationship aggregator based on semantic and geometric relationships is constructed to enable the interaction of spatial geometric structure and information within the local neighborhood. Simultaneously, boundary interaction feature learning module is employed to learn the boundary information of the point cloud, aiming to better describe the local structure. The experimental results indicate that excellent segmentation performance is exhibited by the proposed LSGRNet in benchmark tests on the indoor datasets S3DIS and ScanNetV2, as well as the outdoor datasets SemanticKITTI and Semantic3D.

Cloud Segmentation and Matching Using Deep Learning in All-Sky Images

Cloud Segmentation and Matching Using Deep Learning in All-Sky Images

A transfer learning-based network model integrating kernel convolution with graph attention mechanism for point cloud segmentation of livestock

Non-contact body size measurement has become a hot research topic in intelligent livestock farming. In regard to body size measurement of large livestock, such as cattle, collecting a substantial number of point clouds is frequently involved. The direct calculation of all point clouds for body size measurement can be impacted as point clouds of different body parts may interfere with each other, which poses huge challenges for the positioning of key points and induces inaccurate positioning, resulting in measurement errors. The accuracy of body size measurement can be improved by segmenting point clouds of different body parts from each other, key measurement points can be precisely located, thus enhancing the accuracy of body size measurement. In this paper, we propose a network model initially trained for pig point cloud segmentation based on the Kernel Convolution integrated with Graph Attention Mechanism (KCGATNet for short), which, through transfer learning techniques, can also be used to achieve successful segmentation of various cattle point clouds using only 7 training samples. The model utilizes two core modules, Kernel Convolution (KC) and Point-based Graph Attention Mechanism (P-GAT), to extract local neighborhood features of point clouds. When using pig body point clouds as training data, it achieved precise segmentation of the head, ears, limbs, torso, and tail of pigs through a downsampling-upsampling architecture. On the test set of pig point clouds, Overall Accuracy (OA) reached 98.1% and mean Intersection over Union (mIoU) was up to 90.5%. Furthermore, when this model served as a pre-trained model and underwent transfer learning using 7 sets of annotated data of Simmental cattle, it achieved a mIoU of 90.1% on a test set of 93 Simmental cattle, 89.6% on a test set of 439 dairy buffalo, 90.2% on a test set of 103 Hereford cattle, and 90.0% on a test set of 119 Black Angus cattle. The experimental outcomes fully demonstrate the robustness of the proposed livestock point cloud segmentation model, KCGATNet. With transfer learning of a small sample size, it can reliably perform point cloud segmentation on a wide range of different breeds of quadrupedal livestock, saving a significant amount of time spent on manual annotation and improving the efficiency of livestock point cloud segmentation models.

A Precise Segmentation Algorithm of Pumpkin Seedling Point Cloud Stem Based on CPHNet.

Accurate segmentation of the stem of pumpkin seedlings has a great influence on the modernization of pumpkin cultivation, and can provide detailed data support for the growth of pumpkin plants. We collected and constructed a pumpkin seedling point cloud dataset for the first time. Potting soil and wall background in point cloud data often interfere with the accuracy of partial cutting of pumpkin seedling stems. The stem shape of pumpkin seedlings varies due to other environmental factors during the growing stage. The stem of the pumpkin seedling is closely connected with the potting soil and leaves, and the boundary of the stem is easily blurred. These problems bring challenges to the accurate segmentation of pumpkin seedling point cloud stems. In this paper, an accurate segmentation algorithm for pumpkin seedling point cloud stems based on CPHNet is proposed. First, a channel residual attention multilayer perceptron (CRA-MLP) module is proposed, which suppresses background interference such as soil. Second, a position-enhanced self-attention (PESA) mechanism is proposed, enabling the model to adapt to diverse morphologies of pumpkin seedling point cloud data stems. Finally, a hybrid loss function of cross entropy loss and dice loss (HCE-Dice Loss) is proposed to address the issue of fuzzy stem boundaries. The experimental results show that CPHNet achieves a 90.4% average cross-to-merge ratio (mIoU), 93.1% average accuracy (mP), 95.6% average recall rate (mR), 94.4% F1 score (mF1) and 0.03 plants/second (speed) on the self-built dataset. Compared with other popular segmentation models, this model is more accurate and stable for cutting the stem part of the pumpkin seedling point cloud.

ADAC: an active domain adaptive network with progressive learning strategy for cloud detection of remote sensing imagery

ABSTRACT To enhance the adaptability and application capability of the cloud detection model in different remote sensing satellite domains, unsupervised domain adaptation methods are employed to improve the model’s robustness and transferability. However, conducting fully unlabeled training in the target domain to be adapted may impose difficulty on further improving the accuracy. It is more practical to obtain a few labeled target data for domain adaptation. Additionally, the detection of complex cloud scenes remains a persistent challenge. To address these issues, an Active Domain Adaptation network for Cloud detection (ADAC) is proposed in this paper. This network employs active learning to select a small number of representative samples for annotation, facilitating the transfer of a source domain-trained model to the target domain. Moreover, a novel Generator (G) designed for cloud segmentation is developed, where G utilizes a progressive learning strategy by dividing the network into two stages to learn gradient information, guiding cloud boundary segmentation and enabling the distinction between cloud-like objects and clouds. Furthermore, G integrates Transformer and Convolutional Neural Network to address the issues of misclassifying thin clouds. The proposed method is validated on Landsat 8 and GF-2 datasets, representing the source and target domains, respectively. The experimental results demonstrate ADAC achieved a 4% improvement in the overall accuracy with only 5% labeled training samples in the target domain, achieving the effective adaptation performance in the GF-2 domain.

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion

Camera LiDAR calibration: an automatic and accurate method with novel PLE metrics

Abstract Camera-LiDAR calibration is a crucial aspect of perception systems and sensor fusion in various fields, facilitating the fusion of data from cameras and LiDAR sensors for applications such as autonomous vehicles, robotics, and augmented reality. In this paper, we presented a novel multi-modal fusion method that introduces an efficient camera-LiDAR joint calibration technique using a simple checkerboard. Our method estimates the 3D rigid transformation of the camera system with respect to the LiDAR frame by establishing 2D-to-3D point correspondences for geometric calibration. We improved the idea of the traditional PNP (Perspective-N-Point) algorithm by employing point cloud segmentation and clustering to re-match the checkerboard. When compared with the calibration methods in the Autoware and MATLAB calibration toolbox, the reprojection error was improved by 38.13\% and 58.30\% respectively. The average translation error was improved by 78.26\% and 51.61\% respectively, while the average rotation error is improved by 75\% and 60.78\% respectively, which has a great improvement in all three areas. At the same time, we propose a new evaluation metric for joint calibration results based on reprojected images: PLE (Pixel-Level Evaluation) index to reflect the accuracy of the joint calibration of the camera and LiDAR. An automatic calibration software has been developed for the calibration of the camera's intrinsic parameters as well as for the joint calibration of the camera-LIDAR extrinsic parameters. We have extensively validated our algorithm using the Intel RealSense Depth D435 Camera and LeiShen C16-151B LiDAR.

PA-Net: Trustworthy weakly supervised point cloud semantic segmentation with primary–auxiliary structure

Weakly supervised point cloud segmentation is a hot research topic with significant implications for practical applications such as autonomous driving. However, existing weakly supervised approaches primarily focus on improving prediction results while neglecting the reliability of the inference outcomes. To address this issue, this paper proposes a primary–auxiliary network PA-Net for point cloud segmentation and uncertainty estimation. PA-Net achieves excellent segmentation results and enables effective uncertainty assessment. Specifically, we first train PA-Net on sparsely annotated point cloud data, which allows our network to capture multi-scale information of labeled points and provide additional supervisory signals. Subsequently, we utilize the trained model for inference. We mainly conducted experiments on the indoor point cloud dataset S3DIS in a 1% labeled setting. PA-Net achieved a mean Intersection over Union (mIoU) of 64.5%, which surpasses most weakly-supervised methods. In particular, it exceeded PSD by 3.5% mIoU and SQN by 0.8% mIoU. Similarly, on the outdoor dataset Toronto3D, PA-Net also outperformed most weakly supervised methods. With only 0.01% labeling, it surpassed PSD by 11.99% mIoU and SQN by 20.34% mIoU. Lastly, to validate the effectiveness of uncertainty estimation, we visualized scene uncertainty, thereby demonstrating the practicality of our uncertainty estimation approach. Code will be made publicly available at https://github.com/NiuYingchun/PA-Net.

RTMFusion: An enhanced dual-stream architecture algorithm fusing RGB and depth features for instance segmentation of tomato organs

Accurate and rapid acquisition of semantically annotated point clouds from tomato cultivation regions is crucial for autonomous tomato harvesting robots. This entails the robot’s ability to perceive tomato point clouds from various locations and identify the stems and pedicels connecting the fruits. However, amidst background interference from similar stem and pedicel features, the heterogeneous growth postures of tomato clusters and the slender and short nature of pedicels pose challenges in semantic point cloud segmentation and matching of different tomato fruits, pedicels, and stems. Leveraging the salient feature representation of depth images in fine contours and distance dimensions, this paper introduces RTMFusion, a lightweight dual-stream backbone instance segmentation model that integrates color images and depth images. A Depth Enhanced Feature Fusion module is designed for multi-modal feature fusion. Fruit-bearing organ matching is facilitated by a method based on distance constraints and conditional search. Key characteristics are extracted from semantic point clouds, and an optimal condition search is conducted on candidates meeting constraint conditions. On the test dataset, our model improved the mask mAP by 4.6 % compared to the baseline, achieving 72.1 % with a speed of 17.1 ms. Our matching method achieved 99.2 % pedicel matching accuracy and 97.3 % stem matching accuracy on 150 greenhouse samples. These results demonstrate the effectiveness of accurately identifying and matching tomato organs in real-world scenarios, addressing the challenge from complex greenhouse environments. This integrated visual system enhances the success rate of visual identification and the precision of cutting position localization for tomato harvesting robots.

DSC-Net: learning discriminative spatial contextual features for semantic segmentation of large-scale ancient architecture point clouds

Semantic segmentation of point cloud data of architectural cultural heritage is of significant importance for HBIM modeling, disease extraction and analysis, and heritage restoration research fields. In the semantic segmentation task of architectural point cloud data, especially for the protection and analysis of architectural cultural heritage, the previous deep learning methods have poor segmentation effects due to the complexity and unevenness of the data, the high geometric feature similarity between different components, and the large scale changes. To this end, this paper proposes a novel encoder-decoder architecture called DSC-Net. It consists of an encoder-decoder structure based on point random sampling and several fully connected layers for semantic segmentation. To overcome the loss of key features caused by random downsampling, DSC-Net has developed two new feature aggregation schemes: the enhanced dual attention pooling module and the global context feature module, to learn discriminative features for the challenging scenes mentioned above. The former fully considers the topology and semantic similarity of neighboring points, generating attention features that can distinguish categories with similar structures. The latter uses spatial location and neighboring volume ratio to provide an overall view of different types of architectural scenes, helping the network understand the spatial relationships and hierarchical structures between different architectural elements. The proposed modules can be easily embedded into various network architectures for point cloud semantic segmentation. We conducted experiments on multiple datasets, including the ancient architecture dataset, the ArCH architectural cultural heritage dataset, and the publicly available architectural segmentation dataset S3DIS. The results show that the mIoU reached 63.56%, 55.84%, and 71.03% respectively. The experimental results prove that our method has the best segmentation effect in dealing with challenging architectural cultural heritage data and also demonstrates its practicality in a wider range of architectural point cloud segmentation applications.

TransWallNet: High-Performance Semantic Segmentation of Large-Scale and Multifeatured Point Clouds of Building Gables

TransWallNet: High-Performance Semantic Segmentation of Large-Scale and Multifeatured Point Clouds of Building Gables

Sequential Point Clouds: A Survey.

Point clouds have garnered increasing research attention and found numerous practical applications. However, many of these applications, such as autonomous driving and robotic manipulation, rely on sequential point clouds, essentially adding a temporal dimension to the data (i.e., four dimensions) because the information of the static point cloud data could provide is still limited. Recent research efforts have been directed towards enhancing the understanding and utilization of sequential point clouds. This paper offers a comprehensive review of deep learning methods applied to sequential point cloud research, encompassing dynamic flow estimation, object detection & tracking, point cloud segmentation, and point cloud forecasting. This paper further summarizes and compares the quantitative results of the reviewed methods over the public benchmark datasets. Ultimately, the paper concludes by addressing the challenges in current sequential point cloud research and pointing towards promising avenues for future research.

Learning-Based Sampling Method for Point Cloud Segmentation

Learning-Based Sampling Method for Point Cloud Segmentation

Point Cloud Measurement of Rubber Tread Dimension Based on RGB-Depth Camera

To achieve an accurate measurement of tread size after fixed-length cutting, this paper proposes a point-cloud-based tread size measurement method. Firstly, a mathematical model of corner points and a reprojection error is established, and the optimal solution of the number of corner points is determined by the non-dominated sorting genetic algorithm II (NSGA-II), which reduces the reprojection error of the RGB-D camera. Secondly, to address the problem of the low accuracy of the traditional pixel metric ratio measurement method, the random sampling consensus point cloud segmentation algorithm (RANSAC) and the oriented bounding box (OBB) collision detection algorithm are introduced to complete the accurate detection of the tread size. By comparing the absolute error and relative error data of several groups of experiments, the accuracy of the detection method in this paper reaches 1 mm, and the measurement deviation is between 0.14% and 2.67%, which is in line with the highest accuracy standard of the national standard. In summary, the RGB-D visual inspection method constructed in this paper has the characteristics of low cost and high inspection accuracy, which is a potential solution to enhance the pickup guidance of tread size measurement.

MP-DGCNN for the semantic segmentation of Chinese ancient building point clouds

Point cloud semantic segmentation is a key step in the scan-to-HBIM process. In order to reduce the information in the process of DGCNN, this paper proposes a Mix Pooling Dynamic Graph Convolutional Neural Network (MP-DGCNN) for the segmentation of ancient architecture point clouds. The proposed MP-DGCNN differs from DGCNN mainly in two aspects: (1) to more comprehensively characterize the local topological structure of points, the edge features are redefined, and distance and neighboring points are added to the original edge features; (2) based on a Multilayer Perceptron (MLP), an internal feature adjustment mechanism is established, and a learnable mix pooling operator is designed by fusing adaptive pooling, max pooling, average pooling, and aggregation pooling, to learn local graph features from the point cloud topology. To verify the proposed algorithm, experiments are conducted on the Qutan Temple point cloud dataset, and the results show that compared with PointNet, PointNet++, DGCNN, GACNet and LDGCNN, the MP-DGCNN segmentation network achieves the highest OA, mIOU and mAcc, reaching 90.19%,65.34% and 79.41%, respectively.

A dual attentional skip connection based Swin‐UNet for real‐time cloud segmentation

AbstractDeveloping real‐time cloud segmentation technology is urgent for many remote sensing based applications such as weather forecasting. Existing deep learning based cloud segmentation methods involve two shortcomings. (a): They tend to produce discontinuous boundaries and fail to capture less salient feature, which corresponds to thin cloud pixels; (b): they are unrobust towards different scenarios. Those issues are circumvented by integrating U‐Net and the swin transformer together, with an efficiently designed dual attention mechanism based skip connection. Typically, a swin transformer based encoder‐decoder network, by incorporating a dual attentional skip connection with Swin‐UNet (DASUNet) is proposed. DASUNet captures the global relationship of image patches based on its window attention mechanism, which fits the real‐time requirement. Moreover, DASUNet characterizes the less salient features by equipping with token dual attention modules among the skip connection, which compensates the ignorance of less salient features incurred from traditional attention mechanism during the stacking of transformer layers. Experiments on ground‐based images (SWINySeg) and remote sensing images (HRC‐WHU, 38‐Cloud) show that, DASUNet achieves the state‐of‐the‐art or competitive results for cloud segmentation (six top‐1 positions of six metrics among 11 methods on SWINySeg, two top‐1 positions of five metrics among 10 methods on HRC‐WHU, two top‐1 positions of four metrics among 12 methods with ParaNum on 38‐Cloud), with 100FPS implementation speed averagely for each image.

Semantic segmentation of point clouds of ancient buildings based on weak supervision

Semantic segmentation of point clouds of ancient buildings plays an important role in Historical Building Information Modelling (HBIM). As the annotation task of point cloud of ancient architecture is characterised by strong professionalism and large workload, which greatly restricts the application of point cloud semantic segmentation technology in the field of ancient architecture, therefore, this paper launches a research on the semantic segmentation method of point cloud of ancient architecture based on weak supervision. Aiming at the problem of small differences between classes of ancient architectural components, this paper introduces a self-attention mechanism, which can effectively distinguish similar components in the neighbourhood. Moreover, this paper explores the insufficiency of positional encoding in baseline and constructs a high-precision point cloud semantic segmentation network model for ancient buildings—Semantic Query Network based on Dual Local Attention (SQN-DLA). Using only 0.1% of the annotations in our homemade dataset and the Architectural Cultural Heritage (ArCH) dataset, the mean Intersection over Union (mIoU) reaches 66.02% and 58.03%, respectively, which is an improvement of 3.51% and 3.91%, respectively, compared to the baseline.

Adherent Peanut Image Segmentation Based on Multi-Modal Fusion.

Aiming at the problem of the difficult segmentation of adherent images due to the not fully convex shape of peanut pods, their complex surface texture, and their diverse structures, a multimodal fusion algorithm is proposed to achieve a 2D segmentation of adherent peanut images with the assistance of 3D point clouds. Firstly, the point cloud of a running peanut is captured line by line using a line structured light imaging system, and its three-dimensional shape is obtained through splicing and combining it with a local surface-fitting algorithm to calculate a normal vector and curvature. Seed points are selected based on the principle of minimum curvature, and neighboring points are searched using the KD-Tree algorithm. The point cloud is filtered and segmented according to the normal angle and the curvature threshold until achieving the completion of the point cloud segmentation of the individual peanut, and then the two-dimensional contour of the individual peanut model is extracted by using the rolling method. The search template is established, multiscale feature matching is implemented on the adherent image to achieve the region localization, and finally, the segmentation region is optimized by an opening operation. The experimental results show that the algorithm improves the segmentation accuracy, and the segmentation accuracy reaches 96.8%.

Segmentation of 3D Point Clouds of Heritage Buildings Using Edge Detection and Supervoxel-Based Topology.

This paper presents a novel segmentation algorithm specially developed for applications in 3D point clouds with high variability and noise, particularly suitable for heritage building 3D data. The method can be categorized within the segmentation procedures based on edge detection. In addition, it uses a graph-based topological structure generated from the supervoxelization of the 3D point clouds, which is used to make the closure of the edge points and to define the different segments. The algorithm provides a valuable tool for generating results that can be used in subsequent classification tasks and broader computer applications dealing with 3D point clouds. One of the characteristics of this segmentation method is that it is unsupervised, which makes it particularly advantageous for heritage applications where labelled data is scarce. It is also easily adaptable to different edge point detection and supervoxelization algorithms. Finally, the results show that the 3D data can be segmented into different architectural elements, which is important for further classification or recognition. Extensive testing on real data from historic buildings demonstrated the effectiveness of the method. The results show superior performance compared to three other segmentation methods, both globally and in the segmentation of planar and curved zones of historic buildings.

A point cloud segmentation framework for image-based spatial transcriptomics

Recent progress in image-based spatial RNA profiling enables to spatially resolve tens to hundreds of distinct RNA species with high spatial resolution. It presents new avenues for comprehending tissue organization. In this context, the ability to assign detected RNA transcripts to individual cells is crucial for downstream analyses, such as in-situ cell type calling. Yet, accurate cell segmentation can be challenging in tissue data, in particular in the absence of a high-quality membrane marker. To address this issue, we introduce ComSeg, a segmentation algorithm that operates directly on single RNA positions and that does not come with implicit or explicit priors on cell shape. ComSeg is applicable in complex tissues with arbitrary cell shapes. Through comprehensive evaluations on simulated and experimental datasets, we show that ComSeg outperforms existing state-of-the-art methods for in-situ single-cell RNA profiling and in-situ cell type calling. ComSeg is available as a documented and open source pip package at https://github.com/fish-quant/ComSeg.