3D Point Cloud Research Articles

Open-set 3D semantic instance maps for vision language navigation – O3D-SIM

Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work SI Maps (Nanwani L, Agarwal A, Jain K, et al. Instance-level semantic maps for vision language navigation. In: 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE; 2023 Aug.) showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and feature extraction. We propose a representation that results in a 3D point cloud map with instance-level embeddings, which bring in the semantic understanding that natural language commands can query. Quantitatively, the work improves upon the success rate of language-guided tasks. At the same time, we qualitatively observe the ability to identify instances more clearly and leverage the foundational models and language and image-aligned embeddings to identify objects that, otherwise, a closed-set approach wouldn't be able to identify. Project Page - https://smart-wheelchair-rrc.github.io/o3d-sim-webpage

Towards automatic urban tree inventory: Enhancing tree instance segmentation via moving object removal and a chord length-based DBH estimation approach

To enhance urban forestry efficacy in Hong Kong, implementing a paradigm shift towards an automated urban tree inventory that utilizes advanced sensing technologies and artificial intelligence is essential for streamlined data collection and analysis. This study advances this objective by creating a comprehensive framework for estimating diameter at breast height (DBH) and extracting tree images. This framework encompasses five key stages: (1) data acquisition utilizing StructXray, a mobile mapping system equipped with a 360° camera and a multi-beam flash LiDAR sensor; (2) vegetation point clouds extraction using deep learning techniques; (3) individual tree segmentation through machine learning algorithms; (4) DBH estimation; and (5) tree image extraction. Six datasets were collected, yielding tree detection precision, recall and F1 score of 0.88, 0.95 and 0.91 respectively. The presence of moving objects within the 3D point cloud map, exhibiting diverse geometric structures, hinders precise vegetation point cloud segmentation by the pointwise neural network. To tackle this challenge, SalsaNext was employed to rectify the predictions of a pointwise neural network, specifically RandLA-Net in this study, eliminating 91 % of misclassified moving object point clouds and completely removing them from 47 % of affected individual tree point clouds. Additionally, a chord length-based method was proposed to enhance DBH estimation accuracy by dividing the point cloud slice into sectors and summing the chord lengths to estimate the tree trunk perimeter. Compared to the ellipse least squares fitting method, this approach reduced the root-mean-square error of the estimated DBH by 1.31 cm.

A case study of preliminary damage detection of two heritage vaults through geometric deformation analysis on 3D point clouds

Masonry vaults are common structural elements in heritage buildings that, given their long lifespan, are susceptible to the accumulation of damages and deformations from diverse actions such as overloads, support movements, seismic action, etc. The use of 3D point clouds from terrestrial laser scanners and structure from motion photogrammetry survey to investigate these deformations can offer valuable information for structural assessment and conservation, particularly when historical information is limited. The analysis of the deviation measured between the surveyed as-built point cloud of a vault and a reference geometry could reveal deformation patterns. Those, compared with recurring damage mechanisms discussed in literature, can provide a preliminary identification of the damage mechanisms affecting a structure, and the evaluation guide the recognition of the underlying action. The aim of this paper is to investigate how to generate different reference geometries and the impacts of their choice as reference geometry for the identification and quantification of different deformations. Deviation analyses were performed on the main vaults of two different churches with different reference geometries (i.e., geometries derived from the survey and geometrical primitives fitted on point-cloud), and the results were discussed by comparing the deviation maps obtained with the crack pattern and the already known deformations of the buildings.

Aerial Hybrid Adjustment of LiDAR Point Clouds, Frame Images, and Linear Pushbroom Images

In airborne surveying, light detection and ranging (LiDAR) strip adjustment and image bundle adjustment are customarily performed as separate processes. The bundle adjustment is usually conducted from frame images, while using linear pushbroom (LP) images in the bundle adjustment has been historically challenging due to the limited number of observations available to estimate the exterior image orientations. However, data from these three sensors conceptually provide information to estimate the same trajectory corrections, which is favorable for solving the problems of image depth estimation or the planimetric correction of LiDAR point clouds. Thus, our purpose with the presented study is to jointly estimate corrections to the trajectory and interior sensor states in a scalable hybrid adjustment between 3D LiDAR point clouds, 2D frame images, and 1D LP images. Trajectory preprocessing is performed before the low-frequency corrections are estimated for certain time steps in the following adjustment using cubic spline interpolation. Furthermore, the voxelization of the LiDAR data is used to robustly and efficiently form LiDAR observations and hybrid observations between the image tie-points and the LiDAR point cloud to be used in the adjustment. The method is successfully demonstrated with an experiment, showing the joint adjustment of data from the three different sensors using the same trajectory correction model with spline interpolation of the trajectory corrections. The results show that the choice of the trajectory segmentation time step is not critical. Furthermore, photogrammetric sub-pixel planimetric accuracy is achieved, and height accuracy on the order of mm is achieved for the LiDAR point cloud. This is the first time these three types of sensors with fundamentally different acquisition techniques have been integrated. The suggested methodology presents a joint adjustment of all sensor observations and lays the foundation for including additional sensors for kinematic mapping in the future.

Algorithm for 3D point cloud steganalysis based on composite operator feature enhancement

Algorithm for 3D point cloud steganalysis based on composite operator feature enhancement

Digital Dental Biometrics for Human Identification Based on Automated 3D Point Cloud Feature Extraction and Registration.

Intraoral scans (IOS) provide precise 3D data of dental crowns and gingival structures. This paper explores an application of IOS in human identification. We propose a dental biometrics framework for human identification using 3D dental point clouds based on machine learning-related algorithms, encompassing three stages: data preprocessing, feature extraction, and registration-based identification. In the data preprocessing stage, we use the curvature principle to extract distinguishable tooth crown contours from the original point clouds as the holistic feature identification samples. Based on these samples, we construct four types of local feature identification samples to evaluate identification performance with severe teeth loss. In the feature extraction stage, we conduct voxel downsampling, then extract the geometric and structural features of the point cloud. In the registration-based identification stage, we construct a coarse-to-fine registration scheme in order to realize the identification task. Experimental results on a dataset of 160 individuals demonstrate that our method achieves a Rank-1 recognition rate of 100% using complete tooth crown contours samples. Utilizing the remaining four types of local feature samples yields a Rank-1 recognition rate exceeding 96.05%. The proposed framework proves effective for human identification, maintaining high identification performance even in extreme cases of partial tooth loss.

A domain adaptation framework for cross-modality SAR 3D reconstruction point clouds segmentation utilizing LiDAR data

Synthetic aperture radar (SAR) 3D point clouds reconstruction can eliminate the problems of layover in 2D SAR image projections, the recognition of the reconstructed point cloud can significantly enhance target identification and information extraction. Current SAR 3D point cloud segmentation methods, based on traditional machine learning clustering approaches, suffer from poor automation and accuracy. However, the absence of publicly labeled SAR datasets impedes the progress of deep learning-based SAR 3D point cloud segmentation methods. To tackle the aforementioned challenges, we introduce, for the first time, an alternative training approach for SAR 3D point cloud segmentation using LiDAR annotated data, which offers a more abundant sample pool, thus alleviating the lack of training sets. Nevertheless, a segmentation model trained on LiDAR point clouds exhibits a significant decline in performance when directly applied to SAR 3D reconstructed point clouds due to the cross-domain discrepancy. This research presents a pioneering domain adaptation 3D semantic segmentation framework to implement cross-modal learning for SAR point clouds. In our scheme, a simple yet effective technique is developed to achieve segmentation of SAR double-bounce scattering regions called SARDBS-Mix, which employs a mixing strategy, to capture the distinctive reflection characteristics of SAR data. Furthermore, we implement approaches including center alignment and normalization, local augmentation, and weighted cross entropy to mitigate LiDAR and SAR domain gap and class imbalances. The experimental results validate the feasibility and effectiveness of the proposed method for SAR 3D reconstructed point cloud segmentation.

3DFFL: privacy-preserving Federated Few-Shot Learning for 3D point clouds in autonomous vehicles

This paper presents a comprehensive study of 3D point cloud Federated Few-Shot Learning (3DFFL), focusing on addressing challenges such as limited data availability and privacy concerns in point cloud classification for applications such as autonomous vehicles. We introduce a novel approach that integrates Federated Learning with Few-Shot Learning techniques, with a special emphasis on optimizing network architectures for 3D point cloud data. Our method capitalizes on the strengths of PointNet++ for feature extraction and ProtoNet for classification, all within a federated learning framework to ensure data privacy and collaborative learning. Significantly, the approach is augmented with the use of attention and SoftMax layers, enhancing the feature extraction and classification processes. Extensive experiments on the ModelNet40, ShapeNet, and ScanOnjectNN datasets validate our method’s accuracy and adaptability in handling 3D point cloud classification, especially in privacy-sensitive and collaborative scenarios. This study not only demonstrates the potential of integrating attention mechanisms and SoftMax layers in 3DFFL but also lays a robust foundation for future advancements in this evolving field, particularly in technologies dependent on 3D data processing.

Automatic lockbolt detection method for riveting quality evaluation based on specific geometric feature descriptors of 3D point cloud

Abstract Lockbolt is increasingly utilized in different fields due to its good performance in riveting and locking. However, the digital measurement and evaluation methods for the riveting quality of lockbolts still need to be explored. In this paper, we proposed a point cloud-based automatic lockbolt detecti on method, estimating the riveting quality via the protruding height of the lockbolt pin. The method involves three main stages: density computation for 3D point cloud, identification of real lockbolt regions, and segmentation for the top surface of the lockbolt. Firstly, a noise-weakened density computation model is proposed to suppress the density value of noise points. Secondly, the idea of PCA (principal component analysis) is adopted, and a set of SGFDs (specific geometric feature descriptors) is built to identify the real lockbolts from the candidate regions. Lockbolts can be robustly represented even if noise and missing in the point cloud. Thirdly, an efficient section-by-section segmenting algorithm is proposed to segment the top surface of the pin protrusion of the riveted lockbolt. Finally, the riveting quality of each lockbolt is evaluated based on the height from the top surface of the pin protrusion to the bottom plane of the lockbolt. Experiments on real data verified the accuracy, robustness, and efficiency of the proposed method.

3D young female waist-leg modelling based on a hybrid 3d-scan and 2d-image approach

To enhance the pant fit for individuals, this study proposed a method for constructing a 3D waist-leg mannequin based on body images and 3D point-clouds of young women. A total of 288 females aged 18–25 were measured using 3D body scanner and 2D image-shooting method to obtain 3D point-cloud data and body images. The 3D point-cloud data were analyzed to extract 33 sectional curves, including the cross-sectional curves and crotch curve, to identify key points on each curve. For the body images, key parameters related to curve position and shape, such as height, width, and thickness, were automatically extracted for modelling parameters. Curve generation rules were established based on correlation and regression analysis of the key points. The Individualized 3D mannequin was simulated by adjusting the curve centers at each characteristic position to align with the body images, and was validated by comparing the body sizes of the 3D mannequin with the actual measurements. The results indicated that the final simulated mannequin accurately represents the basic characteristics of the waist-leg shape, with significance values above 0.05, which showed the feasibility of the modelling method. Furthermore, 73.9% of the samples had an absolute error of less than 1 cm between the 3D mannequin and the actual measurements. This study can facilitate 3D body modelling from body images, and provide a reference to develop individualized apparel patterns for clothing customization.

A novel method for the classification of 3D point clouds based on the improved PointNet++

Abstract In deep learning, point clouds are used as the primary input format for 3D data, which can provide detailed geometric information about objects in the original 3D space. PointNet++ is a deep learning network that uses point cloud data as an input format, which avoids the losses associated with the previous conversion of point cloud into 3D voxelization and a collection of 2D images. Although PointNet++ can directly process point cloud data in various ways, due to the disordered, irregular, and unevenly distributed nature of point cloud data, the effect of extracting point cloud features could be better. The large amount of point cloud data also leads to the training model falling into the local optimal solution, which affects the training results. In recent years, some effective methods and strategies have emerged to address these problems. In this thesis, three methods are proposed based on the PointNet++ network: feature similarity-based attention pooling, small kernel convolution, and diverse branch block method to improve the performance of the PointNet++ network. Experiments show that the improvement methods proposed in this paper effectively improve the feature extraction accuracy, which improves the accuracy of the PointNet++ network for classification on the ModelNet40_Normal_Resampled dataset, with an overall improvement of 1% compared with PointNet++.

A Study on the 3D Reconstruction Strategy of a Sheep Body Based on a Kinect v2 Depth Camera Array.

Non-contact measurement based on the 3D reconstruction of sheep bodies can alleviate the stress response in sheep during manual measurement of body dimensions. However, data collection is easily affected by environmental factors and noise, which is not conducive to practical production needs. To address this issue, this study proposes a non-contact data acquisition system and a 3D point cloud reconstruction method for sheep bodies. The collected sheep body data can provide reference data for sheep breeding and fattening. The acquisition system consists of a Kinect v2 depth camera group, a sheep passage, and a restraining pen, synchronously collecting data from three perspectives. The 3D point cloud reconstruction method for sheep bodies is implemented based on C++ language and the Point Cloud Library (PCL). It processes noise through pass-through filtering, statistical filtering, and random sample consensus (RANSAC). A conditional voxel filtering box is proposed to downsample and simplify the point cloud data. Combined with the RANSAC and Iterative Closest Point (ICP) algorithms, coarse and fine registration are performed to improve registration accuracy and robustness, achieving 3D reconstruction of sheep bodies. In the base, 135 sets of point cloud data were collected from 20 sheep. After 3D reconstruction, the reconstruction error of body length compared to the actual values was 0.79%, indicating that this method can provide reliable reference data for 3D point cloud reconstruction research of sheep bodies.

Digital reconstruction of railway steep slope from UAV+TLS using geometric transformer

Accurate representation of railway slopes, especially those that are steep, is vital for real-time risk perception. Also, temporary structures also present certain safety hazards due to lack of monitoring. Traditional point cloud modeling, employing Unmanned Aerial Vehicle (UAV) or Terrestrial Laser Scanning (TLS), often struggles to simultaneously account for the precision of both surface and overhead models, leading to considerable model distortion, roughness, and deviation. Addressing these issues, A new 3D point cloud modeling algorithm for railway slopes based on a geometric transformer is presented in this paper. This involves an innovative rough point cloud denoising technique leveraging adaptive segmentation, multi-scale denoising, and deep learning point cloud registration. Our approach significantly enhances UAV point cloud accuracy and supplements missing portions of the TLS point cloud dataset occluded by objects block, using data from the UAV point cloud set. An experimental study shows that the score-based denoising algorithm improves precision from 37.44 mm to 8.11 mm for a UAV 3D point cloud. Further, by registering the UAV and TLS point cloud sets using the Geometric Transformer algorithm, the precision of the 3D point cloud was further augmented to 5.11 mm, representing a sevenfold enhancement over the initial UAV point cloud accuracy prior to denoising. Consequently, a high-fidelity 3D point cloud model of steep railway slopes has been created.

Graph Transformer for 3D point clouds classification and semantic segmentation

Recently, graph-based and Transformer-based deep learning have demonstrated excellent performances on various point cloud tasks. Most of the existing graph-based methods rely on static graph, which take a fixed input to establish graph relations. Moreover, many graph-based methods apply maximizing and averaging to aggregate neighboring features, so that only a single neighboring point affects the feature of centroid or different neighboring points own the same influence on the centroid’s feature, which ignoring the correlation and difference between points. Most Transformer-based approaches extract point cloud features based on global attention and lack the feature learning on local neighbors. To solve the above issues of graph-based and Transformer-based models, we propose a new feature extraction block named Graph Transformer and construct a 3D point cloud learning network called GTNet to learn features of point clouds on local and global patterns. Graph Transformer integrates the advantages of graph-based and Transformer-based methods, and consists of Local Transformer that use intra-domain cross-attention and Global Transformer that use global self-attention. Finally, we use GTNet for shape classification, part segmentation and semantic segmentation tasks in this paper. The experimental results show that our model achieves good learning and prediction ability on most tasks. The source code and pre-trained model of GTNet will be released on https://github.com/NWUzhouwei/GTNet.

Fast supervoxel segmentation of connectivity median simulation based on Manhattan distance

Supervoxels provide a more natural and compact 3D point cloud representation, which is a fundamental problem in point cloud processing and has attracted extensive attention from many scholars. At present, although the supervoxel segmentation has achieved significant results, the processing efficiency still restricts its further processing in the face of large-scale point cloud processing. Therefore, this paper proposes a new supervoxel segmentation method for point clouds to achieve fast supervoxel over-segmentation of large-scale point clouds while maintaining good boundaries and accuracy. Firstly, the supervoxel segmentation is formulated as a subset selection problem. With the idea of greedy strategy, the interval quick sorting and region growing methods is designed to realize the rapid merging of subsets in the iterative process. Secondly, the Manhattan distance metric and the seed point median simulation method are proposed to enhance the precision of iterative optimization and improve the efficiency of point cloud supervoxel segmentation. Experimental results show that, compared with state-of-the-art supervoxel segmentation method, the proposed method achieves the best results under UE, GCE, BR and other indicators, while reducing the running time of the algorithm by 10.6% to 28.3%.

A novel 3D reconstruction method of blast furnace burden surface based on virtual camera array

Accurate three-dimensional (3D) blast furnace burden surface topography is crucial for optimizing material distribution and furnace operation. However, harsh conditions within the furnace pose challenges to this 3D reconstruction, such as difficulties in matching feature points and sparse 3D point clouds. To overcome these challenges, we propose a 3D reconstruction method for the blast furnace burden surface using virtual camera array. Firstly, an ORB feature extraction and matching method based on bidirectional optical flow tracking is proposed to solve the impact of illumination changes and noise. Then, the virtual camera array is constructed to generate a sparse 3D point cloud from an image sequence. Finally, dense surface reconstruction of the burden surface is achieved based on the improved PMVS and the Poisson surface reconstruction method. Experimental results demonstrate that the proposed method can extract accurate feature points and reconstruct the accurate 3D topography of the burden surface.

Evaluating the utility of combining high resolution thermal, multispectral and 3D imagery from unmanned aerial vehicles to monitor water stress in vineyards

PurposeHigh resolution imagery from unmanned aerial vehicles (UAVs) has been established as an important source of information to perform precise irrigation practices, notably relevant for high value crops often present in semi-arid regions such as vineyards. Many studies have shown the utility of thermal infrared (TIR) sensors to estimate canopy temperature to inform on vine physiological status, while visible-near infrared (VNIR) imagery and 3D point clouds derived from red–green–blue (RGB) photogrammetry have also shown great promise to better monitor within-field canopy traits to support agronomic practices. Indeed, grapevines react to water stress through a series of physiological and growth responses, which may occur at different spatio-temporal scales. As such, this study aimed to evaluate the application of TIR, VNIR and RGB sensors onboard UAVs to track vine water stress over various phenological periods in an experimental vineyard imposed with three different irrigation regimes.MethodsA total of twelve UAV overpasses were performed in 2022 and 2023 where in situ physiological proxies, such as stomatal conductance (gs), leaf (Ψleaf) and stem (Ψstem) water potential, and canopy traits, such as LAI, were collected during each UAV overpass. Linear and non-linear models were trained and evaluated against in-situ measurements.ResultsResults revealed the importance of TIR variables to estimate physiological proxies (gs, Ψleaf, Ψstem) while VNIR and 3D variables were critical to estimate LAI. Both VNIR and 3D variables were largely uncorrelated to water stress proxies and demonstrated less importance in the trained empirical models. However, models using all three variable types (TIR, VNIR, 3D) were consistently the most effective to track water stress, highlighting the advantage of combining vine characteristics related to physiology, structure and growth to monitor vegetation water status throughout the vine growth period.ConclusionThis study highlights the utility of combining such UAV-based variables to establish empirical models that correlated well with field-level water stress proxies, demonstrating large potential to support agronomic practices or even to be ingested in physically-based models to estimate vine water demand and transpiration.

3D surface change detection of slope using 3D point cloud based on construction of double surface

Aiming at the complex geological conditions of slopes and the difficulty of manual investigation, this paper proposes a method of constructing double surface based on 3D point cloud data to estimate the depth of slope deformation and disaster analysis. Terrestrial Laser Scanner is utilized to collect the slope point cloud data. Through the construction of double surface to simulate the original slope, the deformation depth is estimated and the depth data are fused into the point cloud to realize the automatic identification of slope deformation disaster and the calculation of deformation depth, and provide the deformation depth map. The shortcomings of the traditional Multiscale Model and Model Point Cloud Comparison (M3C2) algorithm, which needs to capture multiple time periods of point cloud data for calculation can be overcome. And there is no need to recognize the segmentation of the slope point cloud. The study results show that the maximum relative error of slope deformation is 10.82% in slope deformation. The proposed method is suitable for detecting slope hazards in various scenarios, providing a basis for further analysis and the development of effective erosion control measures. It introduces a new technology for daily monitoring of slope deformations and can be extended to drone-based LiDAR applications.

Calibration and validation of a multi-beam LiDAR onboard China Chang’e lunar probe for landing obstacle avoidance

For automatic exploration of specific areas with high scientific value on the moon and other planets, high-precision obstacle detection and hazard avoidance capabilities are required. Three-dimensional (3D) LiDAR can obtain a high-resolution 3D point cloud, offering significant advantages in landing obstacle avoidance. During the landing of the Chang’e lander, a laser 3D imaging sensor (L3DIS) is used to measure and generate accurate topographic maps of the candidate landing area in real-time to help find a safe landing zone. The L3DIS employs 16 beamlets split from a Gaussian laser beam and 16 channels of linear-array avalanche photodiodes within the same optical path and scans the object with a two-axis galvanometer in a field of view of 29°×33°. The calibration of the systematic errors and the performance of obstacle detection on this sensor are conducted in this paper. First, the instrument components and main error sources of the sensor are introduced, and the systematic errors are calibrated based on planar targets, indicating that the sensor’s accuracy is greater than 4 cm, and the corresponding obstacle detection accuracy is better than 12 cm. Second, the ground validation of obstacle detection is demonstrated, and the obtained point cloud is compared with that obtained by a terrestrial laser scanner, indicating that the sensor has good performance. Finally, the performance of onboard measurement and obstacle detection is analyzed, showing that the sensor can identify obvious craters and rocks, i.e., nine major craters with a maximum and minimum depth of 1.36 m and 0.16 m, respectively, and major rocks with a maximum and minimum height of 0.1 m and 0.05 m, respectively, and main landing area with slop less than 10° except for the edges of craters and rocks.

High-Precision Positioning and Rotation Angle Estimation for a Target Pallet Based on BeiDou Navigation Satellite System and Vision.

In outdoor unmanned forklift unloading scenarios, pallet detection and localization face challenges posed by uncontrollable lighting conditions. Furthermore, the stacking and close arrangement of pallets also increase the difficulty of positioning a target pallet. To solve these problems, a method for high-precision positioning and rotation angle estimation for a target pallet using the BeiDou Navigation Satellite System (BDS) and vision is proposed. Deep dual-resolution networks (DDRNets) are used to segment the pallet from depth images and RGB images. Then, keypoints for calculating the position and rotation angle are extracted and further combined with the 3D point cloud data to achieve accurate pallet positioning. Constraining the pixel coordinates and depth coordinates of the center point of the pallet and setting the priority of the pallet according to the unloading direction allow the target pallet to be identified from multiple pallets. The positioning of the target pallet in the forklift navigation coordinate system is achieved by integrating BDS positioning data through coordinate transformation. This method is robust in response to lighting influences and can accurately locate the target pallet. The experimental results show that the pallet positioning error is less than 20 mm, and the rotation angle error is less than 0.37°, which meets the accuracy requirements for automated forklift operations.