3D Point Cloud Research Articles

Towards intelligent environments: human sensing through 3D point cloud

To realize the concept of “Society 5.0”, which seeks to enhance well-being by integrating the Internet of Things to connect people, objects, and environments, intelligent environments are essential. These environments focus on creating sensor-enhanced spaces capable of recognizing and interpreting human behaviors and needs. This paper explores how we achieve intelligent environment by highlighting current technologies and applications of human sensing and tracking that enables such intelligent environments. Specifically, it introduces the development and application of LiDAR-based human sensing and tracking technologies, including the Hitonavi and Hitonavi-μ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mu $$\\end{document} systems, designed for multi-person tracking in extensive areas and for enabling person detection and activity recognition on edge devices, respectively. Moreover, the paper explains the services and applications of these methodologies to smart buildings, shopping malls, and elderly care, showcasing their potential to enhance safety, convenience, and understanding of human behavior within intelligent environments. Throughout this paper, we present our vision for creating reliable intelligent environments, which are fundamentally supported by sophisticated human sensing technologies.

One-shot structured light illumination based on shearlet transform

Balancing speed and accuracy has always been a challenge in 3D reconstruction. One-shot structured light illuminations are of perfect performance on real-time scanning, while the related 3D point clouds are typically of relatively poor quality, especially in regions with rapid height changes. To solve this problem, we propose a one-shot reconstruction scheme based on shearlet transform, which combines spatial and frequency domain information to enhance reconstruction accuracy. First, we apply the shearlet transform to the deformed fringe pattern to obtain the transform coefficients. Second, pixel-wise select the indices associated with the N largest coefficients in magnitude to obtain a new filter. Finally, we refocus globally to extract phase using these filters and generate a reliable quality map based on coefficient magnitudes to guide phase unwrapping. Simultaneously, we utilize the maximum coefficient value to generate a quality map for guiding the phase unwrapping process. Experimental results show that the proposed method is robust in discontinuous regions, resulting in more accurate 3D point clouds.

Improving Aerial Targeting Precision: A Study on Point Cloud Semantic Segmentation with Advanced Deep Learning Algorithms

The integration of technological advancements has significantly impacted artificial intelligence (AI), enhancing the reliability of AI model outputs. This progress has led to the widespread utilization of AI across various sectors, including automotive, robotics, healthcare, space exploration, and defense. Today, air defense operations predominantly rely on laser designation. This process is entirely dependent on the capability and experience of human operators. Considering that UAV systems can have flight durations exceeding 24 h, this process is highly prone to errors due to the human factor. Therefore, the aim of this study is to automate the laser designation process using advanced deep learning algorithms on 3D point clouds obtained from different sources, thereby eliminating operator-related errors. As different data sources, dense 3D point clouds produced with photogrammetric methods containing color information, and point clouds produced with LiDAR systems were identified. The photogrammetric point cloud data were generated from images captured by the Akinci UAV’s multi-axis gimbal camera system within the scope of this study. For the point cloud data obtained from the LiDAR system, the DublinCity LiDAR dataset was used for testing purposes. The segmentation of point cloud data utilized the PointNet++ and RandLA-Net algorithms. Distinct differences were observed between the evaluated algorithms. The RandLA-Net algorithm, relying solely on geometric features, achieved an approximate accuracy of 94%, while integrating color features significantly improved its performance, raising its accuracy to nearly 97%. Similarly, the PointNet++ algorithm, relying solely on geometric features, achieved an accuracy of approximately 94%. Notably, the model developed as a unique contribution in this study involved enriching the PointNet++ algorithm by incorporating color attributes, leading to significant improvements with an approximate accuracy of 96%. The obtained results demonstrate a notable improvement in the PointNet++ algorithm with the proposed approach. Furthermore, it was demonstrated that the methodology proposed in this study can be effectively applied directly to data generated from different sources in aerial scanning systems.

Evaluation of Traffic Sign Occlusion Rate Based on a 3D Point Cloud Space

The effectiveness of road signs is hindered by obstructions, such as vegetation, mutual obstruction of signs, or the road alignment itself. The traditional evaluation of road sign recognition effectiveness is conducted through in-vehicle field surveys. However, this method has several drawbacks, including discontinuous identification, unclear positioning, incomplete coverage, and being time-consuming. Consequently, it is unable to effectively assess the recognition status of road signs at any arbitrary point within the road space. Therefore, this study employed laser scanning to construct a point-surface model, which was based on a point cloud algorithm and SLAM (Simultaneous Localization and Mapping), integrated LiDAR and inertial navigation system data, and optimized the point model after processing steps such as denoising, resampling, and three-dimensional model construction. Furthermore, a method for assessing the highway sign occlusion rate based on the picking algorithm was proposed. The algorithm was applied to an actual road environment, and the occlusion by other items was simulated. The results demonstrated the effectiveness of the method. This new method provides support for the fast and accurate calculation of road sign occlusion rates, which is of great importance for ensuring the safe traveling of vehicles.

SuPrNet: Super Proxy for 4D occupancy forecasting

Spacetime (4D) occupancy forecasting explicitly models the future development of self-driving vehicles (SDVs) themselves and their surrounding environment, which is highly beneficial for their downstream tasks. Currently, using point cloud forecasting as a proxy to self-supervisedly optimize the 4D occupancy forecasting framework has significant application prospects due to its independence from labeled data. However, existing methods have the following shortcomings: (1) Insufficient mining of spatiotemporal information in the scene. (2) Neglecting the extraction of regional deep features. (3) Inadequate learning ability for spatiotemporal predictive learning. Based on this, we propose SuPrNet, constructing a ”Super Proxy” to address the aforementioned issues. Specifically, SuPrNet first embeds the spatial and temporal information of the current frame into point-wise features through a 4D hybrid representation combined with multi-planar and grid features (4DPG). Then, a Hierarchical Encoder–Decoder Module (HED) with sparsity consideration is used to structure the point cloud and extract deep features. Thanks to the structured feature map representation, SuPrNet proceeds to use a spatiotemporal predictive learning module (STL) to implicitly predict future feature maps. Finally, we decode these feature maps to a specific size as the forecasting of future occupancy. The 4DPG, HED, and STL in the model are designed to address the aforementioned issues, and experiments on Nuscene and KITTI-Odometry show that our method achieves the best 4D occupancy forecasting and point cloud forecasting accuracy, surpassing previous state-of-the-art models by approximately 30%. Additionally, we have expanded and validated the application prospects of SuPrNet through various downstream tasks. Our project is available at https://github.com/AlanLiangC/4DPCF.

Scan-to-HBIM-to-VR: An Integrated Approach for the Documentation of an Industrial Archaeology Building

In this paper, we propose a comprehensive and optimised workflow for the documentation and the future maintenance and management of a historical building, integrating the state of the art of different techniques, in the challenging context of industrial archaeology. This approach has been applied to the hydraulic work of the “Sostegno del Battiferro” in Bologna, Italy, an example of built industrial heritage whose construction began in 1439 and remains in active use nowadays to control the Navile canal water flow rate. The initial step was the definition of a 3D topographic frame, including geodetic measurements, which served as a reference for the complete 3D survey integrating Terrestrial Laser Scanning (TLS), Structured Light Projection scanning, and the photogrammetric processing of Unmanned Aircraft System (UAS) imagery through a Structure from Motion (SfM) approach. The resulting 3D point cloud has supported as-built parametric modelling (Scan-to-BIM) with the consequent extraction of plans and sections. Finally, the Heritage/Historic Building Information Modelling (HBIM) model generated was rendered and tested for a VR-based immersive experience. Building Information Modelling (BIM) and virtual reality (VR) applications were tested as a support for the management of the building, the maintenance of the hydraulic system, and the training of qualified technicians. In addition, considering the historical value of the surveyed building, the methodology was also applied for dissemination purposes.

Introducing an indoor object classification dataset including sparse point clouds from mmWave radar

This document introduces the RadIOCD, which is a dataset that contains sparse point cloud representations of indoor objects, collected by subjects wearing a commercial off-the-shelf mmWave radar. In particular, RadIOCD includes the recordings of 10 volunteers moving towards 5 different objects (i.e., backpack, chair, desk, human, and wall), placed in 3 different environments. RadIOCD includes sparse 3D point cloud data, together with their doppler velocity and intensity provided by the mmWave radar. A total of 5,776 files are available, with each one having an approximate duration of 8s. The scope of RadIOCD is the availability of data for the recognition of objects solely recorded by the mmWave radar, to be used in applications were the vision-based classification is cumbersome though critical (e.g., in search and rescue operation where there is smoke inside a building). Furthermore, we showcase that this dataset after being segmented into 76,821 samples contains enough data to apply Machine Learning-based techniques, ensuring that they could generalize in different environments and “unseen“ subjects.

Impact of discontinuity data acquisition methods on rockfall susceptibility assessment using high-resolution 3D point cloud

This research evaluates the influence of various discontinuity data acquisition methods on three-dimensional rockfall susceptibility assessments, employing a high-resolution 3D point cloud of the Špičunak rock slope in Croatia. Discontinuity mapping was conducted utilising manual and semi-automated methods. Manual mapping was done in open-source software CloudCompare, while semi-automated mapping was done using Discontinuity Set Extractor and two CloudCompare plug-ins, qFacet Fast Marching and qFacet Kd-Tree algorithm. Significant differences were observed between the methods in terms of the number of identified discontinuity planes, sets, mean orientation values, and weighted densities associated with specific sets. Based on the mapping results, three-dimensional rockfall susceptibility maps were generated. The 3D rockfall susceptibility assessment employed a Kinematic Hazard Index (KHI) for planar, wedge, and flexural toppling failure mechanisms, which determines the percentage of discontinuity planes that meet the geometrical conditions specific to each type of failure. The findings underscore that the mapping results greatly affect the 3D rockfall susceptibility assessment, impacting the identification of potential rockfall source areas. The study emphasizes the importance of integrating manual and semi-automated mapping for reliable 3D rockfall susceptibility assessment. Additionally, this study evaluated the effect of varying the number of discontinuity orientation input data on rockfall susceptibility by comparing nine scenarios, each involving a reduction in the number of discontinuity orientation data. The results indicate that accurate discontinuity mapping is more critical than the sheer number of mapped discontinuities for reliable rockfall susceptibility assessment. Therefore, prioritizing the precise determination of discontinuity set numbers and their associated weighted densities is essential for reliable susceptibility evaluations.

PointSurFace: Discriminative point cloud surface feature extraction for 3D face recognition

Due to the geometric information in the 3D face data, the 3D face recognition methods exhibit better robustness against the physical attacks compared to the 2D recognition methods. In this paper, we propose PointSurFace, a 3D face point cloud recognition method integrated with a specially designed facial surface feature extraction (SurF) module. In PointSurFace, SurF comprehensively combines the coordinate positions, normal vectors, and angle relationships, etc., of each patch in the 3D face point cloud, to explore the explicit local geometric information of 3D faces and generate facial surface features. Subsequently, the surface features are fed into an encoder consisting of four set abstraction (SA) modules and three Inverted Residual MLP (InvResMLP) modules to extract more discriminative 3D facial features, where InvResMLP could suppress the overfitting and reduce information loss in the model training. In addition, PointSurFace utilizes channel fusion operations to integrate position information into the extracted 3D facial features, further enhancing the model performance. Experimental results in 3D facial recognition and verification across multiple datasets demonstrate that PointSurFace achieves the best 3D face recognition performance to date. For example, on Lock3DFace, it achieves the state-of-the-art recognition and verification accuracy of 90.03% and 80.01%, respectively, surpassing the previous methods by 2.02% and 3.19%. The ablation studies evaluate each module of PointSurFace, suggesting that the proposed surface feature extraction module significantly enhances the model performance.

Bilinear Distance Feature Network for Semantic Segmentation in PowerLine Corridor Point Clouds.

Semantic segmentation of target objects in power transmission line corridor point cloud scenes is a crucial step in powerline tree barrier detection. The massive quantity, disordered distribution, and non-uniformity of point clouds in power transmission line corridor scenes pose significant challenges for feature extraction. Previous studies have often overlooked the core utilization of spatial information, limiting the network's ability to understand complex geometric shapes. To overcome this limitation, this paper focuses on enhancing the deep expression of spatial geometric information in segmentation networks and proposes a method called BDF-Net to improve RandLA-Net. For each input 3D point cloud data, BDF-Net first encodes the relative coordinates and relative distance information into spatial geometric feature representations through the Spatial Information Encoding block to capture the local spatial structure of the point cloud data. Subsequently, the Bilinear Pooling block effectively combines the feature information of the point cloud with the spatial geometric representation by leveraging its bilinear interaction capability thus learning more discriminative local feature descriptors. The Global Feature Extraction block captures the global structure information in the point cloud data by using the ratio between the point position and the relative position, so as to enhance the semantic understanding ability of the network. In order to verify the performance of BDF-Net, this paper constructs a dataset, PPCD, for the point cloud scenario of transmission line corridors and conducts detailed experiments on it. The experimental results show that BDF-Net achieves significant performance improvements in various evaluation metrics, specifically achieving an OA of 97.16%, a mIoU of 77.48%, and a mAcc of 87.6%, which are 3.03%, 16.23%, and 18.44% higher than RandLA-Net, respectively. Moreover, comparisons with other state-of-the-art methods also verify the superiority of BDF-Net in point cloud semantic segmentation tasks.

Characteristic-Preserving Latent Space for Unpaired Cross-Domain Translation of 3D Point Clouds.

This article aims at unpaired shape-to-shape transformation for 3D point clouds, for instance, turning a chair to its table counterpart. Recent work for 3D shape transfer or deformation highly relies on paired inputs or specific correspondences. However, it is usually not feasible to assign precise correspondences or prepare paired data from two domains. A few methods start to study unpaired learning, but the characteristics of a source model may not be preserved after transformation. To overcome the difficulty of unpaired learning for transformation, we propose alternately training the autoencoder and translators to construct shape-aware latent space. This latent space based on novel loss functions enables our translators to transform 3D point clouds across domains and maintain the consistency of shape characteristics. We also crafted a test dataset to objectively evaluate the performance of point-cloud translation. The experiments demonstrate that our framework can construct high-quality models and retain more shape characteristics during cross-domain translation compared to the state-of-the-art methods. Moreover, we also present shape editing applications with our proposed latent space, including shape-style mixing and shape-type shifting, which do not require retraining a model.

Entropy‐driven Progressive Compression of 3D Point Clouds

Abstract3D point clouds stand as one of the prevalent representations for 3D data, offering the advantage of closely aligning with sensing technologies and providing an unbiased representation of a measured physical scene. Progressive compression is required for real‐world applications operating on networked infrastructures with restricted or variable bandwidth. We contribute a novel approach that leverages a recursive binary space partition, where the partitioning planes are not necessarily axis‐aligned and optimized via an entropy criterion. The planes are encoded via a novel adaptive quantization method combined with prediction. The input 3D point cloud is encoded as an interlaced stream of partitioning planes and number of points in the cells of the partition. Compared to previous work, the added value is an improved rate‐distortion performance, especially for very low bitrates. The latter are critical for interactive navigation of large 3D point clouds on heterogeneous networked infrastructures.

Target-based georeferencing of terrestrial radar images using TLS point clouds and multi-modal corner reflectors in geomonitoring applications

Terrestrial Radar Interferometry (TRI) is widely adopted in geomonitoring applications due to its capability to precisely observe surface displacements along the line of sight, among other key characteristics. As its deployment grows, TRI is also increasingly used to monitor smaller and more dispersed geological phenomena, where the challenge is their precise localization in 3d space if the pose of the radar interferometer is not known beforehand. To tackle this challenge, we introduce a semi-automatic target-based georeferencing method for precisely aligning TRI data with 3d point clouds obtained using long-range Terrestrial Laser Scanning (TLS). To facilitate this, we developed a multi-modal corner reflector (mmCR) that serves as a common reference point recognizable by both technologies, and we accompanied it with a semi-automatic data-processing pipeline, including the algorithms for precise center estimation. Experimental validation demonstrated that the corner reflector can be localized within the TLS data with a precision of 3–5 cm and within the TRI data with 1–2 dm. The targets were deployed in a realistic geomonitoring scenario to evaluate the implemented workflow and the achievable quality of georeferencing. The post-georeferencing mapping uncertainty was found to be on a decimeter level, matching the state-of-the-art results using dedicated targets and achieving more than an order of magnitude lower uncertainty than the existing data-driven approaches. In contrast to the existing target-based approaches, our results were achieved without laborious visual data inspection and manual target detection and on significantly larger distances, surpassing 2 km. The use of the developed mmCR and its associated data-processing pipeline extends beyond precise georeferencing of TRI imagery to TLS point clouds, allowing for alternatively georeferencing using total stations, mapping quality evaluation as well as on-site testing and calibrating TRI systems within the application environment.

3D-ISRNet:Generating 3D point clouds through image similarity retrieval in a complex background from a single image

Reconstructing the 3D shape of objects using limited information from a single image is a complex and necessary task. The existing methods for reconstructing point clouds from a single image haven't effectively addressed the issue of background interference in the input image, thus greatly limited their practical application. To address the underlying issue, this paper proposes a 3D-ISRNet aiming to enhance the accuracy of point cloud reconstruction through image similarity retrieval. The 3D-ISRNet retrieves point clouds similar to the object and employs spatial and channel attention mechanisms for image encoding. This aims to enhance the features of the reconstruction area and improve reconstruction accuracy. After the fusion of retrieved point clouds with image features, the combined data undergoes a symmetric encoder to generate the reconstructed point clouds, thereby alleviating the constraints imposed by the initial point clouds on the reconstruction results. 3D-ISRNet demonstrates favorable results on public and real-world datasets.

Filtering of 3D point clouds using maximum likelihood algorithm

Recently, the 3D point cloud (PC) has become more popular as an innovative object representation. However, there is usually noise and outliers in the raw point cloud. It is essential to eliminate the noise from the point cloud and outlier data while maintaining the features and finer details intact. This paper presents a comprehensive method for filtering and classification point clouds using a maximum likelihood algorithm (ML). TOPCON GLS-2000 3D terrestrial laser scanners (TLS) have been used to collect the 3D PC data set; the scan range is up to 350 m. About 30 m apart from the study area. ScanMaster software has been used to import, view, and filter point cloud information. The position information of the points is linked with the training point cloud and the filtered point cloud to derive the nonlinear model using MATLAB software. To evaluate the quality of the denoising results, two error metrics have been used: the average angle (δ) and distance (Dmean) between the ground truth point and the resulting point. The experimental findings demonstrate that the suggested approach can effectively filter out background noise while improving feature preservation. The filtering and classifying technique is more effective and efficient compared to the selected filtering methods when applied to 3D point clouds containing a large number of points and a variety of natural characteristics.

Research on Student's T-Distribution Point Cloud Registration Algorithm Based on Local Features.

LiDAR offers a wide range of uses in autonomous driving, remote sensing, urban planning, and other areas. The laser 3D point cloud acquired by LiDAR typically encounters issues during registration, including laser speckle noise, Gaussian noise, data loss, and data disorder. This work suggests a novel Student's t-distribution point cloud registration algorithm based on the local features of point clouds to address these issues. The approach uses Student's t-distribution mixture model (SMM) to generate the probability distribution of point cloud registration, which can accurately describe the data distribution, in order to tackle the problem of the missing laser 3D point cloud data and data disorder. Owing to the disparity in the point cloud registration task, a full-rank covariance matrix is built based on the local features of the point cloud during the objective function design process. The combined penalty of point-to-point and point-to-plane distance is then added to the objective function adaptively. Simultaneously, by analyzing the imaging characteristics of LiDAR, according to the influence of the laser waveform and detector on the LiDAR imaging, the composite weight coefficient is added to improve the pertinence of the algorithm. Based on the public dataset and the laser 3D point cloud dataset acquired in the laboratory, the experimental findings demonstrate that the proposed algorithm has high practicability and dependability and outperforms the five comparison algorithms in terms of accuracy and robustness.

Laser line scanner based real-time geometry monitoring using Encoder-Decoder network during Directed Energy deposition

Directed Energy Deposition (DED) is a metal additive manufacturing technology that is gaining popularity for its ability to rapidly manufacture virtually any metal components no matter how complex the shapes and properties are. However, the current lack of real-time geometry monitoring and control is hindering the wider dissemination of DED in industries. This study developed and validated a geometry monitoring methodology which can achieve real-time inspection of the melt pool and newly solidified layer, and layer-wise inspection of the deposited layer during DED process. An encoder-decoder network was developed and applied to the profile images from the laser line scanner to obtain track profiles. A point cloud generation method was proposed to convert the obtained track profiles into 3D point cloud data using intrinsic/extrinsic calibration and printing position. Experiments have been successfully conducted to validate the proposed methodology by depositing multi-layer X-shape objects.

Fast Robust Point Cloud Registration Based on Compatibility Graph and Accelerated Guided Sampling

Point cloud registration is a crucial technique in photogrammetry, remote sensing, etc. A generalized 3D point cloud registration framework has been developed to estimate the optimal rigid transformation between two point clouds using 3D key point correspondences. However, challenges arise due to the uncertainty in 3D key point detection techniques and the similarity of local surface features. These factors often lead to feature descriptors establishing correspondences containing significant outliers. Current point cloud registration algorithms are typically hindered by these outliers, affecting both their efficiency and accuracy. In this paper, we propose a fast and robust point cloud registration method based on a compatibility graph and accelerated guided sampling. By constructing a compatible graph with correspondences, a minimum subset sampling method combining compatible edge sampling and compatible vertex sampling is proposed to reduce the influence of outliers on the estimation of the registration parameters. Additionally, an accelerated guided sampling strategy based on preference scores is presented, which effectively utilizes model parameters generated during the iterative process to guide the sampling toward inliers, thereby enhancing computational efficiency and the probability of estimating optimal parameters. Experiments are carried out on both synthetic and real-world data. The experimental results demonstrate that our proposed algorithm achieves a significant balance between registration accuracy and efficiency compared to state-of-the-art registration algorithms such as RANSIC and GROR. Even with up to 2000 initial correspondences and an outlier ratio of 99%, our algorithm achieves a minimum rotation error of 0.737° and a minimum translation error of 0.0201 m, completing the registration process within 1 s.

Advancing 3D point cloud understanding through deep transfer learning: A comprehensive survey

The 3D point cloud (3DPC) has significantly evolved and benefited from the advances of deep learning (DL). However, the latter faces various issues, including the lack of data or annotated data, the existence of a significant gap between training data and test data, and the requirement for high computational resources. To that end, deep transfer learning (DTL), which decreases dependency and costs by utilizing knowledge gained from a source data/task in training a target data/task, has been widely investigated. Numerous DTL frameworks have been suggested for aligning point clouds obtained from several scans of the same scene. Additionally, DA, which is a subset of DTL, has been modified to enhance the point cloud data’s quality by dealing with noise and missing points. Ultimately, fine-tuning and DA approaches have demonstrated their effectiveness in addressing the distinct difficulties inherent in point cloud data. This paper presents the first review shedding light on this aspect. it provides a comprehensive overview of the latest techniques for understanding 3DPC using DTL and domain adaptation (DA). Accordingly, DTL’s background is first presented along with the datasets and evaluation metrics. A well-defined taxonomy is introduced, and detailed comparisons are presented, considering different aspects such as different knowledge transfer strategies, and performance. The paper covers various applications, such as 3DPC object detection, semantic labeling, segmentation, classification, registration, downsampling/upsampling, and denoising. Furthermore, the article discusses the advantages and limitations of the presented frameworks, identifies open challenges, and suggests potential research directions.

Intelligent method framework for 3D surface manufacturing in cloud‐edge collaboration architecture

AbstractLarge and complex workpieces are core components in fields, such as aerospace, shipbuilding, and other industrial applications. However, the main challenge of curved plate processing comes from the difficulty in determining the nonlinear rebound features with structural design parameters. An intelligent method framework is proposed for 3D surface manufacturing in cloud‐edge collaboration environment. With the construction of an intelligent generation method for machining parameters, a unified data model is effectively integrated with various discrete data, and an intelligent processing mechanism based on 3D point clouds is constructed. In particular, a prediction method for curved panel rebound is constructed to reduce the manual dependency of the manufacturing process. Finally, a related case study is conducted to verify the framework, and the result shows accuracy, interpretability and reusability advantages over other similar methods.