3D Point Cloud Research Articles

Triple-Camera Rectification for Depth Estimation Sensor.

In this study, we propose a novel rectification method for three cameras using a single image for depth estimation. Stereo rectification serves as a fundamental preprocessing step for disparity estimation in stereoscopic cameras. However, off-the-shelf depth cameras often include an additional RGB camera for creating 3D point clouds. Existing rectification methods only align two cameras, necessitating an additional rectification and remapping process to align the third camera. Moreover, these methods require multiple reference checkerboard images for calibration and aim to minimize alignment errors, but often result in rotated images when there is significant misalignment between two cameras. In contrast, the proposed method simultaneously rectifies three cameras in a single shot without unnecessary rotation. To achieve this, we designed a lab environment with checkerboard settings and obtained multiple sample images from the cameras. The optimization function, designed specifically for rectification in stereo matching, enables the simultaneous alignment of all three cameras while ensuring performance comparable to traditional methods. Experimental results with real camera samples demonstrate the benefits of the proposed method and provide a detailed analysis of unnecessary rotations in the rectified images.

MAFNet: a two-stage multiple attention fusion network for partial-to-partial point cloud registration

Abstract 3D point cloud registration is a critical technology in the fields of visual measurement and robot automation processing. In actual large-scale industrial production, the accuracy of point cloud registration directly affects the quality of automated welding processes. However, most existing methods are confronted with serious challenges such as the failure of partial-to-partial point cloud model registration when facing robot automatic processing guidance and error analysis work. Therefore, this paper proposes a novel two-stage network architecture for point cloud registration, which aims at robot pose adjustment and visual guidance in the field of automated welding by using 3D point cloud data. Specifically, we propose a neighborhood-based multi-head attention module in the coarse registration stage. The neighborhood information of each point can be aggregated through focusing on different weight coefficients of multi-head inputs. Then the spatial structure features which is used to establish the overlapping constraint of point clouds are obtained based on above neighborhood information. In the fine registration stage, we propose the similarity matching removal module based on multiple attention fusion features to explore deeper features from different aspects. By using deep fusion features to guide the similarity calculation, the interference of non-overlapping points is removed to achieve the finer registration. Eventually, we compare and analyze the proposed method with the SOTA ones through several error metrics and overlap estimation experiments based on the ModelNet40 dataset. The test results indicate that our method, relative to other mainstream techniques, achieves lower error rates and the most superior accuracy of 98.61% and recall of 98.37%. To demonstrate the generalization performance of proposed algorithm, extensive experiments on the Stanford 3D Scanning Repository, 7-Scenes and our own scanning dataset using partially overlapping point clouds individually under clean and noisy conditions show the validity and reliability of our proposed registration network.

Autonomous Vehicles Traversability Mapping Fusing Semantic–Geometric in Off-Road Navigation

This paper proposes an evaluating and mapping methodology of terrain traversability for off-road navigation of autonomous vehicles in unstructured environments. Terrain features are extracted from RGB images and 3D point clouds to create a traversal cost map. The cost map is then employed to plan safe trajectories. Bayesian generalized kernel inference is employed to assess unknown grid attributes due to the sparse raw point cloud data. A Kalman filter also creates density local elevation maps in real time by fusing multiframe information. Consequently, the terrain semantic mapping procedure considers the uncertainty of semantic segmentation and the impact of sensor noise. A Bayesian filter is used to update the surface semantic information in a probabilistic manner. Ultimately, the elevation map is utilized to extract geometric characteristics, which are then integrated with the probabilistic semantic map. This combined map is then used in conjunction with the extended motion primitive planner to plan the most effective trajectory. The experimental results demonstrate that the autonomous vehicles obtain a success rate enhancement ranging from 4.4% to 13.6% and a decrease in trajectory roughness ranging from 5.1% to 35.8% when compared with the most developed outdoor navigation algorithms. Additionally, the autonomous vehicles maintain a terrain surface selection accuracy of over 85% during the navigation process.

Automated recognition and rebar dimensional assessment of prefabricated bridge components from low-cost 3D laser scanner

Dimension quality inspections for rebar spacing and length of prefabricated bridge components are critical for subsequent efficient assembly on-site. Currently, the traditional manual inspection method is time-consuming, labor-intensive, and error-prone. The existing commercial 3D LiDAR-based methods are difficult in making balances between inspection cost, efficiency, and accuracy. Thus, the automated recognition and segmentation method of 3D point clouds acquired by a self-developed low-cost LiDAR is proposed to evaluate the rebar spacing and length of bridge prefabricated components. Due to the high-cost commercial 3D laser scanner, a spinning 2D LiDAR-based low-cost 3D laser scanner device and geometric calibration model were developed to acquire the 3D spatial point cloud. Then, two-stage automated point cloud segmentation framework is proposed with coarse extraction of the point cloud in the interest region and fine recognition of point cloud using machine learning methods, including plane segmentation and circle fitting employing random sample consensus (RANSAC) algorithm, and rebars segmentation utilizing Gaussian mixture model (GMM) and density-based spatial clustering of applications with noise (DBSCAN) algorithm. The proposed system was evaluated in two prefabricated concrete columns and shows the potential for improving the quality inspection efficiency and accuracy.

Leveraging occupancy map to accelerate video-based point cloud compression

Video-based Point Cloud Compression enables point cloud streaming over the internet by converting dynamic 3D point clouds to 2D geometry and attribute videos, which are then compressed using 2D video codecs like H.266/VVC. However, the complex encoding process of H.266/VVC, such as the quadtree with nested multi-type tree (QTMT) partition, greatly hinders the practical application of V-PCC. To address this issue, we propose a fast CU partition method dedicated to V-PCC to accelerate the coding process. Specifically, we classify coding units (CUs) of projected images into three categories based on the occupancy map of a point cloud: unoccupied, partially occupied, and fully occupied. Subsequently, we employ either statistic-based rules or machine-learning models to manage the partition of each category. For unoccupied CUs, we terminate the partition directly; for partially occupied CUs with explicit directions, we selectively skip certain partition candidates; for the remaining CUs (partially occupied CUs with complex directions and fully occupied CUs), we train an edge-driven LightGBM model to predict the partition probability of each partition candidate automatically. Only partitions with high probabilities are retained for further Rate–Distortion (R–D) decisions. Comprehensive experiments demonstrate the superior performance of our proposed method: under the V-PCC common test conditions, our method reduces encoding time by 52% and 44% in geometry and attribute, respectively, while incurring only 0.68% (0.66%) BD-Rate loss in D1 (D2) measurements and 0.79% (luma) BD-Rate loss in attribute, significantly surpassing state-of-the-art works.

Sports action recognition algorithm based on multi-modal data recognition

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

PCAlign: a general data augmentation framework for point clouds

With the advancement of 3D scanning technologies and deep learning theories, point cloud-based deep learning networks have gained considerable attention in the fields of 3D vision and computer graphics. Leveraging the rich geometric information present in 3D point clouds, these networks facilitate more accurate feature learning tasks. However, existing networks often suffer from generalization defects caused by variations in pose and inconsistent representations of training data. In this paper, we propose a novel data augmentation framework to overcome these limitations. Our approach utilizes principal component analysis (PCA) to generate four aligned copies of a point cloud. These copies are then input into a multi-channel structure, which is compatible with popular backbones of point cloud-based deep networks. Finally, the outputs of the multi-channel structure are merged to generate rotation-invariant feature learning results. Experimental evaluations demonstrate the efficacy of our framework, showcasing significant improvements in various existing point cloud-based deep learning methods. Notably, our method exhibits enhanced robustness in classification tasks, particularly when dealing with point clouds containing random pose variations and non-uniform densities. Project link: https://github.com/LAB123-tech/PCAlign.

R-C-D-F machine learning method to measure for geological structures in 3D point cloud of rock tunnel face

This study introduces an innovative Roughness-CANUPO-Dip-Facet (R-C-D-F) methodology for the measurement of dip angle and direction in geological rock facets. The R-C-D-F method is distinguished by its comprehensive four-step approach, encompassing filtration through roughness analysis, CANUPO analysis, and dip angle filtration, followed by facet segmentation as the measurement step. To achieve precise and efficient results, the method specifically focuses on isolating joint embedment, achieved by systematically filtering out joint bands. This selective filtration process ensures that measurements are conducted exclusively on relevant joint embedment points. The novelty of this methodology lies in its capability to automatically eliminate joint bands while retaining the joint embedment points, facilitating precise measurements without manual intervention. Three site models were evaluated using the R-C-D-F method, alongside four different techniques for measuring dip angle and direction: plane fitting, normal vector conversion, facet segmentation, and compass measurements. The results demonstrated that all methods accurately calculated the dip angle, with an accuracy ranging from 97 % to 99.4 %. The facet segmentation method was selected as the optimal measurement tool due to its automatic nature and capacity to provide accurate results without manual intervention. Furthermore, the optimal local neighbour radius (LNR) for calculating normal vectors was determined, with findings indicating that a larger LNR value enhances accuracy but also increases computational time. A verification was conducted to estimate the dip angle used for filtering and discarding additional points representing joint rock bands, with the optimal value being 45, 30, and 45 degrees for the respective sites.The R-C-D-F method effectively detected and eliminated 100 % of joint band points while retaining 81 % of joint embedment points, and the facet segmentation method provided accurate dip angle and direction measurements for each joint embedment segment. These outcomes underscore the robustness and precision of the R-C-D-F method in geological engineering and rock stability studies.

3D reconstruction and volume measurement of irregular objects based on RGB-D camera

Abstract To address the challenge of measuring volumes of irregular objects, this paper proposes a volume measurement method based on 3D point cloud reconstruction. The point clouds of the object with multiple angles are obtained from an RGB-D camera mounted on a robotic arm, and then are reconstructed to form a whole complete point cloud to calculate the volume of the object. Firstly, the robotic arm is controlled to move to four angles for capturing the original point clouds of the target. Then, by using the rotation and translation matrices obtained from the calibration block pre-registration, the point clouds data from the four angles are fused and reconstructed. Subsequently, the issue of missing bottom point cloud data is addressed using a bottom-filling algorithm. Following this, the efficiency of the point cloud volume calculation algorithm is enhanced through the application of AABB filtering. Finally, the reconstructed point cloud volume is calculated using a slicing algorithm that integrates 2D point cloud segmentation and point cloud sorting. Experimental results show that this method achieves a volume measurement accuracy of over 95% for irregular objects and exhibits good robustness.

Weld seam object detection system based on the fusion of 2D images and 3D point clouds using interpretable neural networks

This study introduces a novel approach that addresses the limitations of existing methods by integrating 2D image processing with 3D point cloud analysis, enhanced by interpretable neural networks. Unlike traditional methods that rely on either 2D or 3D data alone, our approach leverages the complementary strengths of both data types to improve detection accuracy in environments adversely affected by welding spatter and smoke. Our system employs an improved Faster R-CNN model with a ResNet50 backbone for 2D image analysis, coupled with an innovative orthogonal plane intersection line extraction algorithm for 3D point cloud processing. By incorporating explainable components such as visualizable feature maps and a transparent region proposal network, we address the “black box” issue common in deep learning models.This architecture enables a more transparent decision-making process, providing technicians with necessary insights to understand and trust the system’s outputs. The Faster-RCNN structure is designed to break down the object detection process into distinct, understandable steps, from initial feature extraction to final bounding box refinement. This fusion of 2D-3D data analysis and interpretability not only improves detection performance but also sets a new standard for transparency and reliability in automated welding systems, facilitating wider adoption in industrial applications.

Learning 3D human–object interaction graphs from transferable context knowledge for construction monitoring

We propose a novel framework for detecting 3D human–object interactions (HOI) in construction sites and a toolkit for generating construction-related human–object interaction graphs. Computer vision methods have been adopted for construction site safety surveillance in recent years. The current computer vision methods rely on videos and images, with which safety verification is performed on common-sense knowledge, without considering 3D spatial relationships among the detected instances. We propose a new method to incorporate spatial understanding by directly inferring the interactions from 3D point cloud data. The proposed model is trained on a 3D construction site dataset generated from our crafted simulation toolkit. The model achieves 54.11% mean interaction over union (mIOU) and 72.98% average mean precision(mAP) for the worker–object interaction relationship recognition. The model is also validated on PiGraphs, a benchmarking dataset with 3D human–object interaction types, and compared against other existing 3D interaction detection frameworks. It was observed that it achieves superior performance from the state-of-the-art model, increasing the interaction detection mAP by 17.01%. Besides the 3D interaction model, we also simulate interactions from industrial surveillance footage using MoCap and physical constraints, which will be released to foster future studies in the domain.

Algorithm-Driven Extraction of Point Cloud Data Representing Bottom Flanges of Beams in a Complex Steel Frame Structure for Deformation Measurement

Laser scanning has become a popular technology for monitoring structural deformation due to its ability to rapidly obtain 3D point clouds that provide detailed information about structures. In this study, the deformation of a complex steel frame structure is estimated by comparing the associated point clouds captured at two epochs. To measure its deformations, it is essential to extract the bottom flanges of the steel beams in the captured point clouds. However, manual extraction of numerous bottom flanges is laborious and the separation of beam bottom flanges and webs is especially challenging. This study presents an algorithm-driven approach for extracting all beams’ bottom flanges of a complex steel frame. RANdom SAmple Consensus (RANSAC), Euclidean clustering, and an originally defined point feature is sequentially used to extract the beam bottom flanges. The beam bottom flanges extracted by the proposed method are used to estimate the deformation of the steel frame structure before and after the removal of temporary supports to beams. Compared to manual extraction, the proposed method achieved an accuracy of 0.89 in extracting the beam bottom flanges while saving hours of time. The maximum observed deformation of the steel beams is 100 mm at a location where the temporal support was unloaded. The proposed method significantly improves the efficiency of the deformation measurement of steel frame structures using laser scanning.

Development of a Scanning Protocol for Anthropological Remains: A Preliminary Study

Structured-light scanning is a fast and efficient technique for the acquisition of 3D point clouds. However, the extensive and daily application of this class of scanners can be challenging because of the technical know-how necessary to validate the low-cost instrumentation. This challenge is worth accepting because of the large amount of data that can be collected accurately with the aid of specific technical protocols. This work is a preliminary study of the development of an acquisition protocol for anthropological remains performing tests in two opposite and extreme contexts: one characterised by a dark environment and one located in an open area and characterised by a very bright environment. This second context showed the influence of sunlight in the acquisition process, resulting in a colourless point cloud. It is a first step towards the development of a technical protocol for the acquisition of anthropological remains, based on the research of limits and problems associated with an instrument.

High‐order multilayer attention fusion network for 3D object detection

AbstractThree‐dimensional object detection based on the fusion of 2D image data and 3D point clouds has become a research hotspot in the field of 3D scene understanding. However, different sensor data have discrepancies in spatial position, scale, and alignment, which severely impact detection performance. Inappropriate fusion methods can lead to the loss and interference of valuable information. Therefore, we propose the High‐Order Multi‐Level Attention Fusion Network (HMAF‐Net), which takes camera images and voxelized point clouds as inputs for 3D object detection. To enhance the expressive power between different modality features, we introduce a high‐order feature fusion module that performs multi‐level convolution operations on the element‐wise summed features. By incorporating filtering and non‐linear activation, we extract deep semantic information from the fused multi‐modal features. To maximize the effectiveness of the fused salient feature information, we introduce an attention mechanism that dynamically evaluates the importance of pooled features at each level, enabling adaptive weighted fusion of significant and secondary features. To validate the effectiveness of HMAF‐Net, we conduct experiments on the KITTI dataset. In the “Car,” “Pedestrian,” and “Cyclist” categories, HMAF‐Net achieves mAP performances of 81.78%, 60.09%, and 63.91%, respectively, demonstrating more stable performance compared to other multi‐modal methods. Furthermore, we further evaluate the framework's effectiveness and generalization capability through the KITTI benchmark test, and compare its performance with other published detection methods on the 3D detection benchmark and BEV detection benchmark for the “Car” category, showing excellent results. The code and model will be made available on https://github.com/baowenzhang/High‐order‐Multilayer‐Attention‐Fusion‐Network‐for‐3D‐Object‐Detection.

Multi-Scale Classification and Contrastive Regularization: Weakly Supervised Large-Scale 3D Point Cloud Semantic Segmentation

With the proliferation of large-scale 3D point cloud datasets, the high cost of per-point annotation has spurred the development of weakly supervised semantic segmentation methods. Current popular research mainly focuses on single-scale classification, which fails to address the significant feature scale differences between background and objects in large scenes. Therefore, we propose MCCR (Multi-scale Classification and Contrastive Regularization), an end-to-end semantic segmentation framework for large-scale 3D scenes under weak supervision. MCCR first aggregates features and applies random downsampling to the input data. Then, it captures the local features of a random point based on multi-layer features and the input coordinates. These features are then fed into the network to obtain the initial and final prediction results, and MCCR iteratively trains the model using strategies such as contrastive learning. Notably, MCCR combines multi-scale classification with contrastive regularization to fully exploit multi-scale features and weakly labeled information. We investigate both point-level and local contrastive regularization to leverage point cloud augmentor and local semantic information and introduce a Decoupling Layer to guide the loss optimization in different spaces. Results on three popular large-scale datasets, S3DIS, SemanticKITTI and SensatUrban, demonstrate that our model achieves state-of-the-art (SOTA) performance on large-scale outdoor datasets with only 0.1% labeled points for supervision, while maintaining strong performance on indoor datasets.

RD-SLAM: Real-Time Dense SLAM Using Gaussian Splatting

Simultaneous localization and mapping (SLAM) is fundamental for intelligent mobile units to perform diverse tasks. Recent work has shown that integrating neural rendering and SLAM showed promising results in photorealistic environment reconstruction. However, existing methods estimate pose by minimizing the error between rendered and input images, which is time-consuming and cannot be run in real-time, deviating from the original intention of SLAM. In this paper, we propose a dense RGB-D SLAM based on 3D Gaussian splatting (3DGS) while employing generalized iterative closest point (G-ICP) for pose estimation. We actively utilize 3D point cloud information to improve the tracking accuracy and operating speed of the system. At the same time, we propose a dual keyframe selection strategy and its corresponding densification method, which can effectively reconstruct new observation scenes and improve the quality of previously constructed maps. In addition, we introduce regularization loss to address the scale explosion of the 3D Gaussians and over-elongate in the camera viewing direction. Experiments on the Replica, TUM-RGBD, and ScanNet datasets show that our method achieves state-of-the-art tracking accuracy and runtime while being competitive in rendering quality.

Assessment of NavVis VLX and BLK2GO SLAM Scanner Accuracy for Outdoor and Indoor Surveying Tasks

The Simultaneous Localization and Mapping (SLAM) scanner is an easy and portable Light Detection and Ranging (LiDAR) data acquisition device. Its main output is a 3D point cloud covering the scanned scene. Regarding the importance of accuracy in the survey domain, this paper aims to assess the accuracy of two SLAM scanners: the NavVis VLX and the BLK2GO scanner. This assessment is conducted for both outdoor and indoor environments. In this context, two types of reference data were used: the total station (TS) and the static scanner Z+F Imager 5016. To carry out the assessment, four comparisons were tested: cloud-to-cloud, cloud-to-mesh, mesh-to-mesh, and edge detection board assessment. However, the results of the assessments confirmed that the accuracy of indoor SLAM scanner measurements (5 mm) was greater than that of outdoor ones (between 10 mm and 60 mm). Moreover, the comparison of cloud-to-cloud provided the best accuracy regarding direct accuracy measurement without manipulations. Finally, based on the high accuracy, scanning speed, flexibility, and the accuracy differences between tested cases, it was confirmed that SLAM scanners are effective tools for data acquisition.

Dense Feature Pyramid Deep Completion Network

Most current point cloud super-resolution reconstruction requires huge calculations and has low accuracy when facing large outdoor scenes; a Dense Feature Pyramid Network (DenseFPNet) is proposed for the feature-level fusion of images with low-resolution point clouds to generate higher-resolution point clouds, which can be utilized to solve the problem of the super-resolution reconstruction of 3D point clouds by turning it into a 2D depth map complementation problem, which can reduce the time and complexity of obtaining high-resolution point clouds only by LiDAR. The network first utilizes an image-guided feature extraction network based on RGBD-DenseNet as an encoder to extract multi-scale features, followed by an upsampling block as a decoder to gradually recover the size and details of the feature map. Additionally, the network connects the corresponding layers of the encoder and decoder through pyramid connections. Finally, experiments are conducted on the KITTI deep complementation dataset, and the network performs well in various metrics compared to other networks. It improves the RMSE by 17.71%, 16.60%, 7.11%, and 4.68% compared to the CSPD, Spade-RGBsD, Sparse-to-Dense, and GAENET.

MSL-Net: Sharp Feature Detection Network for 3D Point Clouds.

As a significant geometric feature of 3D point clouds, sharp features play an important role in shape analysis, 3D reconstruction, registration, localization, etc. Current sharp feature detection methods are still sensitive to the quality of the input point cloud, and the detection performance is affected by random noisy points and non-uniform densities. In this paper, using the prior knowledge of geometric features, we propose a Multi-scale Laplace Network (MSL-Net), a new deep-learning-based method based on an intrinsic neighbor shape descriptor, to detect sharp features from 3D point clouds. First, we establish a discrete intrinsic neighborhood of the point cloud based on the Laplacian graph, which reduces the error of local implicit surface estimation. Then, we design a new intrinsic shape descriptor based on the intrinsic neighborhood, combined with enhanced normal extraction and cosine-based field estimation function. Finally, we present the backbone of MSL-Net based on the intrinsic shape descriptor. Benefiting from the intrinsic neighborhood and shape descriptor, our MSL-Net has simple architecture and is capable of establishing accurate feature prediction that satisfies the manifold distribution while avoiding complex intrinsic metric calculations. Extensive experimental results demonstrate that with the multi-scale structure, MSL-Net has a strong analytical ability for local perturbations of point clouds. Compared with state-of-the-art methods, our MSL-Net is more robust and accurate.

Enhancing Integrated Sensing and Communication (ISAC) Performance for a Searching–Deciding Alternation Radar-Comm System with Multi-Dimension Point Cloud Data

In developing modern intelligent transportation systems, integrated sensing and communication (ISAC) technology has become an efficient and promising method for vehicle road services. To enhance traffic safety and efficiency through real-time interaction between vehicles and roads, this paper proposes a searching–deciding scheme for an alternation radar-communication (radar-comm) system. Firstly, its communication performance is derived for a given detection probability. Then, we process the echo data from real-world millimeter-wave (mmWave) radar into four-dimensional (4D) point cloud datasets and thus separate different hybrid modes of single-vehicle and vehicle fleets into three types of scenes. Based on these datasets, an efficient labeling method is proposed to assist accurate vehicle target detection. Finally, a novel vehicle detection scheme is proposed to classify various scenes and accurately detect vehicle targets based on deep learning methods. Extensive experiments on collected real-world datasets demonstrate that compared to benchmarks, the proposed scheme obtains substantial radar performance and achieves competitive communication performance.