Deep Learning-based Point Research Articles

LPL-VIO: monocular visual-inertial odometry with deep learning-based point and line features

LPL-VIO: monocular visual-inertial odometry with deep learning-based point and line features

Noise4Denoise: Leveraging noise for unsupervised point cloud denoising

Existing deep learning-based point cloud denoising methods are generally trained in a supervised manner that requires clean data as ground-truth labels. However, in practice, it is not always feasible to obtain clean point clouds. In this paper, we introduce a novel unsupervised point cloud denoising method that eliminates the need to use clean point clouds as groundtruth labels during training. We demonstrate that it is feasible for neural networks to only take noisy point clouds as input, and learn to approximate and restore their clean versions. In particular, we generate two noise levels for the original point clouds, requiring the second noise level to be twice the amount of the first noise level. With this, we can deduce the relationship between the displacement information that recovers the clean surfaces across the two levels of noise, and thus learn the displacement of each noisy point in order to recover the corresponding clean point. Comprehensive experiments demonstrate that our method achieves outstanding denoising results across various datasets with synthetic and real-world noise, obtaining better performance than previous unsupervised methods and competitive performance to current supervised methods.

Automated BIM-to-scan point cloud semantic segmentation using a domain adaptation network with hybrid attention and whitening (DawNet)

Deep learning-based point cloud semantic segmentation facilitates scene understanding and BIM modelling, but its success requires vast amount of labelled point clouds, which is laborious and time-consuming. To reduce point cloud annotation cost, researchers attempt to leverage synthetic point clouds, but the domain gap between synthetic and real point clouds deteriorates the segmentation accuracy. To address this issue, this study develops a BIM-to-Scan point cloud semantic segmentation approach to mitigate the domain gap between BIM and real point clouds, improving the segmentation performance on real point clouds. To this end, this study starts by proposing a BIM-based point cloud generation method, which uses FME and BIM models to automatically generate and label synthetic point clouds, decreasing the annotation cost. To fill the domain gap, a DawNet is invented by integrating a domain adaptation network with ZCA whitening operation and hybrid attention mechanism. Specifically, ResPointNet++ is used to extract geometric features and execute the segmentation task, which is then combined with a domain discriminator to perform domain adversarial learning, aligning the data distribution of BIM and real point clouds. To improve the performance of the DawNet, a residual learning block with whitening and a hybrid attention module are designed. These two modules help extract and exploit domain-invariant features to boost the generalisation and segmentation performance of the DawNet. Finally comprehensive experiments show that the proposed BIM-based method spends 0.5 person-hours to generate 0.45 billion labelled BIM point clouds, and that the developed DawNet achieves 17% and 11% more mIoU than ResPointNet++ and DANN. The ablation study also confirms the effectiveness of the hybrid attention module and ZCA whitening operation.

Plant-Denoising-Net (PDN): A plant point cloud denoising network based on density gradient field learning

Effective point cloud denoising is critical in 3D plant phenotyping applications, which reduces interference in subsequent algorithms and improves the accuracy of plant phenotypes measurement. Deep learning-based point cloud denoising algorithms have shown excellent denoising performance on simple CAD models. However, these algorithms suffer from issues including over-smoothing or shrinkage and low efficiency when applied on density uneven, incomplete, various types of noise and complex plant point clouds. We proposed a plant point cloud denoising network (PDN) based on point cloud density gradient field learning, which can effectively address the challenges posed by plant point clouds. PDN consists of three main modules: point density feature (PDF) exception module, umbrella operator feature (UOF) computation module, and point density gradient (DG) estimation module. The performance of PDN was evaluated in experiments using point clouds of multiple plant species with noise of different types. Under different levels of Gaussian noise, our method achieved a relative performance improvement of 7.6%-19.3% compared to the state-of-the-art baseline methods, reaching state-of-the-art denoising performance. For noise of different types, the majority of our denoising results outperformed the baseline methods. In addition, our method was 0.5 and 8.6 times faster than the baseline methods when processing point clouds with low and high noise level, respectively. The good robustness, generalization, and computational efficacy of PDN are expected to facilitate the acquisition of high-precision 3D point clouds for various plant species, enhance the versatility of 3D phenotyping methods, improve the accuracy of the measurement of structural phenotypes, and increase the throughput of data processing, therefore facilitate the development of modern breeding research. The source code and the datasets used in this study is available on GitHub at https://github.com/suetme/PDN-plant-denoising-net.

Denoising point clouds with fewer learnable parameters

Point cloud denoising is a crucial task in the field of geometric processing. Recent years have witnessed significant advancements in deep learning-based point cloud denoising algorithms. These methods, compared to traditional techniques, demonstrate enhanced robustness against noise and produce point cloud data of higher fidelity. Despite their impressive performance, achieving a balance between denoising efficacy and computational efficiency remains a formidable challenge in learning-based methods. To solve this problem, we introduce LPCDNet, a novel lightweight point cloud denoising network. LPCDNet consists of three main components: a lightweight feature extraction module utilizing trigonometric functions for relative position encoding; a non-parametric feature aggregation module to leverage semantic similarities for global context comprehension; and a decoder module designed to realign noise points with the underlying surface. The network is designed to capture both local details and non-local structures, thereby ensuring high-quality denoising outcomes with a minimal parameter count. Extensive experimental evaluations demonstrate that LPCDNet achieves comparable or superior performance to state-of-the-art methods, while significantly reducing the number of learnable parameters and the necessary running time.

Research on deep learning-based point cloud semantic segmentation for offshore drilling platforms

Offshore drilling platforms (ODP) require effective deformation monitoring and damage detection in the wellhead space for operational maintenance. LiDAR technology, replacing traditional methods, plays a key role in damage detection through 3D reconstruction, and the accurate semantic segmentation of point clouds is fundamental to this process. Following advancements in deep learning, neural network models have emerged as effective tools for the semantic segmentation of point clouds in a range of settings. ODP point cloud data, characterized by large volumes, high density, class imbalance, and complex structures, present unique challenges for feature extraction. To address these challenges, this paper proposed a multi-scale bounding box (MSB-Box) feature extraction strategy based on spatial relation dynamic encoding (SRD-encoding) and a multi-scale point cloud down-sampling (MPDS) strategy to enhance computational efficiency. Utilizing these proposed strategies, we developed an ODP-Net, a deep learning neural network based on the attention mechanism. The ODP-Net's semantic segmentation performance was evaluated using a self-constructed ODP scene point cloud dataset and the S3DIS dataset. The results demonstrate that ODP-Net, based on MPDS and MSB-Box, outperforms other networks in diverse scenes, confirming its high accuracy and wide applicability in the semantic segmentation of point cloud data in ODP scenes.

Integrating Contextual Information and Attention Mechanisms with Sparse Convolution for the Extraction of Internal Objects within Buildings from Three-Dimensional Point Clouds

Deep learning-based point cloud semantic segmentation has gained popularity over time, with sparse convolution being the most prominent example. Although sparse convolution is more efficient than regular convolution, it comes with the drawback of sacrificing global context information. To solve this problem, this paper proposes the OcspareNet network, which uses sparse convolution as the backbone and captures global contextual information using the offset attention module and context aggregation module. The offset attention module improves the network’s capacity to obtain global contextual information about the point cloud. The context aggregation module utilizes contextual information in the training and testing phases, which increases the network’s capacity to discern the overall structure and successfully improves the network’s capacity and the accuracy of the difficult-scene segmentation category. Compared to the state-of-the-art (SOTA) models, our model has a smaller parameter count and achieves higher accuracy on challenging segmentation categories such as ‘pictures’, ‘counters’, and ‘desks’ in the ScanNetV2 dataset, with IoU scores of 41.1%, 70.3%, and 72.5%, respectively. Furthermore, ablation experiments confirmed the efficacy of our designed modules.

Deep learning-based low overlap point cloud registration for complex scenario: The review

Most studies on point cloud registration have established the problem in the case of ideal point cloud data. Although the state-of-the-art approaches have achieved amazing results on multiple public datasets, the issue of low overlap point cloud data invalidating state-of-the-art methods is acting as a latent challenge that has not been solved. Therefore, a profound analysis about why existing registration architectures break down in the low-overlap regime and how to select the appropriate strategies to improve the low overlap point cloud correspondence estimation is necessary and useful. Unfortunately, there are few survey works about low overlap cloud registration solving strategies and the corresponding datasets are very limited. This work briefly reviews mainstream deep learning-based point cloud registration and provides an in-depth analysis of the reasons why these architectures are not generalizable to scenarios with low overlapping areas. It is the first survey that mainly focuses on representative low overlap registration methods, their techniques, and related datasets for training/testing. It is worth noting that we also design and construct a large 3D dataset to eliminate the gap in Semantic-assisted point cloud registration with low overlap. Finally, challenges about low overlap point cloud registration and future directions in addressing these challenges are also pointed out. [dataset]

A review of rigid point cloud registration based on deep learning.

With the development of 3D scanning devices, point cloud registration is gradually being applied in various fields. Traditional point cloud registration methods face challenges in noise, low overlap, uneven density, and large data scale, which limits the further application of point cloud registration in actual scenes. With the above deficiency, point cloud registration methods based on deep learning technology gradually emerged. This review summarizes the point cloud registration technology based on deep learning. Firstly, point cloud registration based on deep learning can be categorized into two types: complete overlap point cloud registration and partially overlapping point cloud registration. And the characteristics of the two kinds of methods are classified and summarized in detail. The characteristics of the partially overlapping point cloud registration method are introduced and compared with the completely overlapping method to provide further research insight. Secondly, the review delves into network performance improvement summarizes how to accelerate the point cloud registration method of deep learning from the hardware and software. Then, this review discusses point cloud registration applications in various domains. Finally, this review summarizes and outlooks the current challenges and future research directions of deep learning-based point cloud registration.

Deep Learning-Based Point Cloud Coding and Super-Resolution: a Joint Geometry and Color Approach

Deep Learning-Based Point Cloud Coding and Super-Resolution: a Joint Geometry and Color Approach

Deep learning-based point cloud upsampling: a review of recent trends

Deep learning-based point cloud upsampling: a review of recent trends

점군 위치 정보 인코딩 활용을 통한 딥러닝 기반 포인트 클라우드 샘플링 성능 개선 연구

점군 위치 정보 인코딩 활용을 통한 딥러닝 기반 포인트 클라우드 샘플링 성능 개선 연구

MSaD-Net: A Mix Self-Attention Networks for 3D Point Cloud Denoising

In the process of acquiring 3D point cloud data, due to environmental interference or unstable scanning equipment, the acquired data often have noisy points. Recently, with the development of neural networks for point clouds, great progress has been made in deep learning-based point cloud denoising. However, most of the existing methods adopt a pointnet-like structure to predict point offsets. Simple pooling operation loses much important information, such as local neighborhood information, and global information. The loss of information makes the algorithm ineffective when dealing with some complex cases. In order to solve the above problems, we propose a self-attention-based point cloud denoising network architecture, through the Transformer structure, to establish long-range dependencies of the points. In addition, we propose a local information embedding module to fast select meaningful points and serve as the input of the Transformer. We also consider the correlation between channels of point cloud features and further introduce a channel attention module. Extensive experiments prove that our method outperforms existing methods, and maintains a high running speed.

Paper] Hybrid Spatial and Deep Learning-based Point Cloud Compression with Layered Representation on 3D Shape

Paper] Hybrid Spatial and Deep Learning-based Point Cloud Compression with Layered Representation on 3D Shape

MIX-NET: Deep Learning-Based Point Cloud Processing Method for Segmentation and Occlusion Leaf Restoration of Seedlings

In this paper, a novel point cloud segmentation and completion framework is proposed to achieve high-quality leaf area measurement of melon seedlings. In particular, the input of our algorithm is the point cloud data collected by an Azure Kinect camera from the top view of the seedlings, and our method can enhance measurement accuracy from two aspects based on the acquired data. On the one hand, we propose a neighborhood space-constrained method to effectively filter out the hover points and outlier noise of the point cloud, which can enhance the quality of the point cloud data significantly. On the other hand, by leveraging the purely linear mixer mechanism, a new network named MIX-Net is developed to achieve segmentation and completion of the point cloud simultaneously. Different from previous methods that separate these two tasks, the proposed network can better balance these two tasks in a more definite and effective way, leading to satisfactory performance on these two tasks. The experimental results prove that our methods can outperform other competitors and provide more accurate measurement results. Specifically, for the seedling segmentation task, our method can obtain a 3.1% and 1.7% performance gain compared with PointNet++ and DGCNN, respectively. Meanwhile, the R2 of leaf area measurement improved from 0.87 to 0.93 and MSE decreased from 2.64 to 2.26 after leaf shading completion.

DOPNet: Achieving Accurate and Efficient Point Cloud Registration Based on Deep Learning and Multi-Level Features.

Point cloud registration aims to find a rigid spatial transformation to align two given point clouds; it is widely deployed in many areas of computer vision, such as target detection, 3D localization, and so on. In order to achieve the desired results, registration error, robustness, and efficiency should be comprehensively considered. We propose a deep learning-based point cloud registration method, called DOPNet. DOPNet extracts global features of point clouds with a dynamic graph convolutional neural network (DGCNN) and cascading offset-attention modules, and the transformation is predicted by a multilayer perceptron (MLP). To enhance the information interaction between the two branches, the feature interaction module is inserted into the feature extraction pipeline to implement early data association. We compared DOPNet with the traditional method of using the iterative closest point (ICP) algorithm along with four learning-based registration methods on the Modelnet40 data set. In the experiments, the source and target point clouds were generated by sampling the original point cloud twice independently; we also conducted additional experiments with asymmetric objects. Further evaluation experiments were conducted with point cloud models from Stanford University. The results demonstrated that our DOPNet method outperforms these comparative methods in general, achieving more accurate and efficient point cloud registration.

Sparse 3D Point Cloud Parallel Multi-Scale Feature Extraction and Dense Reconstruction with Multi-Headed Attentional Upsampling

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

An Efficient Ensemble Deep Learning Approach for Semantic Point Cloud Segmentation Based on 3D Geometric Features and Range Images.

Mobile light detection and ranging (LiDAR) sensor point clouds are used in many fields such as road network management, architecture and urban planning, and 3D High Definition (HD) city maps for autonomous vehicles. Semantic segmentation of mobile point clouds is critical for these tasks. In this study, we present a robust and effective deep learning-based point cloud semantic segmentation method. Semantic segmentation is applied to range images produced from point cloud with spherical projection. Irregular 3D mobile point clouds are transformed into regular form by projecting the clouds onto the plane to generate 2D representation of the point cloud. This representation is fed to the proposed network that produces semantic segmentation. The local geometric feature vector is calculated for each point. Optimum parameter experiments were also performed to obtain the best results for semantic segmentation. The proposed technique, called SegUNet3D, is an ensemble approach based on the combination of U-Net and SegNet algorithms. SegUNet3D algorithm has been compared with five different segmentation algorithms on two challenging datasets. SemanticPOSS dataset includes the urban area, whereas RELLIS-3D includes the off-road environment. As a result of the study, it was demonstrated that the proposed approach is superior to other methods in terms of mean Intersection over Union (mIoU) in both datasets. The proposed method was able to improve the mIoU metric by up to 15.9% in the SemanticPOSS dataset and up to 5.4% in the RELLIS-3D dataset.

A hybrid deep learning model with error correction for photovoltaic power forecasting

The penetration of photovoltaic (PV) power into modern power systems brings enormous economic and environmental benefits due to its cleanness and inexhaustibility. Therefore, accurate PV power forecasting is a pressing and rigid demand to reduce the negative impact of its randomness and intermittency on modern power systems. In this paper, we explore the application of deep learning based hybrid technologies for ultra-short-term PV power forecasting consisting of a feature engineering module, a deep learning-based point prediction module, and an error correction module. The isolated forest based feature preprocessing module is used to detect the outliers in the original data. The non-pooling convolutional neural network (NPCNN), as the deep learning based point prediction module, is developed and trained using the processed data to identify non-linear features. The historical forecasting errors between the forecasting and actual PV data are further constructed and trained to correct the forecasting errors, by using an error correction module based on a hybrid of wavelet transform (WT) and k-nearest neighbor (KNN). In the simulations, the proposed method is extensively evaluated on actual PV data in Limburg, Belgium. Experimental results show that the proposed hybrid model is beneficial for improving the performance of PV power forecasting compared with the benchmark methods.

PCUNet: A Context-Aware Deep Network for Coarse-to-Fine Point Cloud Completion

Point cloud completion aims at predicting a complete 3D shape from an incomplete input. It has important applications in the fields of intelligent manufacturing, augmented reality, virtual reality, self-driving cars, and intelligent robotics. Although deep learning-based point cloud completion technology has developed rapidly in recent years, there are still unsolved problems. Previous approaches predict each point independently and ignore contextual information. And, they usually predict a complete 3D shape based on a global feature vector extracted from an incomplete input, which leads to missing of some fine-grained details. In this paper, motivated by the transposed convolution and the “UNet” structure in neural networks for image processing, we propose a context-aware deep network termed as PCUNet for coarse-to-fine point cloud completion. It adopts an encoder-decoder structure, in which the encoder follows the design of the relation-shape convolutional neural network (RS-CNN), and the decoder consists of fully-connected layers and two stacked decoder modules for predicting complete point clouds. The contributions are twofold. First, we design the decoder module as a coordinate-guided context-aware upsampling module, in which contextual information can be taken into full account by neighbor aggregation. Second, to preserve fine-grained details in the input, we propose attention-enhanced skip connections for effective information propagation from the encoder to the decoder. Experiments are conducted on the widely used PCN and KITTI datasets. The results show that our proposed approach achieves competitive performance compared to the existing state-of-the-art approaches in terms of the Chamfer distance and the computational complexity metrics.