Sparse 3D Point Cloud Research Articles

A novel 3D reconstruction method of blast furnace burden surface based on virtual camera array

Accurate three-dimensional (3D) blast furnace burden surface topography is crucial for optimizing material distribution and furnace operation. However, harsh conditions within the furnace pose challenges to this 3D reconstruction, such as difficulties in matching feature points and sparse 3D point clouds. To overcome these challenges, we propose a 3D reconstruction method for the blast furnace burden surface using virtual camera array. Firstly, an ORB feature extraction and matching method based on bidirectional optical flow tracking is proposed to solve the impact of illumination changes and noise. Then, the virtual camera array is constructed to generate a sparse 3D point cloud from an image sequence. Finally, dense surface reconstruction of the burden surface is achieved based on the improved PMVS and the Poisson surface reconstruction method. Experimental results demonstrate that the proposed method can extract accurate feature points and reconstruct the accurate 3D topography of the burden surface.

Introducing an indoor object classification dataset including sparse point clouds from mmWave radar

This document introduces the RadIOCD, which is a dataset that contains sparse point cloud representations of indoor objects, collected by subjects wearing a commercial off-the-shelf mmWave radar. In particular, RadIOCD includes the recordings of 10 volunteers moving towards 5 different objects (i.e., backpack, chair, desk, human, and wall), placed in 3 different environments. RadIOCD includes sparse 3D point cloud data, together with their doppler velocity and intensity provided by the mmWave radar. A total of 5,776 files are available, with each one having an approximate duration of 8s. The scope of RadIOCD is the availability of data for the recognition of objects solely recorded by the mmWave radar, to be used in applications were the vision-based classification is cumbersome though critical (e.g., in search and rescue operation where there is smoke inside a building). Furthermore, we showcase that this dataset after being segmented into 76,821 samples contains enough data to apply Machine Learning-based techniques, ensuring that they could generalize in different environments and “unseen“ subjects.

Stereo reconstruction from microscopic images for computer-assisted ophthalmic surgery.

This work presents a novel platform for stereo reconstruction in anterior segment ophthalmic surgery to enable enhanced scene understanding, especially depth perception, for advanced computer-assisted eye surgery by effectively addressing the lack of texture and corneal distortions artifacts in the surgical scene. The proposed platform for stereo reconstruction uses a two-step approach: generating a sparse 3D point cloud from microscopic images, deriving a dense 3D representation by fitting surfaces onto the point cloud, and considering geometrical priors of the eye anatomy. We incorporate a pre-processing step to rectify distortion artifacts induced by the cornea's high refractive power, achieved by aligning a 3D phenotypical cornea geometry model to the images and computing a distortion map using ray tracing. The accuracy of 3D reconstruction is evaluated on stereo microscopic images of ex vivo porcine eyes, rigid phantom eyes, and synthetic photo-realistic images. The results demonstrate the potential of the proposed platform to enhance scene understanding via an accurate 3D representation of the eye and enable the estimation of instrument to layer distances in porcine eyes with a mean average error of 190 , comparable to the scale of surgeons' hand tremor. This work marks a significant advancement in stereo reconstruction for ophthalmic surgery by addressing corneal distortions, a previously often overlooked aspect in such surgical scenarios. This could improve surgical outcomes by allowing for intra-operative computer assistance, e.g., in the form of virtual distance sensors.

Enhancing Building Point Cloud Reconstruction from RGB UAV Data with Machine-Learning-Based Image Translation.

The performance of three-dimensional (3D) point cloud reconstruction is affected by dynamic features such as vegetation. Vegetation can be detected by near-infrared (NIR)-based indices; however, the sensors providing multispectral data are resource intensive. To address this issue, this study proposes a two-stage framework to firstly improve the performance of the 3D point cloud generation of buildings with a two-view SfM algorithm, and secondly, reduce noise caused by vegetation. The proposed framework can also overcome the lack of near-infrared data when identifying vegetation areas for reducing interferences in the SfM process. The first stage includes cross-sensor training, model selection and the evaluation of image-to-image RGB to color infrared (CIR) translation with Generative Adversarial Networks (GANs). The second stage includes feature detection with multiple feature detector operators, feature removal with respect to the NDVI-based vegetation classification, masking, matching, pose estimation and triangulation to generate sparse 3D point clouds. The materials utilized in both stages are a publicly available RGB-NIR dataset, and satellite and UAV imagery. The experimental results indicate that the cross-sensor and category-wise validation achieves an accuracy of 0.9466 and 0.9024, with a kappa coefficient of 0.8932 and 0.9110, respectively. The histogram-based evaluation demonstrates that the predicted NIR band is consistent with the original NIR data of the satellite test dataset. Finally, the test on the UAV RGB and artificially generated NIR with a segmentation-driven two-view SfM proves that the proposed framework can effectively translate RGB to CIR for NDVI calculation. Further, the artificially generated NDVI is able to segment and classify vegetation. As a result, the generated point cloud is less noisy, and the 3D model is enhanced.

Mm-CasGAN: A cascaded adversarial neural framework for mmWave radar point cloud enhancement

Handling and interpreting sparse 3D point clouds, especially from mmWave radar, presents unique challenges due to the inherent data sparsity and the vast domain difference compared to denser point clouds like those from LiDAR. In this paper, we introduce a novel cascaded generative adversarial network (GAN) approach to bridge this domain gap. The core principle is to progressively refine the radar-based point cloud through a series of GANs, each targeting a higher resolution. By leveraging multi-level features and a hybrid loss function that combines adversarial, geometric, and consistency components, our method ensures a smooth transition from the sparse radar representation to a high-resolution LiDAR-like point cloud. Our cascaded approach operates at a patch level, and the integrated loss function ensures that the generated points not only resemble the target domain but also maintain geometric and structural fidelity. Real-life dataset consisting mostly of moving pedestrians were collected using a system made of Radar, LiDAR, and RGB Camera. Through an extensive experiment on the collected real-world pedestrian dataset, we validate the efficacy of our approach. Inference from the network indicates that our method can upsample mmWave radar point clouds with enhanced density, uniformity, and closer alignment to the ground truth LiDAR point clouds, which is the first of its kind network to do so.

Towards end-to-end deep RNN based networks to precisely regress of the lettuce plant height by single perspective sparse 3D point cloud

Nowadays, 3D point cloud is supposed to be the most direct and effective data form for studying plant morphology structure. However, automatic and high-throughput acquisition of accurate individual plant height traits from 3D point cloud remains an urgent challenging problem. Summarizing the related research results in recent years, the factors limiting its application mainly come from these aspects: (1) Many existing methods require spatial auxiliary information such as ground control points (GCP), digital terrain models (DTM) and digital surface models (DSM) to obtain accurate plant height; (2) For 3D point cloud data in different environments, specialized modeling and careful parameter fine-tuning are usually required; (3) Sometimes, the point cloud processing involves the combined utilization of multiple programming languages and software, which is difficult for system integration. Focusing on these challenges, firstly, we proposed a novel end-to-end deep Recurrent Neural Network (RNN) based regression network framework called DRN, which consists of three parts: point cloud feature extraction network, deep RNN and regression network. The convolution operations-based point cloud feature extraction network is function as filtering noise, outliers and redundant information; The deep RNN network with long and short-term memory (LSTM) ability is used to learn the relationships between the feature points on the high-dimensional feature sequence separated by a certain distance; regression network is used to regress the output from deep RNN to plant height value. Experiments results on the 3rd Greenhouse Growing Challenge datasets show that DRN can directly regress the plant height of a single plant effectively without manual operations and the participation of spatial auxiliary information with an R2 of 0.948 and a relative root mean square error (RRMSE) of 10.06% in four different varieties of lettuce at different growth period. After studying the influence of the weights of the x, y, z coordinate of the input 3D point cloud on the regression result, then, we design a Dimension Attention (DA) module at the front end of the feature extraction network to learning the characteristic coordinate weight for every input point cloud sample. The DRN network with a DA module is called D-DRN, experiment results indicate D-DRN tend to achieve better result (R2 =0.960 ; RRMSE=8.680%) than DRN. Considering the end-to-end-based DRN and D-DRN network capable of ease of integration and their considerable prediction accuracy on public datasets, we believe they has a certain complementary effect on the existing study methods of obtaining plant morphological structure phenotype by point cloud data.

PIDFusion: Fusing Dense LiDAR Points and Camera Images at Pixel-Instance Level for 3D Object Detection

In driverless systems (scenarios such as subways, buses, trucks, etc.), multi-modal data fusion, such as light detection and ranging (LiDAR) points and camera images, is essential for accurate 3D object detection. In the fusion process, the information interaction between the modes is challenging due to the different coordinate systems of various sensors and the significant difference in the density of the collected data. It is necessary to fully consider the consistency and complementarity of multi-modal information, make up for the gap between multi-source data density, and achieve the joint interactive processing of multi-source information. Therefore, this paper is based on Transformer to improve a new multi-modal fusion model called PIDFusion for 3D object detection. Firstly, the method uses the results of 2D instance segmentation to generate dense 3D virtual points to enhance the original sparse 3D point clouds. This optimizes the issue that the nearest Euclidean distance in the 2D image space cannot ensure the nearest in the 3D space. Secondly, a new cross-modal fusion architecture is designed to maintain individual per-modality features to take advantage of their unique characteristics during 3D object detection. Finally, an instance-level fusion module is proposed to enhance semantic consistency through cross-modal feature interaction. Experiments show that PIDFusion is far ahead of existing 3D object detection methods, especially for small and long-range objects, with 70.8 mAP and 73.5 NDS on the nuScenes test set.

Development of vehicle- recognition method on water surfaces using LiDAR data: SPD2 (spherically stratified point projection with diameter and distance)

Swarm robot systems are an important application of autonomous unmanned surface vehicles on water surfaces. For monitoring natural environments and conducting security activities within a certain range using a surface vehicle, the swarm robot system is more efficient than the operation of a single object as the former can reduce cost and save time. It is necessary to detect adjacent surface obstacles robustly to operate a cluster of unmanned surface vehicles. For this purpose, a LiDAR (light detection and ranging) sensor is used as it can simultaneously obtain 3D information for all directions, relatively robustly and accurately, irrespective of the surrounding environmental conditions. Although the GPS (global-positioning-system) error range exists, obtaining measurements of the surface-vessel position can still ensure stability during platoon maneuvering. In this study, a three-layer convolutional neural network is applied to classify types of surface vehicles. The aim of this approach is to redefine the sparse 3D point cloud data as 2D image data with a connotative meaning and subsequently utilize this transformed data for object classification purposes. Hence, we have proposed a descriptor that converts the 3D point cloud data into 2D image data. To use this descriptor effectively, it is necessary to perform a clustering operation that separates the point clouds for each object. We developed voxel-based clustering for the point cloud clustering. Furthermore, using the descriptor, 3D point cloud data can be converted into a 2D feature image, and the converted 2D image is provided as an input value to the network. We intend to verify the validity of the proposed 3D point cloud feature descriptor by using experimental data in the simulator. Furthermore, we explore the feasibility of real-time object classification within this framework.

SparseFormer: Sparse transformer network for point cloud classification

Compared to the traditional self-attention structure of Transformers, the MLP-like structure offers advantages such as simplicity and improved performance. However, effectively and efficiently learning features from sparse, irregular, and unordered 3D point cloud data remains a challenge. To address this issue, we propose SparseFormer, a sparse transformer network designed specifically for point cloud processing tasks. SparseFormer incorporates a sparse MLP module that enables accurate feature learning while considering the unique characteristics of 3D point cloud data. Additionally, we enhance the context information by utilizing a multi-scale feature aggregation module. Experimental results demonstrate the superior performance of SparseFormer on classification tasks using benchmark datasets, including the ModelNet40 synthetic dataset and the ScanObjectNN real-world dataset. In the classification experiment on the ScanObjectNN dataset, SparseFormer achieves a mean accuracy of 84.1% and an overall accuracy of 85.5%.

A Category-Contrastive Guided-Graph Convolutional Network Approach for the Semantic Segmentation of Point Clouds

The semantic segmentation of light detection and ranging (LiDAR) point clouds plays an important role in 3D scene intelligent perception and semantic modeling. The unstructured, sparse and uneven characteristics of point clouds pose great challenges to the representation of the local geometric shapes, which degrades semantic segmentation performance. To address the challenges of describing local geometric shapes due to unstructured and sparse 3D point clouds, this paper proposes a category-contrastive-guided graph convolutional network (CGGC-Net) for the semantic segmentation of LiDAR point clouds. First, a detailed geometric structure of the raw point clouds is encoded to represent the inherent geometric pattern within the local neighborhood. At the same time, the geometric structures information is transmitted across multiple layers, so that the geometric structure encoding information containing different receptive fields and richer neighborhood spatial structure can be aggregated. Following this, the graph convolution neural network uses the edge convolution layer to adaptively describe the semantic correlation between the query point and its neighboring points, and combines the attention mechanism to gather the surrounding feature information to the query point. As a result, the graph convolution neural network and attention mechanism are iteratively stacked for the aggregation and fusion of spatial context semantic information, to generate highly discriminative semantic feature representation. Finally, the super parameters of the model are learned through a multitask optimization strategy guided by category-aware contrastive loss and cross-entropy loss. Experiments are conducted on the public SemanticKITTI dataset and the Stanford Large-Scale 3D Indoor Spaces dataset to demonstrate the effectiveness and reliability of the proposed CGGC-Net from both quantitative and qualitative perspectives. The results indicate its capability of automatically classifying LiDAR point clouds, with a mean intersection-over-union (mIoU) of 58.4%. Moreover, multiple comparative experiments also demonstrate the superior performance of the proposed method, exceeding state-of-the-art methods.

Multi-Level Structure-Enhanced Network for 3D Single Object Tracking in Sparse Point Clouds

3D single object tracking is the task of localizing a target object in a search point cloud frame. In this letter, we present a multi-level structure-enhanced tracking model to improve the tracking performance in sparse 3D point clouds. Towards this end, we first encode the target and the search point clouds efficiently in two near-neighbors graphs, which allows structural information to flow between neighbors along graph edges. We then design a cross-graph attention mechanism, which associates similar nodes across the target graph and the search graph, and further dissimilar graph nodes for apart. Integrating the proposed mechanism into the above Siamese feature learning of both the target and the search frame, we strengthen the structural correlation between the target and the search frame. In that case, distinguishing the potential target from the background in the search frame would be much simpler. Finally, we design a U-shaped sparse convolutional block to aggregate the structural features of the potential target in the search frame. Integrating the proposed block into an existing target localization module from (Hui et al., 2021), we localize target centers accurately. Experiments on the KITTI benchmark demonstrate that our method outperforms some state-of-the-art models, achieving at least a 3.2% improvement in terms of average tracking precision.

Sparse 3D Point Cloud Parallel Multi-Scale Feature Extraction and Dense Reconstruction with Multi-Headed Attentional Upsampling

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

Structure-aware fusion network for 3D scene understanding

In this paper, we propose a Structure-Aware Fusion Network (SAFNet) for 3D scene understanding. As 2D images present more detailed information while 3D point clouds convey more geometric information, fusing the two complementary data can improve the discriminative ability of the model. Fusion is a very challenging task since 2D and 3D data are essentially different and show different formats. The existing methods first extract 2D multi-view image features and then aggregate them into sparse 3D point clouds and achieve superior performance. However, the existing methods ignore the structural relations between pixels and point clouds and directly fuse the two modals of data without adaptation. To address this, we propose a structural deep metric learning method on pixels and points to explore the relations and further utilize them to adaptively map the images and point clouds into a common canonical space for prediction. Extensive experiments on the widely used ScanNetV2 and S3DIS datasets verify the performance of the proposed SAFNet.

Convolutional Neural Network Using for Multi-Sensor 3D Object Detection

The purpose of this article is to detect 3D objects inside the independent vehicle with great accuracy. The method proposed a Multi-View 3D System (MV3D) framework which encodes the sparse 3d-point cloud with a compact multi-view image, using LIDAR satellite image and RGB pictures as inputs, and predicts 3D boundary boxes. The network comprises two sub-networks: one for creating 3D artifacts and one for multi-visual fusion functionality. Propose an autonomous 3D object tracking approach to manipulate sparse and dense knowledge about romanticizing and geometry in stereo images. The Stereo R-CNN strategy applies Faster R-CNNs to stereo inputs such that objects are simultaneously identified and linked in conservative and liberal images. Such charts were then combined and fed into a 3D proposal generator to generate accurate 3D proposals for vehicles. In the second step, the refining network extended the features of the proposal regions further and carried through the classification, regression of a 3D package box, and guidance estimates, to predict vehicle location and heading in 3D area and add additional branches after the stereo region Proposal Network (RPN).

Dehazing cost volume for deep multi-view stereo in scattering media with airlight and scattering coefficient estimation

We propose a learning-based multi-view stereo (MVS) method in scattering media, such as fog or smoke, with a novel cost volume, called the dehazing cost volume. Images captured in scattering media are degraded due to light scattering and attenuation caused by suspended particles. This degradation depends on scene depth; thus, it is difficult for traditional MVS methods to evaluate photometric consistency because the depth is unknown before three-dimensional (3D) reconstruction. The dehazing cost volume can solve this chicken-and-egg problem of depth estimation and image restoration by computing the scattering effect using swept planes in the cost volume. We also propose a method of estimating scattering parameters, such as airlight, and a scattering coefficient, which are required for our dehazing cost volume. The output depth of a network with our dehazing cost volume can be regarded as a function of these parameters; thus, they are geometrically optimized with a sparse 3D point cloud obtained at a structure-from-motion step. Experimental results on synthesized hazy images indicate the effectiveness of our dehazing cost volume against the ordinary cost volume regarding scattering media. We also demonstrated the applicability of our dehazing cost volume to real foggy scenes.

3D URBAN CHANGE DETECTION WITH POINT CLOUD SIAMESE NETWORKS

Abstract. As the majority of the earth population is living in urban environments, cities are continuously evolving and efficient monitoring tools are needed to retrieve and classify their evolution. In this context, analysing changes between two dates is a crucial point. In urban environments, most changes occur along the vertical axis (with new construction or demolition of buildings) and the use of 3D data is therefore mandatory. Among them, LiDAR constitutes a valuable source of information. However, With the difficulty of processing sparse and unordered 3D point clouds, most of existing methods start by rasterizing point clouds (for example to Digital Surface Models) before using more conventional image processing tools. This implies a significant loss of information. Among existing studies dealing directly with point clouds, and to the best of our knowledge, no deep neural network-based method has been explored yet. Thus, in order to fill this gap and to test the ability of deep methods to deal with change detection and characterization of 3D point clouds, we propose a Siamese network with Kernel Point Convolution inspired by Siamese architectures that have already shown their performances on change detection in 2D images and on KPConv network which achieves high-quality results for semantic segmentation of raw 3D point clouds. We show quantitatively and qualitatively that our method outperforms by more than 25% (in terms of average Intersection over Union for classes of change) existing machine learning methods based on hand-crafted features.

3D Point Clouds Data Super Resolution-Aided LiDAR Odometry for Vehicular Positioning in Urban Canyons

LiDAR odometry algorithms are extensively studied for vehicular positioning. However, achieving high-precision positioning using low-cost 16-channel LiDAR in urban canyons remains a challenging problem due to the limited point cloud density from low-cost LiDAR and excessive dynamic surrounding objects. To fill this gap, this paper proposes enriching sparse 3D point clouds to denser clouds via a novel deep learning-based superresolution (SR) algorithm before its utilization in 3D LiDAR odometry. We validate the effectiveness of the proposed method using the KITTI dataset and a challenging dataset collected in an urban canyon (with complex environmental structures and dynamic objects) of Hong Kong. We conclude that significantly denser point clouds are achieved with considerable accuracy. In addition, significantly improved performance of 3D LiDAR odometry is obtained in the evaluated dataset collected in an urban canyon of Hong Kong.

FusionVLAD: A Multi-View Deep Fusion Networks for Viewpoint-Free 3D Place Recognition

Real-time 3D place recognition is a crucial technology to recover from localization failure in applications like autonomous driving, last-mile delivery, and service robots. However, it is challenging for 3D place retrieval methods to be accurate, efficient, and robust to the variant viewpoints differences. In this letter, we propose FusionVLAD, a fusion-based network that encodes a multi-view representation of sparse 3D point clouds into viewpoint-free global descriptors. The system consists of two parallel branches: a spherical-view branch for orientation-invariant feature extraction, and the top-down view branch for translation-insensitive feature extraction. Furthermore, we design a parallel fusion module to enhance the combination of region-wise feature connection between the two branches. Experiments on two public datasets and two generated datasets show that our method outperforms state-of-the-art with robust place recognition accuracy and efficient inference time. Besides, FusionVLAD requires limited computation resources and makes it extremely suitable for low-cost robots' long-term place recognition task.

Pantomime

We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatio-temporal properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.

Real-Time Segmentation of Sparse 3D Point Clouds on a TX2

In this paper, we propose a real-time 3D object detection method, which from sparse 3D point clouds and based on NVIDIA Jetson TX2. This effective method first uses the external parameter matrix to transform the coordinates of the point clouds data, then supplements the missing data in the original data through the interpolation algorithm, and then uses the RANSAC algorithm to remove the ground, uses the improved DBSCAN clustering algorithm, which clustering segmentation is performed for the point clouds fusion partition distance threshold and angle threshold of different distance intervals. Finally, we regress the obstacle boundary to the mini-box by gradient descent, output for the result is visualized in the form of cuboid detection. Compared to traditional DBSCAN algorithm, our method reduces the missed detection rate and improves the real-time performance. Above all, we implements our approach in C++ and ROS, and the experimental results show that our method produces high quality object detection by using NVIDIA Jetson TX2 online.