Raw LiDAR Point Research Articles

Multimodal spatiotemporal aggregation for point cloud accumulation

Point cloud accumulation is a crucial technique in point cloud analysis, facilitating various downstream tasks like surface reconstruction. Current methods merely rely on raw LiDAR points, yielding unsatisfactory performance due to the limited geometric information, particularly in complex scenarios characterized by intricate motions, diverse objects, and an increased number of frames. In this paper, we introduce camera modality data, which is usually acquired alongside LiDAR data at minimal expense. To this end, we present the Multimodal Spatiotemporal Aggregation solution (termed MSA) to thoroughly explore and aggregate these two distinct modalities (sparse 3D points and multi-view 2D images). Concretely, we propose a multimodal spatial aggregation module to bridge the data gap between different modalities in the Bird’s-Eye-View (BEV) space and further fuse them by learnable adaptive channel-wise weights. By assembling their respective strengths, this module generates a reliable and consistent scene representation. Subsequently, we design a temporal aggregation module to capture continuous motion information across consecutive sequences, which is beneficial for identifying the motion state of the foreground scene and enabling the model to extend robustly to longer sequences. Experiments demonstrate MSA outperforms state-of-the-art (SoTA) point cloud accumulation methods across all evaluation metrics in the public benchmark, especially with more frames.

Pre-Segmented Down-Sampling Accelerates Graph Neural Network-Based 3D Object Detection in Autonomous Driving.

Graph neural networks (GNNs) have been proven to be an ideal approach to deal with irregular point clouds, but involve massive computations for searching neighboring points in the graph, which limits their application in large-scale LiDAR point cloud processing. Down-sampling is a straightforward and indispensable step in current GNN-based 3D detectors to reduce the computational burden of the model, but the commonly used down-sampling methods cannot distinguish the categories of the LiDAR points, which leads to an inability to effectively improve the computational efficiency of the GNN models without affecting their detection accuracy. In this paper, we propose (1) a LiDAR point cloud pre-segmented down-sampling (PSD) method that can selectively reduce background points while preserving the foreground object points during the process, greatly improving the computational efficiency of the model without affecting its 3D detection accuracy. (2) A lightweight GNN-based 3D detector that can extract point features and detect objects from the raw down-sampled LiDAR point cloud directly without any pre-transformation. We test the proposed model on the KITTI 3D Object Detection Benchmark, and the results demonstrate its effectiveness and efficiency for autonomous driving 3D object detection.

Depth- and semantics-aware multi-modal domain translation: Generating 3D panoramic color images from LiDAR point clouds

This work presents a new depth- and semantics-aware conditional generative model, named TITAN-Next, for cross-domain image-to-image translation in a multi-modal setup between LiDAR and camera sensors. The proposed model leverages scene semantics as a mid-level representation and is able to translate raw LiDAR point clouds to RGB-D camera images by solely relying on semantic scene segments. We claim that this is the first framework of its kind and it has practical applications in autonomous vehicles such as providing a fail-safe mechanism and augmenting available data in the target image domain. The proposed model is evaluated on the large-scale and challenging Semantic-KITTI dataset, and experimental findings show that it considerably outperforms the original TITAN-Net and other strong baselines by 23.7% margin in terms of IoU.

Adaptive LiDAR Reconstruction by Convolutional Compressive Sensing Network and Multivariate Empirical Mode Decomposition

With the recent advancement of LiDAR data processing pipelines integrated with signal processing techniques, it is now possible to rapidly acquire, process, and infer multidimensional information of 3D surroundings. This work proposes a novel compressive sensing (CS) framework (named MEMD-CSNet), based on Multivariate Empirical Mode Decomposition (MEMD) for adaptive 3D reconstruction of airborne urban LiDAR point clouds. EMD is a data-driven transform for decomposing the input data into Intrinsic Mode Functions (IMFs), capturing the implicit features in terms of variation in spatial frequency. Our approach generates a new filtered 3D signal constituting an optimal subset of IMFs based on the understanding of their statistical significance. Further, this filtered signal hierarchically extracts the overall shape in the last IMF (Residual function). Our proposed MEMD-CSNet is a deep convolutional network that (1) computes a multivariate EMD on input raw 3D LiDAR point clouds for generating data-dependent feature-rich filtered signal and (2) additionally augments this signal with convolutional features, thus learning a data-centric representation function (Ψ) for transforming the input into a low-dimensional sparse basis. We present extensive qualitative and quantitative analysis using the proposed MEMD-CSNet for a set of chosen CS measurement ratios. MEMD-CSNet achieves a maximum reconstruction PSNR of 108.22 dB (approx. 30 dB/1.36x better than without using MEMD) for reconstruction fidelity on selected urban LiDAR scenes. We envisage that our proposed MEMD-CSNet has a high potential (as a data-centric framework) for generating data-driven sparse feature maps and reconstructing entire 3D LiDAR point clouds from a few sparse measurements.

Hyperspectral image-aided LiDAR point cloud labeling via spatio-spectral feature representation learning

Urban scene-level 3D point cloud labeling is a very laborious and expensive task compared to images. Conversely however, image processing techniques, deep learning or otherwise are more established and mature. Thus, in a multi-source data environment, the labeling of a point cloud scene via an automated image process as an initial step, followed by a manual human verification process is an effective way to save man hours and cost. With the above as the goal, this study presents a simple but robust spatio-spectral feature representation approach. In this approach, a class-aware band selection and reduction (CBSR) technique is developed for optimal hyperspectral feature representation. A double-branched convolutional Gaussian Bernoulli deep belief network (CGBDBN) is then used for hierarchical spatial feature extraction from LiDAR-derived data and the CBSR data. Using stacked ensemble learning, spatio-spectral features are generated from the two feature streams via a fusion rule and then classified — the results of which are used in labeling a raw 3D LiDAR point cloud through projection. To evaluate this study, extensive experiments were conducted on the IEEE 2018 Houston dataset — the only publicly available dataset with both hyperspectral image (HSI) and 3D point cloud covering the same area — for urban scene classification. The results indicate that the developed CBSR attained comparatively competitive results with state-of-the-art approaches, thus making it a robust spectral feature representation technique. Also, the weight-sharing property, probabilistic modeling, and hierarchical nature of CGBDBN gives our approach the ability to capture high-level contextual features. Furthermore, compared to the spatial- or spectral-only features, the generated spatio-spectral features are more discriminative and significantly aided in improving the proposed model’s efficacy. Overall, the proposed approach, based on the evaluation metrics, is a robust and effective approach for both coarse- and fine-grained raw LiDAR point cloud labeling tasks.

Efficient graph attentional network for 3D object detection from Frustum-based LiDAR point clouds

LiDAR-based 3D object detection is important for autonomous driving scene perception, but point clouds produced by LiDAR are irregular and unstructured in nature, and cannot be adopted by the conventional Convolutional Neural Networks (CNN). Recently, Graph Convolutional Networks (GCN) has been proved as an ideal way to handle non-Euclidean structure data, as well as for point cloud processing. However, GCN involves massive computation for searching adjacent nodes, and the heavy computational cost limits its applications in processing large-scale LiDAR point cloud in autonomous driving. In this work, we adopt a frustum-based point cloud-image fusion scheme to reduce the amount of LiDAR point clouds, thus making the GCN-based large-scale LiDAR point clouds feature learning feasible. On this basis, we propose an efficient graph attentional network to accomplish the goal of 3D object detection in autonomous driving, which can learn features from raw LiDAR point cloud directly without any conversions. We evaluate the model on the public KITTI benchmark dataset, the 3D detection mAP is 63.72% on KITTI Cars, Pedestrian and Cyclists, and the inference speed achieves 7.9 fps on a single GPU, which is faster than other methods of the same type.

Extracting Urban Road Footprints from Airborne LiDAR Point Clouds with PointNet++ and Two-Step Post-Processing

In this paper, a novel framework for the automatic extraction of road footprints from airborne LiDAR point clouds in urban areas is proposed. The extraction process consisted of three phases: The first phase is to extract road points by using the deep learning model PointNet++, where the features of the input data include not only those selected from raw LiDAR points, such as 3D coordinate values, intensity, etc., but also the digital number (DN) of co-registered images and generated geometric features to describe a strip-like road. Then, the road points from PointNet++ were post-processed based on graph-cut and constrained triangulation irregular networks, where both the commission and omission errors were greatly reduced. Finally, collinearity and width similarity were proposed to estimate the connection probability of road segments, thereby improving the connectivity and completeness of the road network represented by centerlines. Experiments conducted on the Vaihingen data show that the proposed framework outperformed others in terms of completeness and correctness; in addition, some narrower residential streets with 2 m width, which have normally been neglected by previous studies, were extracted. The completeness and the correctness of the extracted road points were 84.7% and 79.7%, respectively, while the completeness and the correctness of the extracted centerlines were 97.0% and 86.3%, respectively.

Boosting Monocular 3D Object Detection With Object-Centric Auxiliary Depth Supervision

Recent advances in monocular 3D detection leverage a depth estimation network explicitly as an intermediate stage of the 3D detection network. Depth map approaches yield more accurate depth to objects than other methods thanks to the depth estimation network trained on a large-scale dataset. However, depth map approaches can be limited by the accuracy of the depth map, and sequentially using two separated networks for depth estimation and 3D detection significantly increases computation cost and inference time. In this work, we propose a method to boost the RGB image-based 3D detector by jointly training the detection network with a depth prediction loss analogous to the depth estimation task. In this way, our 3D detection network can be supervised by more depth supervision from raw LiDAR points, which does not require any human annotation cost, to estimate accurate depth without explicitly predicting the depth map. Our novel object-centric depth prediction loss focuses on depth around foreground objects, which is important for 3D object detection, to leverage pixel-wise depth supervision in an object-centric manner. Our depth regression model is further trained to predict the uncertainty of depth to represent the 3D confidence of objects. To effectively train the 3D detector with raw LiDAR points and to enable end-to-end training, we revisit the regression target of 3D objects and design a network architecture. Extensive experiments on KITTI and nuScenes benchmarks show that our method can significantly boost the monocular image-based 3D detector to outperform depth map approaches while maintaining the real-time inference speed.

Realtime Single-Shot Refinement Neural Network With Adaptive Receptive Field for 3D Object Detection From LiDAR Point Cloud

Object detection plays an important role in autonomous driving systems. LiDAR is widely used in autonomous driving vehicles and robots as a sensor for environmental perception. Recently, the development of computational power and deep learning technology makes it possible to classify and locate objects from LiDAR point cloud in a single end-to-end learnable network. However, objects are sparsely distributed in large point cloud field, and are always been partly scanned by LiDAR, which pose a challenge for accurate and rapid object positioning and classification from the raw point cloud. In this paper, we introduce a new single-shot refinement neural network for fast and accurate 3D object detection from the raw LiDAR point cloud. Firstly, we exploit self-attention mechanism in main object detection branch to enhance object feature representation. Secondly, we apply deformable convolution for learning adaptive receptive fields to fully capture the features of rotating and partially visible objects. Thirdly, an object refinement branch is introduced to produce a finer regression of objects upon the primary estimation from the main detection branch. All proposed modules have been proven to effectively improve the accuracy of object detection. Our method is evaluated on KITTI 3D detection benchmark and achieves state-of-the-art results while maintains real-time efficiency. Furthermore, real-time test in autonomous driving vehicle demonstrates that our method is robust to 16 channels LiDAR and can meet the demands of accuracy, efficiency, and visibility of object detection in various urban scenarios.

A novel algorithm for regularization of building footprints using raw LiDAR point clouds

This study proposes an automatic building outline regularization approach (ABORE) that produces two-dimensional orthogonal building boundaries from irregular building footprint polygons with building edge points using raw LiDAR point cloud data. The proposed approach was evaluated on the combinations of single, double, and multiple buildings from seven LiDAR datasets with different point density. The accuracy of the building edge geometries regularized by the ABORE method was compared with the reference data without requiring any additional data. Test results showed that the mean completeness (Cp), correctness (Cr), and quality (Q) values for single buildings were estimated as 98.96, 99.32, 98.30% and for multiple buildings as 99.55, 99.74, 99.29%, respectively. These results indicate superior performance (by about 5% F-score improvement) in comparison to state-of-the-art methods with a higher correctness rate in building boundary regularization. ABORE was also found effective in the delineation of complex buildings with many corners and short edges.

GENERATING SYNTHETIC 3D POINT SEGMENTS FOR IMPROVED CLASSIFICATION OF MOBILE LIDAR POINT CLOUDS

Abstract. Mobile lidar point clouds are commonly used for 3d mapping of road environments as they provide a rich, highly detailed geometric representation of objects on and around the road. However, raw lidar point clouds lack semantic information about the type of objects, which is necessary for various applications. Existing methods for the classification of objects in mobile lidar data, including state of the art deep learning methods, achieve relatively low accuracies, and a primary reason for this under-performance is the inadequacy of available 3d training samples to sufficiently train deep networks. In this paper, we propose a generative model for creating synthetic 3d point segments that can aid in improving the classification performance of mobile lidar point clouds. We train a 3d Adversarial Autoencoder (3dAAE) to generate synthetic point segments that exhibit a high resemblance to and share similar geometric features with real point segments. We evaluate the performance of a PointNet-like classifier trained with and without the synthetic point segments. The evaluation results support our hypothesis that training a classifier with training data augmented with synthetic samples leads to significant improvement in the classification performance. Specifically, our model achieves an F1 score of 0.94 for vehicles and pedestrians and 1.00 for traffic signs.

Robust shape extraction for automatically segmenting raw LiDAR data of outdoor scenes

ABSTRACTLight detection and ranging (LiDAR) scanning has become a prevalent technique for digitalizing outdoor scenes with three-dimensional (3D) point clouds. An automatic segmentation of the LiDAR data is important for understanding and reconstructing the outdoor scenes. However, it is still a challenging task due to complex and various objects in the outdoor scenes and some scanning drawbacks in the LiDAR data. Observing that most of the objects, such as the ground, road, roof, and facade, can be locally described with a group of geometric shapes, e.g. plane, sphere, and cylinder, we propose an automatic method to segment raw LiDAR data by robustly extracting the shapes in this paper. Firstly, our method divides the raw LiDAR data into a number of supervoxels considering the geometric and spectral consistencies of LiDAR data. Secondly, we robustly extract shapes from each supervoxels using a random sample consensus (RANSAC)-based method and evaluates and optimises the extracted shapes with a density loss estimation technique. Finally, our method outputs the segmentation result by merging the extracted shapes into a group of complete shapes, each of them represents a meaningful object in the outdoor scene. Experiments show that the method is efficient and robust for extracting most of the shapes in the outdoor scenes from a given raw LiDAR point cloud, and no preprocessing is required.

Automatic 3D reconstruction of electrical substation scene from LiDAR point cloud

3D reconstruction of a large-scale electrical substation scene (ESS) is fundamental to navigation, information inquiry, and supervisory control of 3D scenes. However, automatic reconstruction of ESS from a raw LiDAR point cloud is challenging due to its incompleteness, noise and anisotropy in density. We propose an automatic and efficient approach to reconstruct ESSs, by mapping raw LiDAR data to our well-established electrical device database (EDD). We derive a flexible and hierarchical representation of the ESS automatically by exploring the internal topology of the corresponding LiDAR data, followed by extracting various devices from the ESS. For each device, a quality mesh model is retrieved in the EDD, based on the proposed object descriptor that can balance descriptiveness, robustness and efficiency. With the high-level representation of the ESS, we map all retrieved models into raw data to achieve a high-fidelity scene reconstruction. Extensive experiments on large and complex ESSs modeling demonstrate the efficiency and accuracy of the proposed method.

AUTOMATIC 3D BUILDING MODEL GENERATIONS WITH AIRBORNE LiDAR DATA

Abstract. LiDAR systems become more and more popular because of the potential use for obtaining the point clouds of vegetation and man-made objects on the earth surface in an accurate and quick way. Nowadays, these airborne systems have been frequently used in wide range of applications such as DEM/DSM generation, topographic mapping, object extraction, vegetation mapping, 3 dimensional (3D) modelling and simulation, change detection, engineering works, revision of maps, coastal management and bathymetry. The 3D building model generation is the one of the most prominent applications of LiDAR system, which has the major importance for urban planning, illegal construction monitoring, 3D city modelling, environmental simulation, tourism, security, telecommunication and mobile navigation etc. The manual or semi-automatic 3D building model generation is costly and very time-consuming process for these applications. Thus, an approach for automatic 3D building model generation is needed in a simple and quick way for many studies which includes building modelling. In this study, automatic 3D building models generation is aimed with airborne LiDAR data. An approach is proposed for automatic 3D building models generation including the automatic point based classification of raw LiDAR point cloud. The proposed point based classification includes the hierarchical rules, for the automatic production of 3D building models. The detailed analyses for the parameters which used in hierarchical rules have been performed to improve classification results using different test areas identified in the study area. The proposed approach have been tested in the study area which has partly open areas, forest areas and many types of the buildings, in Zekeriyakoy, Istanbul using the TerraScan module of TerraSolid. The 3D building model was generated automatically using the results of the automatic point based classification. The obtained results of this research on study area verified that automatic 3D building models can be generated successfully using raw LiDAR point cloud data.

Urban building reconstruction from raw LiDAR point data

We present a method for automatic reconstruction of the volumetric structures of urban buildings, directly from raw LiDAR point clouds. Given the large-scale LiDAR data from a group of urban buildings, we take advantage of the “divide-and-conquer” strategy to decompose the entire point clouds into a number of subsets, each of which corresponds to an individual building. For each urban building, we determine its upward direction and partition the corresponding point data into a series of consecutive blocks, achieved by investigating the distributions of feature points of the building along the upward direction. Next, we propose a novel algorithm, Spectral Residual Clustering (SRC), to extract the primitive elements within the contours of blocks from the sectional point set, which is formed by registering the series of consecutive slicing points. Subsequently, we detect the geometric constraints among primitive elements through individual fitting, and perform constrained fitting over all primitive elements to obtain the accurate contour. On this basis, we execute 3D modeling operations, like extrusion, lofting or sweeping, to generate the 3D models of blocks. The final accurate 3D models are generated by applying the union Boolean operations over the block models. We evaluate our reconstruction method on a variety of raw LiDAR scans to verify its robustness and effectiveness.

Multiple Kernel Sparse Representation for Airborne LiDAR Data Classification

To effectively learn heterogeneous features extracted from raw LiDAR point cloud data for landcover classification, a multiple kernel sparse representation classification (MKSRC) framework is proposed in this paper. In the MKSRC, multiple kernel learning (MKL) is embedded into sparse representation classification (SRC). The heterogeneous features are first extracted from the raw LiDAR point cloud data before classification. These features contain useful information from different dimensions, including single point features and neighbor features. Based on feature extraction, on the one hand, MKL is reasonably integrated into the SRC, namely, different base kernels that are constructed with each heterogeneous feature separately are utilized in the process of sparse representation. Furthermore, joint sparsity model is also introduced into the MKSRC framework and multiple kernel joint SRC (MKJSRC) is then proposed. On the other hand, improved kernel alignment (IKA) methods are proposed to more effectively determine the weights of base kernels in both of MKSRC and MKJSRC. Experiments are conducted on three real airborne LiDAR data sets. The experimental results demonstrate that MKSRC and MKJSRC frameworks can effectively learn the heterogeneous features for LiDAR point cloud classification and outperforms the other state-of-the-art sparse representation-based classifiers and the recent MKL algorithm. Moreover, the proposed IKA is helpful to better determine the “optimal” weights of the base kernels in both MKSRC and MKJSRC than in the existing kernel alignment method.

A semi-automated method to create a lidar-based hydro-flattened DEM

ABSTRACTThe response of water surfaces to light detection and ranging (lidar) pulses is unpredictable, which results in sparse lidar point density with varying intensity values. Due to the sparseness of the point cloud and lack of natural breaklines, lidar-derived digital elevation model (DEM) can produce unnatural surface over waterbodies. Such surfaces are not cartographically pleasing and can cause issues in the hydrologic and hydraulic modelling of a river. Hydro-flattening is the process of creating a lidar-derived DEM in which water surfaces appear and behave as they would in traditional topographic DEMs generated from photogrammetric digital terrain models. Hydro-flattened DEMs, created using lidar data, exclude the lidar points over waterbodies and include three-dimensional (3D) bank shorelines. In this article, a semi-automated method is presented for extracting bank shorelines for the purpose of creating lidar-derived hydro-flatten DEMs. Lidar point cloud and an approximate stream centreline are the primary data for this process. In the first step, a continuous bare ground surface (CBGS) is created by eliminating non-ground lidar points and by adding artificial underwater points. In the second step, the lowest elevation from the lidar point cloud within a radius distance from the river centreline is used to create a virtual water surface (VWS). This VWS is revised to consider water surface undulations such as ripples or waves, protruding underwater objects, etc. The revised VWS is then intersected with the CBGS to locate the two-dimensional (2D) bank shorelines. The 2D shorelines are assigned the elevations of the VWS and are used to produce a hydro-flattened DEM. The planimetric absolute mean separation of 0.94, 0.69, and 0.63 m for the three water surfaces is observed between the bank shoreline extracted using raw lidar points and a GPS (global positioning system) survey. The mean separation using vendor classified lidar points is 0.74, 0.67, and 0.64 m which are very similar to those using raw lidar.

Automatic Modeling of Urban Facades from Raw LiDAR Point Data

AbstractModeling of urban facades from raw LiDAR point data remains active due to its challenging nature. In this paper, we propose an automatic yet robust 3D modeling approach for urban facades with raw LiDAR point clouds. The key observation is that building facades often exhibit repetitions and regularities. We hereby formulate repetition detection as an energy optimization problem with a global energy function balancing geometric errors, regularity and complexity of facade structures. As a result, repetitive structures are extracted robustly even in the presence of noise and missing data. By registering repetitive structures, missing regions are completed and thus the associated point data of structures are well consolidated. Subsequently, we detect the potential design intents (i.e., geometric constraints) within structures and perform constrained fitting to obtain the precise structure models. Furthermore, we apply structure alignment optimization to enforce position regularities and employ repetitions to infer missing structures. We demonstrate how the quality of raw LiDAR data can be improved by exploiting data redundancy, and discovering high level structural information (regularity and symmetry). We evaluate our modeling method on a variety of raw LiDAR scans to verify its robustness and effectiveness.

Automatic Modeling of Urban Facades from Raw LiDAR Point Data

Modeling of urban facades from raw LiDAR point data remains active due to its challenging nature. In this paper, we propose an automatic yet robust 3D modeling approach for urban facades with raw L...

CLASSIFICATION OF LiDAR DATA WITH POINT BASED CLASSIFICATION METHODS

Abstract. LiDAR is one of the most effective systems for 3 dimensional (3D) data collection in wide areas. Nowadays, airborne LiDAR data is used frequently in various applications such as object extraction, 3D modelling, change detection and revision of maps with increasing point density and accuracy. The classification of the LiDAR points is the first step of LiDAR data processing chain and should be handled in proper way since the 3D city modelling, building extraction, DEM generation, etc. applications directly use the classified point clouds. The different classification methods can be seen in recent researches and most of researches work with the gridded LiDAR point cloud. In grid based data processing of the LiDAR data, the characteristic point loss in the LiDAR point cloud especially vegetation and buildings or losing height accuracy during the interpolation stage are inevitable. In this case, the possible solution is the use of the raw point cloud data for classification to avoid data and accuracy loss in gridding process. In this study, the point based classification possibilities of the LiDAR point cloud is investigated to obtain more accurate classes. The automatic point based approaches, which are based on hierarchical rules, have been proposed to achieve ground, building and vegetation classes using the raw LiDAR point cloud data. In proposed approaches, every single LiDAR point is analyzed according to their features such as height, multi-return, etc. then automatically assigned to the class which they belong to. The use of un-gridded point cloud in proposed point based classification process helped the determination of more realistic rule sets. The detailed parameter analyses have been performed to obtain the most appropriate parameters in the rule sets to achieve accurate classes. The hierarchical rule sets were created for proposed Approach 1 (using selected spatial-based and echo-based features) and Approach 2 (using only selected spatial-based features) and have been tested in the study area in Zekeriyaköy, Istanbul which includes the partly open areas, forest areas and many types of the buildings. The data set used in this research obtained from Istanbul Metropolitan Municipality which was collected with ‘Riegl LSM-Q680i’ full-waveform laser scanner with the density of 16 points/m2. The proposed automatic point based Approach 1 and Approach 2 classifications successfully produced the ground, building and vegetation classes which were very similar although different features were used.