Real LiDAR Research Articles

Three-dimensional spatiotemporal wind field reconstruction based on LiDAR and multi-scale PINN

In this numerical study, we use a multi-scale version of physics-informed neural network (PINN) for wind field reconstruction from LiDAR measurements. The reference velocity field is obtained from high-fidelity large-eddy simulation of neutral atmospheric boundary layer, and the LiDAR measurement strategy is restricted to that of a real LiDAR. The multi-scale PINN reconstructs the velocity field by minimizing a combination of the L2-error with respect to the LiDAR measurements and residuals of the governing equations. Our most significant finding is that adding a multi-scale layer to PINN enables us to: (i) capture wider range of scales, (ii) reconstruct the flow field outside the LiDAR scanning area, and (iii) reach faster convergence. We also find that for volumetric flow reconstruction and to deal with uncertainty in the wind direction, the reconstruction accuracy can be greatly improved by employing multiple LiDAR devices. Overall, our results show that the normalized errors in the reconstructed wind field are lower than 5% for all the experiments conducted in this study and that the errors do not increase even in the presence of measurement noise levels of typical LiDAR. These findings show the feasibility of three-dimensional dynamic wind field reconstruction across large wind farms using LiDAR devices.

Bridging the Domain Gap in Scene Flow Estimation via Hierarchical Smoothness Refinement

This article introduces SmoothFlowNet3D, an innovative encoder-decoder architecture specifically designed for bridging the domain gap in scene flow estimation. To achieve this goal, SmoothFlowNet3D divides the scene flow estimation task into two stages: initial scene flow estimation and smoothness refinement. Specifically, SmoothFlowNet3D comprises a hierarchical encoder that extracts multi-scale point cloud features from two consecutive frames, along with a hierarchical decoder responsible for predicting the initial scene flow and further refining it to achieve smoother estimation. To generate the initial scene flow, a cross-frame nearest-neighbor search operation is performed between the features extracted from two consecutive frames, resulting in forward and backward flow embeddings. These embeddings are then combined to form the bidirectional flow embedding, serving as input for predicting the initial scene flow. Additionally, a flow smoothing module based on the self-attention mechanism is proposed to predict the smoothing error and facilitate the refinement of the initial scene flow for more accurate and smoother estimation results. Extensive experiments demonstrate that the proposed SmoothFlowNet3D approach achieves state-of-the-art performance on both synthetic datasets and real LiDAR point clouds, confirming its effectiveness in enhancing scene flow smoothness.

Real-virtual Point Cloud Augmentation Method for Test and Evaluation of Autonomous Weapon Systems

Autonomous weapon systems act according to artificial intelligence-based judgement based on recognition through various sensors. Test and evaluation for various scenarios is required depending on the characteristics that artificial intelligence-based judgement is made. As a part of this approach, this paper proposed a LiDAR point cloud augmentation method for mixed-reality based test and evaluation. The augmentation process is achieved by mixing real and virtual LiDAR signals based on the virtual LiDAR synchronized with the pose of the autonomous weapon system. For realistic augmentation of test and evaluation purposes, appropriate intensity values were inserted when generating a point cloud of a virtual object and its validity was verified. In addition, when mixing the generated point cloud of the virtual object with the real point cloud, the proposed method enhances realism by considering the occlusion phenomenon caused by the insertion of the virtual object.

NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields

Labelling LiDAR point clouds for training autonomous driving is extremely expensive and difficult. LiDAR simulation aims at generating realistic LiDAR data with labels for training and verifying self-driving algorithms more efficiently. Recently, Neural Radiance Fields (NeRF) have been proposed for novel view synthesis using implicit reconstruction of 3D scenes. Inspired by this, we present NeRF-LIDAR, a novel LiDAR simulation method that leverages real-world information to generate realistic LIDAR point clouds. Different from existing LiDAR simulators, we use real images and point cloud data collected by self-driving cars to learn the 3D scene representation, point cloud generation and label rendering. We verify the effectiveness of our NeRF-LiDAR by training different 3D segmentation models on the generated LiDAR point clouds. It reveals that the trained models are able to achieve similar accuracy when compared with the same model trained on the real LiDAR data. Besides, the generated data is capable of boosting the accuracy through pre-training which helps reduce the requirements of the real labeled data. Code is available at https://github.com/fudan-zvg/NeRF-LiDAR

S2S-Sim: A Benchmark Dataset for Ship Cooperative 3D Object Detection

The rapid development of vehicle cooperative 3D object-detection technology has significantly improved the perception capabilities of autonomous driving systems. However, ship cooperative perception technology has received limited research attention compared to autonomous driving, primarily due to the lack of appropriate ship cooperative perception datasets. To address this gap, this paper proposes S2S-sim, a novel ship cooperative perception dataset. Ship navigation scenarios were constructed using Unity3D, and accurate ship models were incorporated while simulating sensor parameters of real LiDAR sensors to collect data. The dataset comprises three typical ship navigation scenarios, including ports, islands, and open waters, featuring common ship classes such as container ships, bulk carriers, and cruise ships. It consists of 7000 frames with 96,881 annotated ship bounding boxes. Leveraging this dataset, we assess the performance of mainstream vehicle cooperative perception models when transferred to ship cooperative perception scenes. Furthermore, considering the characteristics of ship navigation data, we propose a regional clustering fusion-based ship cooperative 3D object-detection method. Experimental results demonstrate that our approach achieves state-of-the-art performance in 3D ship object detection, indicating its suitability for ship cooperative perception.

Development and Calibration of 532 nm Standard Aerosol Lidar with Low Blind Area

To better calibrate the aerosol lidar network constructed by the China Meteorological Administration, and ensure the data quality observed by the network, the Meteorological Observation Center (China Meteorological Administration) and the University of Naples (Italy) jointly developed a “high quality 532 nm Raman aerosol lidar” (REAL lidar) in 2018. The ability to detect Raman–Mie scattering signals was improved through signal detection in a large dynamic range. This study compared the REAL lidar with the reference lidar (European ACTRIS aerosol lidar network) considering three wavelengths and eight channels. The results show that both the original signals and data products of the two radars exhibited good consistency. In the calibration application of China’s domestic lidar network, after REAL calibration, the relative average and standard deviations of the backscattering coefficient of the in-station lidar decreased from 55.4% to 7.9% and from 64% to 9.9%, respectively. The effect was significant, which indicates that REAL is an aerosol lidar with a high-performance index. The results satisfy the demand for calibration of the aerosol lidar network, and the REAL was successfully applied to the calibration of the aerosol lidar network.

ENTROPY-BASED INDOOR CHANGE DETECTION USING LIDAR DATA AND A 3D MODEL

Abstract. Indoor change detection is important for building monitoring, building management and model-based localization and navigation systems because the real building environment may not always be the same as the design model. This paper presents a novel indoor building change detection method based on entropy. A sequence of real LiDAR scans is acquired with a static LiDAR scanner and the pose of the LiDAR scanner for each scan is then estimated. Synthetic LiDAR scans are generated with the pose of the LiDAR scanner using the 3D model. The real LiDAR scans and synthetic LiDAR scans are sliced horizontally with a certain angular interval and the entropy of all slices of LiDAR scans is then calculated. Differenced entropy between two corresponding slices of real LiDAR scans and synthetic LiDAR scans is calculated for the classification of the changes. Each slice of real LiDAR scans will be classified into one of the four categories of changes: unchanged, moving objects, structural change and non-structural change. Experimental results show that unchanged slices and slices containing moving objects can be accurately detected, achieving 100% accuracy while non-structural and structural changes are detected with an accuracy of 98.5% and 86.3% respectively.

Comparing Multiple Extended Object Tracking with Point Based Multi Object Tracking for LiDAR in a Maritime Context

In this paper, we compare a recently developed multiple extended object tracking method with a point-based tracker for maritime applications by evaluating the methods on both real and simulated LiDAR data. Being able to track other vessels is a key part of maritime situational awareness, and multi-object tracking is well suited to this task. Traditionally, multi-object tracking uses the point approximation to simplify the target tracking problem, meaning that an object is assumed to only generate a single measurement. With recent advances in sensor technology, this approximation is no longer valid. Multiple extended object tracking instead makes use of methods that estimates the extent of a target using all available measurements. However, the target models used in extended object tracking are by necessity more complex. We compare these two approaches on real and simulated LiDAR data, which was gathered by tracking smaller vessels. We find that the extended object method has greater performance on simulated data and experimental data with comparatively little wake clutter. However, on experimental data with more wake clutter, the point-based method outperforms the extended object method since it is less sensitive to the disturbance caused by wake clutter. We can conclude that multiple extended object tracking methods have the potential to be more accurate for LiDAR data, but since the models are more complex, measurement sources need to be modeled correctly.

Cascade Graph Neural Networks for Few-Shot Learning on Point Clouds

Point cloud data, a flexible 3D object representation, is critical for various applications such as autonomous driving, robotics and remote sensing. Despite the recent success of deep neural networks (DNNs) on supervised point cloud analysis tasks, they still rely on tedious manual annotation of point clouds and cannot make predictions for new classes. Unlike few-shot learning for 2D images with the advantages of large-scale datasets and high-quality deep pre-trained models like ResNet, for 3D few-shot learning, obtaining discriminative representations of unseen classes with high intra-class similarity and inter-class difference is very challenging. To address this issue, this work proposes a novel cascade graph neural network for few-shot learning on point clouds, termed as CGNN, in which two cascade GNNs are adopted to extract the intra-object topological information and learn the inter-object relations respectively. To further increase the discriminability of point cloud features, we first design a novel discriminative edge label to model the intra-class similarity and inter-class dissimilarity based on channel-wise feature variance and class consistency. Second, we propose a novel few-shot circle loss which classifies the nodes into two subsets, i.e., support to support pairs and support to query pairs, and optimizes the pair-wise similarity on two subsets independently. Extensive experiments on benchmark CAD and real LiDAR point cloud datasets have demonstrated that CGNN improves accuracy by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.98\%$</tex-math> </inline-formula> over the state-of-the-art GNN-based few-shot classification methods.

Urban Forest Above-Ground Biomass Estimation Based on UAV 3D Real Scene

(1) Background: A three-dimensional (3D) real scene is a digital representation of the multidimensional dynamic real-world structure that enables the realistic and stereoscopic expression of actual scenarios, and is an important technological tool for urban refinement management. The above-ground biomass (AGB) of urban forests is an important indicator that reflects the urban ecological environment; therefore, the accurate estimation of AGB is of great significance for evaluating urban ecological functions. (2) Methods: In this study, multiangle aerial photographs of urban street trees were obtained via an unmanned aerial vehicle (UAV) single-lens five-way flight, from 0°, 0°, 90°, 180°, 270°, and five other directions. The multiple view stereo (MVS) algorithm was used to construct three-dimensional realistic models of two tree species: ginkgo and camphor. Then, structural parameters such as tree height, crown diameter, and crown volume were estimated from the 3D real-scene models. Lastly, single-tree AGB models were developed based on structural parameters. (3) Results: The results of this study indicated the following: (A) The UAV visible-light realistic 3D model had clear texture and truly reflected the structural characteristics of two tree species, ginkgo and camphor. (B) There was a significant correlation between the reference tree height, crown diameter and crown volume obtained from the realistic 3D model and the measured values; the R2 for ginkgo height was 0.90, the R2 for camphor crown diameter was 0.87, and the R2 for ginkgo crown volume was 0.89. (C) The accuracy of the AGB estimation models constructed with tree height and canopy volume as variables was generally higher than that of models with tree height and canopy diameter; the model with the highest accuracy of AGB estimation for ginkgo was the linear model with a validation accuracy R2 of 0.96 and RMSE of 8.21 kg, while the model with the highest accuracy of AGB estimation for camphor was the quadratic polynomial model with a validation accuracy R2 of 0.92 and RMSE of 27.74 kg. (4) Conclusions: This study demonstrated that the UAV 3D real-scene model can achieve high accuracy in estimating single-wood biomass in urban forests. In addition, for both tree species, there was no significant difference between the AGB estimates based on the UAV 3D real scene and LiDAR and the measured AGB. These results of urban single-wood AGB estimation based on the UAV 3D real-scene model were consistent with those of LiDAR and even with the measured AGB. Therefore, based on the UAV 3D real-scene model, the single-wood biomass can be estimated with high accuracy. This represents a new technical approach to urban forest resource monitoring and ecological environment function evaluation.

E-LOAM: LiDAR Odometry and Mapping With Expanded Local Structural Information

This paper investigates the real time LiDAR odometry and mapping (LOAM) problem in unstructured environments. We propose E-LOAM (LOAM with Expanded Local Structural Information), a paradigm which expands the pre-extracted geometric information with local point cloud information around the geometric feature points. State-of-the-art approaches usually extract pointed geometric features as the only correspondence primitives for point cloud scan-to-scan and scan-to-map registration. We argue that, in unstructured environments, sometimes, the extracted geometric features are too sparse for adequate point cloud registration. Therefore, E-LOAM expands the ‘pointed’ geometric correspondence primitives with the point clouds around them, <i>i.e.,</i> the local point clouds in the voxel around the feature point. The local point clouds, approximated by a multivariate normal distribution, offer additional local structural information, on top of the <i>pointed </i>geometric information. Additionally, to enrich the sparse geometric features, we make use of the intensity information of point clouds, and extract the places with high intensity variations as additional feature points. Experimental results with KITTI dataset show the efficacy of E-LOAM, when compared with state of the arts. We further implement E-LOAM on a real robot platform, and evaluate E-LOAM with in-field tests.

Concept of an Automotive LiDAR Target Simulator for Direct Time-of-Flight LiDAR

Autonomous driving can make traffic safer by reducing human errors. Different sensor types in autonomous vehicles could introduce additional technical failures. We offer a target simulator testing LiDAR systems under automotive conditions. Therefore, data are projected over-the-air by laser signals on the LiDAR detector. This work presents a concept of a LiDAR target simulator with regards to LiDAR systems using the direct time-of-flight principle. We develope design considerations for a screen discussing undesired screen reflections, a curved screen form and the positioning of light sources on this screen. As one main solution, we introduce a concept of an antireflective screen. For the scenario simulation, we derive a model delivering the required optical power representing a simulated target, which is combined with the simulated time-of-flight. Considering no prior knowledge of the LiDAR system under test, we discuss the required calibration data and timing resolution. Thereby, we suggest an optimized time-of-flight concept requiring only one optical trigger to determine the LiDAR measurement start. All requirements are supported by calculated example parameters based on real LiDAR systems. Altogether, we discuss main challenges and possible solutions of our LiDAR target simulator, which will allow a safer and more efficient development of LiDAR systems.

Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Depth completion aims to predict a dense depth map from a sparse depth input. The acquisition of dense ground-truth annotations for depth completion settings can be difficult and, at the same time, a significant domain gap between real LiDAR measurements and synthetic data has prevented from successful training of models in virtual settings. We propose a domain adaptation approach for sparse-to-dense depth completion that is trained from synthetic data, without annotations in the real domain or additional sensors. Our approach simulates the real sensor noise in an RGB + LiDAR set-up, and consists of three modules: simulating the real LiDAR input in the synthetic domain via projections, filtering the real noisy LiDAR for supervision and adapting the synthetic RGB image using a CycleGAN approach. We extensively evaluate these modules in the KITTI depth completion benchmark.

Mitigating Shadows in LIDAR Scan Matching Using Spherical Voxels

In this paper we propose an approach to mitigate shadowing errors in Lidar scan matching, by introducing a preprocessing step based on spherical gridding. Because the grid aligns with the Lidar beam, it is relatively easy to eliminate shadow edges which cause systematic errors in Lidar scan matching. As we show through simulation, our proposed algorithm provides better results than ground-plane removal, the most common existing strategy for shadow mitigation. Unlike ground plane removal, our method applies to arbitrary terrains (e.g. shadows on urban walls, shadows in hilly terrain) while retaining key Lidar points on the ground that are critical for estimating changes in height, pitch, and roll. Our preprocessing algorithm can be used with a range of scan-matching methods; however, for voxel-based scan matching methods, it provides additional benefits by reducing computation costs and more evenly distributing Lidar points among voxels.

Transfer Learning from Synthetic to Real LiDAR Point Cloud for Semantic Segmentation

Knowledge transfer from synthetic to real data has been widely studied to mitigate data annotation constraints in various computer vision tasks such as semantic segmentation. However, the study focused on 2D images and its counterpart in 3D point clouds segmentation lags far behind due to the lack of large-scale synthetic datasets and effective transfer methods. We address this issue by collecting SynLiDAR, a large-scale synthetic LiDAR dataset that contains point-wise annotated point clouds with accurate geometric shapes and comprehensive semantic classes. SynLiDAR was collected from multiple virtual environments with rich scenes and layouts which consists of over 19 billion points of 32 semantic classes. In addition, we design PCT, a novel point cloud translator that effectively mitigates the gap between synthetic and real point clouds. Specifically, we decompose the synthetic-to-real gap into an appearance component and a sparsity component and handle them separately which improves the point cloud translation greatly. We conducted extensive experiments over three transfer learning setups including data augmentation, semi-supervised domain adaptation and unsupervised domain adaptation. Extensive experiments show that SynLiDAR provides a high-quality data source for studying 3D transfer and the proposed PCT achieves superior point cloud translation consistently across the three setups. The dataset is available at https://github.com/xiaoaoran/SynLiDAR.

Scalable Scene Flow From Point Clouds in the Real World

Autonomous vehicles operate in highly dynamic environments necessitating an accurate assessment of which aspects of a scene are moving and where they are moving to. A popular approach to 3D motion estimation, termed scene flow, is to employ 3D point cloud data from consecutive LiDAR scans, although such approaches have been limited by the small size of real-world, annotated LiDAR data. In this work, we introduce a new large-scale dataset for scene flow estimation derived from corresponding tracked 3D objects, which is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim 1,000\times$</tex-math></inline-formula> larger than previous real-world datasets in terms of the number of annotated frames. We demonstrate how previous works were bounded by the amount of real LiDAR data available, suggesting that larger datasets are required to achieve state-of-the-art predictive performance. Furthermore, we show how previous heuristics such as down-sampling heavily degrade performance, motivating a new class of models that are tractable on the full point cloud. To address this issue, we introduce the FastFlow3D architecture which provides real time inference on the full point cloud. Additionally, we design human-interpretable metrics that better capture real world aspects by accounting for ego-motion and providing breakdowns per object type. We hope that this dataset may provide new opportunities for developing real world scene flow systems.

A Deep Learning-Based Novel Approach for Weed Growth Estimation

Automation of agricultural food production is growing in popularity in scientific communities and industry. The main goal of automation is to identify and detect weeds in the crop. Weed intervention for the duration of crop establishment is a serious difficulty for wheat in North India. The soil nutrient is important for crop production. Weeds usually compete for light, water and air of nutrients and space from the target crop. This research paper assesses the growth rate of weeds due to macronutrients (nitrogen, phosphorus and potassium) absorbed from various soils (fertile, clay and loamy) in the rabi crop field. The weed image data have been collected from three different places in Madhya Pradesh, India with 10 different rabi crops (Maize, Lucerne, Cumin, Coriander, Wheat, Fenugreek, Gram, Onion, Mustard and Tomato) and 10 different weeds (Corchorus Capsularis, Cynodondactylon, Chloris barbata, Amaranthaceae, Argemone mexicana, Carthamus oxyacantha, Capsella bursa Pastoris, Chenopodium Album, Dactyloctenium aegyptium and Convolvulus Ravens). Intel Real Sense LiDAR digital camera L515 and Canon digital SLR DIGICAM EOS 850 D 18-55IS STM cameras were mounted over the wheat crop in 10 × 10 square feet area of land and 3670 different weed images have been collected. The 2936 weed images were used for training and 734 images for testing and validation. The Efficient Net-B7 and Inception V4 architectures have been used to train the model that has provided accuracy of 97% and 94% respectively. The Image classification using Inspection V4 was unsuccessful with less accurate results as compared to EfficientNet-B7.

Multisensory Testing Framework for Advanced Driver Assistant Systems Supported by High-Quality 3D Simulation.

ADAS and autonomous technologies in vehicles become more and more complex, which increases development time and expenses. This paper presents a new real-time ADAS multisensory validation system, which can speed up the development and implementation processes while lowering its cost. The proposed test system integrates a high-quality 3D CARLA simulator with a real-time-based automation platform. We present system experimental verifications on several types of sensors and testing system architectures. The first, open-loop experiment explains the real-time capabilities of the system based on the Mobileye 6 camera sensor detections. The second experiment runs a real-time closed-loop test of a lane-keeping algorithm (LKA) based on the Mobileye 6 line detection. The last experiment presents a simulation of Velodyne VLP-16 lidar, which runs a free space detection algorithm. Simulated lidar output is compared with the real lidar performance. We show that the platform generates reproducible results and allows closed-loop operation which, combined with a real-time collection of event information, promises good scalability toward complex ADAS or autonomous functionalities testing.

Enhancement method of weak Lidar signal based on adaptive variational modal decomposition and wavelet threshold denoising

Aiming to identify and detect the weak signal of lidar, a cascade method of adaptive variational modal decomposition (VMD) and wavelet threshold (WT) denoising is proposed in this paper. Firstly, VMD is used to decompose the echo signal, and the criterion of adaptive mode selection is improved. Furthermore, the “db4” wavelet and soft threshold denoising method are used to process the denoised signal by AVMD. The optimal threshold value is obtained by minimizing Stein's unbiased risk estimator. In order to verify the superity of the proposed method, Bumps signal is employed in simulation. When the input signal-to-noise ratio (SNR) is −3dB, the SNR of proposed method can reach 7.8587 dB. Compared with three methods, the SNR of proposed method is the highest under different input SNRs. The real Lidar signal is recorded to be denoised. The root mean square errors of WT-db4, EMD-DT, SVD-WT and the proposed method are 0.0046, 0.0044, 0.0044 and 0.0043. Compared with three state-of-art denoising methods, the proposed denoising method can effectively eliminate the spike noise and fully retain the echo signal.

DepthLiDAR: Active Segmentation of Environment Depth Map Into Mobile Sensors

This paper presents a novel approach for creating virtual LiDAR scanners through the active segmentation of point clouds. The method employs top-view point cloud segmentation in virtual LiDAR sensors that can be applied to the intelligent behavior of autonomous agents. Segmentation is correlated with the visual tracking of the agent for localization in the environment and point cloud. Virtual LiDAR sensors with different characteristics and positions can then be generated. This method is referred to as the DepthLiDAR approach, and is rigorously evaluated to quantify its performance and determine its advantages and limitations. An extensive set of experiments is conducted using real and virtual LiDAR sensors to compare both approaches. The objective is to propose a novel method to incorporate spatial perception in warehouses, aiming to achieve Industry 4.0. Thus, it is tested in a low-scale warehouse to incorporate realistic features. The analysis of the experiments shows a measurement improvement of 52.24% compared to the conventional LiDAR.