Lidar Fusion Research Articles

Object Tracking Based on the Fusion of Roadside LiDAR and Camera Data

Tracking road users with high resolution is important for connected vehicles. Due to the complicated environments, tracking objects with a single sensor could not meet the requirements of high-resolution trajectories due to occlusions. How to acquire accurate and complete trajectories based on multi-source data is a major challenge for researchers and engineers. This paper developed a novel tracking method based on the fusion of roadside LiDAR and camera. According to the relationship between the number of points and distance, the adaptive weight coefficient related to 3D trajectory information was determined. The performance of the proposed method was evaluated at five selected sites. The proposed tracking method had high performance in terms of speed calculation, tracking range, the rate of object loss, and the repairing rate of disconnected trajectories. The proposed method can benefit many transportation areas, such as traffic volume counting, vehicle speed tracking, and traffic safety analysis.

LiCaNet: Further Enhancement of Joint Perception and Motion Prediction Based on Multi-Modal Fusion

The safety and reliability of autonomous driving pivots on the accuracy of perception and motion prediction pipelines, which reckons primarily on the sensors deployed onboard. Slight confusion in perception and motion prediction can result in catastrophic consequences due to misinterpretation in later pipelines. Therefore, researchers have recently devoted considerable effort towards enhancing perception and motion prediction models. However, targeting pixel-wise joint perception and motion prediction using different sensor modalities are often ignored. In this paper, we push performance even further by leveraging a multi-modal fusion network. We propose a novel LIDAR Camera Network (LiCaNet) that achieves accurate pixel-wise joint perception and motion prediction in real-time. LiCaNet expands on our earlier fusion network by incorporating a camera image into the fusion of LIDAR sourced sequential bird’s-eye view (BEV) and range view (RV) images. We present a comprehensive evaluation using nuScenes dataset to validate the outstanding performance of LiCaNet compared to the state-of-the-art. Experiments reveal that utilizing a camera sensor results in a substantial gain in perception and motion prediction. Moreover, most of the improvements achieved fall within the camera range, with the highest registered for small and distant objects, confirming the significance of incorporating a camera sensor into a fusion network.

基于改进EKF算法的电动叉车组合导航定位方法

Aiming at the problem of inaccurate navigation and positioning of electric forklifts in a complex environment with multiple placement racks when carrying stored crops in a warehouse, this paper proposes a combined navigation and positioning system based on information fusion of LiDAR and inertial measurement units. The method proposed in this paper improves the traditional EKF algorithm by introducing factors affecting the prior covariance matrix and changing the weights of processing old and new data in the filtering equation to achieve the desired goal of suppressing system dispersion and to accomplish accurate estimation of the position of electric forklifts in the storage room. The simulation of robot positioning and navigation in indoor environment shows that the improved algorithm improves the position estimation accuracy by about 30% compared with the traditional algorithm, the new algorithm can effectively improve the efficiency of electric forklift for handling and storage, and it can ensure the robustness of robot position estimation.

Surrounding Objects Detection and Tracking for Autonomous Driving Using LiDAR and Radar Fusion

Radar and LiDAR are two environmental sensors commonly used in autonomous vehicles, Lidars are accurate in determining objects’ positions but significantly less accurate as Radars on measuring their velocities. However, Radars relative to Lidars are more accurate on measuring objects velocities but less accurate on determining their positions as they have a lower spatial resolution. In order to compensate for the low detection accuracy, incomplete target attributes and poor environmental adaptability of single sensors such as Radar and LiDAR, in this paper, an effective method for high-precision detection and tracking of surrounding targets of autonomous vehicles. By employing the Unscented Kalman Filter, Radar and LiDAR information is effectively fused to achieve high-precision detection of the position and speed information of targets around the autonomous vehicle. Finally, the real vehicle test under various driving environment scenarios is carried out. The experimental results show that the proposed sensor fusion method can effectively detect and track the vehicle peripheral targets with high accuracy. Compared with a single sensor, it has obvious advantages and can improve the intelligence level of autonomous cars.

UAV low-altitude obstacle detection based on the fusion of LiDAR and camera

In this paper, aiming at the flying scene of the small unmanned aerial vehicle (UAV) in the low-altitude suburban environment, we choose the sensor configuration scheme of LiDAR and visible light camera, and design the static and dynamic obstacle detection algorithms based on sensor fusion. For static obstacles such as power lines and buildings in the low-altitude environment, the way that image-assisted verification of point clouds is used to fuse the contour information of the images and the depth information of the point clouds to obtain the location and size of static obstacles. For unknown dynamic obstacles such as rotary-wing UAVs, the IMM-UKF algorithm is designed to fuse the distance measurement information of point clouds and the high precision angle measurement information of image to achieve accurate estimation of the location and velocity of the dynamic obstacles. We build an experimental platform to verify the effectiveness of the obstacle detection algorithm in actual scenes and evaluate the relevant performance indexes.

A spatially enhanced network with camera-lidar fusion for 3D semantic segmentation

In autonomous vehicle technology, environmental perception is a very important part of the whole system. The robustness and accuracy of the perception determine the performance of the whole system. Compared with the camera, lidar can generate point cloud data which can provide accurate 3D information better. At present, more researches focus on environmental perception through point cloud. However, the point cloud has the characteristics of sparsity, unordered and non-uniform distribution, its processing method is also different from the general image-based method. In this paper, we proposed a method of semantic segmentation of point cloud which is realized by convolutional neural network. Through spherical mapping, the scattered point cloud is transformed into 2D spherical map with dense and uniform distribution, which is convenient for the processing of the network. To reduce the loss of point information, the spatial position, reflection intensity information and angle value of point are preserved during spherical mapping. Our work includes a lightweight convolutional neural network that fuses point cloud and image information. In addition, a spatial module is added to supplement the loss of spatial information and improve the performance of segmentation. Finally, a series of comparative experiments and performance evaluation are carried out, which show that the proposed method is not only much better than the original structure, but also better than another image and lidar fusion network in the segmentation of each category.

A Convolutional Neural Network-Based End-to-End Self-Driving Using LiDAR and Camera Fusion: Analysis Perspectives in a Real-World Environment

In this paper, we develop end-to-end autonomous driving based on a 2D LiDAR sensor and camera sensor that predict the control value of the vehicle from the input data, instead of modeling rule-based autonomous driving. Different from many studies utilizing simulated data, we created an end-to-end autonomous driving algorithm with data obtained from real driving and analyzing the performance of our proposed algorithm. Based on the data obtained from an actual urban driving environment, end-to-end autonomous driving was possible in an informal environment such as a traffic signal by predicting the vehicle control value based on a convolution neural network. In addition, this paper solves the data imbalance problem by eliminating redundant data for each frame during stopping and driving in the driving environment so we can improve the performance of self-driving. Finally, we verified through the activation map how the network predicts the vertical and horizontal control values by recognizing the traffic facilities in the driving environment. Experiments and analysis will be shown to show the validity of the proposed algorithm.

Multiple Road-Objects Detection and Tracking for Autonomous Driving

In this paper, a real-time road-Object Detection and Tracking (LR_ODT) method for autonomous driving is proposed. The method is based on the fusion of lidar and radar measurement data, where they are installed on the ego car, and a customized Unscented Kalman Filter (UKF) is employed for their data fusion. The merits of both devices are combined using the proposed fusion approach to precisely provide both pose and velocity information for objects moving in roads around the ego car. Unlike other detection and tracking approaches, the balanced treatment of both pose estimation accuracy and its real-time performance is the main contribution in this work. The proposed technique is implemented using the high-performance language C++ and utilizes highly optimized math and optimization libraries for best real-time performance. Simulation studies have been carried out to evaluate the performance of the LR_ODT for tracking bicycles, cars, and pedestrians. Moreover, the performance of the UKF fusion is compared to that of the Extended Kalman Filter fusion (EKF) showing its superiority. The UKF has outperformed the EKF on all test cases and all the state variable levels (-24% average RMSE). The employed fusion technique show how outstanding is the improvement in tracking performance compared to the use of a single device (-29% RMES with lidar and -38% RMSE with radar).

Bayesian Localization in Real-Time using Probabilistic Maps and Unscented-Kalman-Filters

In this paper, based on the fusion of Lidar and Radar measurement data, high-definition probabilistic maps, and a tailored particle filter, a Real-Time Monte Carlo Localization (RT_MCL) method for autonomous cars is proposed. The lidar and radar devices are installed on the ego car, and a customized Unscented Kalman Filter (UKF) is used for their data fusion. Lidars are accurate in determining objects' positions and have a much higher spatial resolution. On the other hand, Radars are more accurate in measuring objects velocities and perform well in extreme weather conditions. Therefore, the merits of both sensors are combined using the UKF to provide pole-like static-objects pose estimations that are well suited to serve as landmarks for vehicle localization in urban environments. These pose estimations are then clustered using the Grid-Based Density-Based Spatial Clustering of Applications with Noise (GB-DBSCAN) algorithm to represent each pole landmarks in the form of a source-point model to reduce computational cost and memory requirements. A reference map that includes pole landmarks is generated off-line and extracted from a 3-D lidar to be used by a carefully designed Particle Filter (PF) for accurate ego-car localization. The particle filter is initialized by the combined GPS+IMU reading and used an ego-car motion model to predict the states of the particles. The data association between the estimated landmarks by the UKF and that in the reference map is performed using Iterative Closest Point (ICP) algorithm. The proposed pipeline is implemented using the high-performance language C++ and utilizes highly optimized math and optimization libraries for best real-time performance. Extensive simulation studies have been carried out to evaluate the performance of the RT_MCL in both longitudinal and lateral localization.

DV-LOAM: Direct Visual LiDAR Odometry and Mapping

Self-driving cars have experienced rapid development in the past few years, and Simultaneous Localization and Mapping (SLAM) is considered to be their basic capabilities. In this article, we propose a direct vision LiDAR fusion SLAM framework that consists of three modules. Firstly, a two-staged direct visual odometry module, which consists of a frame-to-frame tracking step, and an improved sliding window based thinning step, is proposed to estimate the accurate pose of the camera while maintaining efficiency. Secondly, every time a keyframe is generated, a dynamic objects considered LiDAR mapping module is utilized to refine the pose of the keyframe to obtain higher positioning accuracy and better robustness. Finally, a Parallel Global and Local Search Loop Closure Detection (PGLS-LCD) module that combines visual Bag of Words (BoW) and LiDAR-Iris feature is applied for place recognition to correct the accumulated drift and maintain a globally consistent map. We conducted a large number of experiments on the public dataset and our mobile robot dataset to verify the effectiveness of each module in our framework. Experimental results show that the proposed algorithm achieves more accurate pose estimation than the state-of-the-art methods.

Application of airborne LiDAR and UAV image fusion for complicated terrain

Application of airborne LiDAR and UAV image fusion for complicated terrain

P‐17.3: Obstacle detection based on the fusion of multiple solid‐state lidars

In automatic driving, mechanical lidar is often used to detect the surrounding environment. Due to cost‐effective reasons, solidstate lidar is now also a good choice. It can obtain a larger number of point clouds. But this means that there are more point clouds that need to be processed, and the requirements for real‐time are more stringent. At the same time, when the vehicle is moving, it is difficult to segment obstacles, which is prone to missed detection and false detection. exist. To solve the above problems, this paper adopts the detection method of multi‐solidstate lidar fusion, and proposes a RANSAC ground detection algorithm with adaptive iteration times. At the same time, the Euclidean clustering method is improved to make it based on the change of the distance radius detected by the point cloud. The threshold is automatically corrected, so that the automatic driving can detect the surrounding environment quickly and accurately. The method in this paper has also been put into actual vehicle experiments, and the test results show that the method in this paper can indeed achieve the expected goal.

Monitoring restored tropical forest diversity and structure through UAV-borne hyperspectral and lidar fusion

Remote sensors, onboard orbital platforms, aircraft, or unmanned aerial vehicles (UAVs) have emerged as a promising technology to enhance our understanding of changes in ecosystem composition, structure, and function of forests, offering multi-scale monitoring of forest restoration. UAV systems can generate high-resolution images that provide accurate information on forest ecosystems to aid decision-making in restoration projects. However, UAV technological advances have outpaced practical application; thus, we explored combining UAV-borne lidar and hyperspectral data to evaluate the diversity and structure of restoration plantings. We developed novel analytical approaches to assess twelve 13-year-old restoration plots experimentally established with 20, 60 or 120 native tree species in the Brazilian Atlantic Forest. We assessed (1) the congruence and complementarity of lidar and hyperspectral-derived variables, (2) their ability to distinguish tree richness levels and (3) their ability to predict aboveground biomass (AGB). We analyzed three structural attributes derived from lidar data—canopy height, leaf area index (LAI), and understory LAI—and eighteen variables derived from hyperspectral data—15 vegetation indices (VIs), two components of the minimum noise fraction (related to spectral composition) and the spectral angle (related to spectral variability). We found that VIs were positively correlated with LAI for low LAI values, but stabilized for LAI greater than 2 m2/m2. LAI and structural VIs increased with increasing species richness, and hyperspectral variability was significantly related to species richness. While lidar-derived canopy height better predicted AGB than hyperspectral-derived VIs, it was the fusion of UAV-borne hyperspectral and lidar data that allowed effective co-monitoring of both forest structural attributes and tree diversity in restoration plantings. Furthermore, considering lidar and hyperspectral data together more broadly supported the expectations of biodiversity theory, showing that diversity enhanced biomass capture and canopy functional attributes in restoration. The use of UAV-borne remote sensors can play an essential role during the UN Decade of Ecosystem Restoration, which requires detailed forest monitoring on an unprecedented scale.

Lidar-Camera Semi-Supervised Learning for Semantic Segmentation.

In this work, we investigated two issues: (1) How the fusion of lidar and camera data can improve semantic segmentation performance compared with the individual sensor modalities in a supervised learning context; and (2) How fusion can also be leveraged for semi-supervised learning in order to further improve performance and to adapt to new domains without requiring any additional labelled data. A comparative study was carried out by providing an experimental evaluation on networks trained in different setups using various scenarios from sunny days to rainy night scenes. The networks were tested for challenging, and less common, scenarios where cameras or lidars individually would not provide a reliable prediction. Our results suggest that semi-supervised learning and fusion techniques increase the overall performance of the network in challenging scenarios using less data annotations.

Lidar and radar fusion for real-time road-objects detection and tracking

In this paper, based on the fusion of Lidar and Radar measurement data, a real-time road-Object Detection and Tracking (LR_ODT) method for autonomous driving is proposed. The lidar and radar devices are installed on the ego car, and a customized Unscented Kalman Filter (UKF) is used for their data fusion. Lidars are accurate in determining objects’ positions but significantly less accurate on measuring their velocities. However, Radars are more accurate on measuring objects velocities but less accurate on determining their positions as they have a lower spatial resolution. Therefore, the merits of both sensors are combined using the proposed fusion approach to provide both pose and velocity data for objects moving in roads precisely. The Grid-Based Density-Based Spatial Clustering of Applications with Noise (GB-DBSCAN) clustering algorithm is used to detect objects and estimate their centroids from the lidar and radar raw data. Then, the estimation of the object’s velocity as well as determining its corresponding geometrical shape is performed by the RANdom SAmple Consensus (RANSAC) algorithm. The proposed technique is implemented using the high-performance language C+⁣+ and utilizes highly optimized math and optimization libraries for best real-time performance. The performance of the UKF fusion is compared to that of the Extended Kalman Filter fusion (EKF) showing its superiority. Simulation studies have been carried out to evaluate the performance of the LR_ODT for tracking bicycles, cars, and pedestrians.

Obstacle detection based on depth fusion of lidar and radar in challenging conditions

PurposeTo the industrial application of intelligent and connected vehicles (ICVs), the robustness and accuracy of environmental perception are critical in challenging conditions. However, the accuracy of perception is closely related to the performance of sensors configured on the vehicle. To enhance sensors’ performance further to improve the accuracy of environmental perception, this paper aims to introduce an obstacle detection method based on the depth fusion of lidar and radar in challenging conditions, which could reduce the false rate resulting from sensors’ misdetection.Design/methodology/approachFirstly, a multi-layer self-calibration method is proposed based on the spatial and temporal relationships. Next, a depth fusion model is proposed to improve the performance of obstacle detection in challenging conditions. Finally, the study tests are carried out in challenging conditions, including straight unstructured road, unstructured road with rough surface and unstructured road with heavy dust or mist.FindingsThe experimental tests in challenging conditions demonstrate that the depth fusion model, comparing with the use of a single sensor, can filter out the false alarm of radar and point clouds of dust or mist received by lidar. So, the accuracy of objects detection is also improved under challenging conditions.Originality/valueA multi-layer self-calibration method is conducive to improve the accuracy of the calibration and reduce the workload of manual calibration. Next, a depth fusion model based on lidar and radar can effectively get high precision by way of filtering out the false alarm of radar and point clouds of dust or mist received by lidar, which could improve ICVs’ performance in challenging conditions.

Obstacles detection based on IMU and 3D lidar fusion

Obstacles detection based on IMU and 3D lidar fusion

Design of 2D LiDAR and camera fusion system improved by differential evolutionary PID with nonlinear tracking compensator

An improved 2D LiDAR and camera fusion system is proposed for the 3D reconstruction of unknown environments. It combines the advantages of dense 2D point cloud and rich color image, adopting a differential evolutionary nonlinear tracking PID to control the pitching motion of LiDAR and camera accurately. The quadratic polynomial transition function is used to optimize the pitching trajectory. The environment was scanned by the system and converted into a 3D colored point cloud by the data fusion algorithm. The experimental results show: the proposed PID control algorithm can accurately control the pitching motion with a small average error (0.0267°) and significantly reduce the point cloud inhomogeneity (0.00698); the processing time for converting each 2D point cloud into the 3D point cloud is about 0.6 ms; combined with the data fusion algorithm, the system can obtain the dense colored 3D point cloud; compared with binocular camera, depth camera and 3D LiDAR under the condition of strong light interference, the fusion system outperforms, with the reconstruction object errors of distance, length and width of 0.23%, 0.17% and 0.46% respectively. In conclusion, the system can obtain homogeneous, and dense colored 3D point cloud in real time while ensuring stable refresh frame rate.

Fast Closed-Loop SLAM based on the fusion of IMU and Lidar

In the large-scale mapping of Lidar SLAM, there are problems of slow closed-loop detection matching speed and false detection. To solve this problem, this paper proposed a method based on the fusion of IMU and Lidar. Use IMU geomagnetic data for geomagnetic matching and filter the pose nodes in the map to reduce the search space during closed-loop detection. At the same time, due to the reduction of the search space, it can effectively reduce the high local similarity in Lidar SLAM false detection caused. This experiment verified the method by collecting Lidar point cloud data and IMU three-axis geomagnetic data in a real environment, the closed-loop detection speed and map average absolute error of frame-by-frame method, branch-and-bound method, DTW fusion algorithm and Fast-DTW fusion algorithm were verified and compared. Experimental results shown that, compared with DTW fusion algorithm, the proposed algorithm improved the closed-loop detection speed by 17%, reduced the average absolute error of translation by 16%, and reduced the average absolute error of rotation by 10%.

Real-time Track Obstacle Detection from 3D LIDAR Point Cloud

Obstacle detection is one of the important works in the field of driverless vehicle. Camera, millimeter wave radar, LIDAR and other sensors are widely used in the obstacle detection of driverless vehicle. However, cameras and millimeter wave radar are highly dependent on the environment and greatly affected by external factors, especially in the process of high-speed driving, which is easy to produce serious system errors, and even unable to detect obstacles, which has a great potential for the safety of driverless vehicle. LIDAR is used to detect obstacles by launching laser, which is not easy to be interfered by the environment. It has high precision and is suitable for outdoor environment. The main contributions of this paper are as follows: 1) a 16 line LIDAR track recognition method based on driverless equation is proposed. 2) it is easy to get the obstacle coordinates through plane segmentation and clustering for the use of subsequent camera and LIDAR fusion.