Fusion Of Light Detection Research Articles

A Spatiotemporal Calibration Algorithm for IMU-LiDAR Navigation System Based on Similarity of Motion Trajectories.

The fusion of light detection and ranging (LiDAR) and inertial measurement unit (IMU) sensing information can effectively improve the environment modeling and localization accuracy of navigation systems. To realize the spatiotemporal unification of data collected by the IMU and the LiDAR, a two-step spatiotemporal calibration method combining coarse and fine is proposed. The method mainly includes two aspects: (1) Modeling continuous-time trajectories of IMU attitude motion using B-spline basis functions; the motion of the LiDAR is estimated by using the normal distributions transform (NDT) point cloud registration algorithm, taking the Hausdorff distance between the local trajectories as the cost function and combining it with the hand–eye calibration method to solve the initial value of the spatiotemporal relationship between the two sensors’ coordinate systems, and then using the measurement data of the IMU to correct the LiDAR distortion. (2) According to the IMU preintegration, and the point, line, and plane features of the lidar point cloud, the corresponding nonlinear optimization objective function is constructed. Combined with the corrected LiDAR data and the initial value of the spatiotemporal calibration of the coordinate systems, the target is optimized under the nonlinear graph optimization framework. The rationality, accuracy, and robustness of the proposed algorithm are verified by simulation analysis and actual test experiments. The results show that the accuracy of the proposed algorithm in the spatial coordinate system relationship calibration was better than 0.08° (3δ) and 5 mm (3δ), respectively, and the time deviation calibration accuracy was better than 0.1 ms and had strong environmental adaptability. This can meet the high-precision calibration requirements of multisensor spatiotemporal parameters of field robot navigation systems.

LiDAR and Camera Fusion Approach for Object Distance Estimation in Self-Driving Vehicles

The fusion of light detection and ranging (LiDAR) and camera data in real-time is known to be a crucial process in many applications, such as in autonomous driving, industrial automation, and robotics. Especially in the case of autonomous vehicles, the efficient fusion of data from these two types of sensors is important to enabling the depth of objects as well as the detection of objects at short and long distances. As both the sensors are capable of capturing the different attributes of the environment simultaneously, the integration of those attributes with an efficient fusion approach greatly benefits the reliable and consistent perception of the environment. This paper presents a method to estimate the distance (depth) between a self-driving car and other vehicles, objects, and signboards on its path using the accurate fusion approach. Based on the geometrical transformation and projection, low-level sensor fusion was performed between a camera and LiDAR using a 3D marker. Further, the fusion information is utilized to estimate the distance of objects detected by the RefineDet detector. Finally, the accuracy and performance of the sensor fusion and distance estimation approach were evaluated in terms of quantitative and qualitative analysis by considering real road and simulation environment scenarios. Thus the proposed low-level sensor fusion, based on the computational geometric transformation and projection for object distance estimation proves to be a promising solution for enabling reliable and consistent environment perception ability for autonomous vehicles.

A LiDAR–optical data fusion approach for identifying and measuring small stream impoundments and dams

AbstractThis article outlines a semi‐autonomous approach for using a fusion of light detection and ranging (LiDAR) and optical remote sensing data to identify and measure small impoundments (SIs) and their dams. Quantifying such water bodies as hydrologic network features is critical for ecosystem and species conservation, emergency management, and water resource planning; however, such features are incompletely mapped at national and state levels. By merging an airborne LiDAR‐derived point cloud with a normalized water index using airborne optical imagery we demonstrate an improvement upon single‐source methods for identifying these water bodies; classification accuracies increased over 10% by using this multi‐source fusion method. Furthermore, the method presented here illustrates a cost‐effective pathway to improve the National Inventory of Dams (NID) and includes a framework for estimating dam heights, with results showing strong correlations between derived dam heights and those recorded in the NID (r=.78). With the steady increase in available LiDAR coverage, the 87,000+ dams in the NID could be updated using this technique, a method which could also be expanded for global inventories of SIs and dams.

Pixels and 3-D Points Alignment Method for the Fusion of Camera and LiDAR Data

The fusion of light detection and ranging (LiDAR) and camera data is a promising approach to improve the environmental perception and recognition for intelligent vehicles because of the combination of depth and color information. One of the difficulties in achieving the fusion is the accurate alignment of the 3-D points with the image pixels. Current methods of data alignment involve the steps of estimating the camera intrinsic parameters and developing a transformation matrix between the camera and LiDAR frame. The drawback of these methods is the accumulation of errors during the calculation of the camera intrinsic parameters and the transformation matrix. In order to improve the data alignment accuracy, we propose a novel algorithm that directly calculates the alignment between the 3-D points and the pixels without the need for camera parameters and calibration of the coordinate transformation matrix. We call the proposed method the pixel and 3-D point alignment (PPA) method. The alignment procedure is achieved by using the extracted corresponding points. First, we calculate a linear alignment matrix without considering the image distortion; and second, we optimize the parameters using the maximum likelihood estimation to consider the camera distortion. Simulation and experimental results indicate that the PPA method is able to align the 3-D points in LiDAR frame with the pixels in image frame with higher accuracy and increased robustness against noise in calibration process than comparable state-of-the-art methods.

Integrated Airborne LiDAR Data and Imagery for Suburban Land Cover Classification Using Machine Learning Methods.

It is valuable to study the land use/land cover (LULC) classification for suburbs. The fusion of Light Detection and Ranging (LiDAR) data and aerial imagery is often regarded as an effective method for the LULC classification; however, more in-depth analysis would be required to explore effective information for enhancing the suburban LULC classification. In this study, first, both aerial imageries and point clouds were simultaneously collected. Then, LiDAR-derived models, i.e., normalized digital surface model (nDSM) and surface intensity model (IM), were generated from the elevation and intensity of point clouds. Further, considering the surface characteristics of ground objects in suburb, we proposed a new LiDAR-derived model, namely surface roughness model (RM), to reveal the degree of surface fluctuations. Additionally, various combinations of aerial imageries and the LiDAR-derived data were used to analyze the effects of multi-variable fusion under different scenarios and optimize the multi-variable integration for suburban LULC classification. The mean decrease impurity method was used to identify the importance of variables; three machine learning classifiers, i.e., random forest (RF), k-nearest neighbor (KNN) and artificial neural network (ANN) were adopted in various scenarios. The results were as follows. The fusion of aerial imagery and all the LiDAR-derived models, i.e., nDSM, RM and IM, with RF classifier performs best in the suburban LULC classification (overall accuracy = 84.75%, kappa coefficient = 0.80). Variable importance analysis shows that nDSM has the highest variable importance proportion (VIP) value, followed by RM, IM, and spectral information, indicating the feasibility of this proposed LiDAR-derived model-RM. This research presents effective methods relating to the application of aerial imagery and LiDAR-derived model for the complex suburban surface scenarios.

Deep Learning Approach for Building Detection Using LiDAR–Orthophoto Fusion

This paper reports on a building detection approach based on deep learning (DL) using the fusion of Light Detection and Ranging (LiDAR) data and orthophotos. The proposed method utilized object-based analysis to create objects, a feature-level fusion, an autoencoder-based dimensionality reduction to transform low-level features into compressed features, and a convolutional neural network (CNN) to transform compressed features into high-level features, which were used to classify objects into buildings and background. The proposed architecture was optimized for the grid search method, and its sensitivity to hyperparameters was analyzed and discussed. The proposed model was evaluated on two datasets selected from an urban area with different building types. Results show that the dimensionality reduction by the autoencoder approach from 21 features to 10 features can improve detection accuracy from 86.06% to 86.19% in the working area and from 77.92% to 78.26% in the testing area. The sensitivity analysis also shows that the selection of the hyperparameter values of the model significantly affects detection accuracy. The best hyperparameters of the model are 128 filters in the CNN model, the Adamax optimizer, 10 units in the fully connected layer of the CNN model, a batch size of 8, and a dropout of 0.2. These hyperparameters are critical to improving the generalization capacity of the model. Furthermore, comparison experiments with the support vector machine (SVM) show that the proposed model with or without dimensionality reduction outperforms the SVM models in the working area. However, the SVM model achieves better accuracy in the testing area than the proposed model without dimensionality reduction. This study generally shows that the use of an autoencoder in DL models can improve the accuracy of building recognition in fused LiDAR–orthophoto data.

Building extraction from fused LiDAR and hyperspectral data using Random Forest Algorithm

In this study, the fusion of Light Detection and Ranging (LiDAR) and hyperspectral data was used to propose a method for building detection. The number of hyperspectral bands was first reduced from 144 to 8 layers using the Linear Discriminant Analysis (LDA) algorithm to remove highly redundant bands and reduce computational costs. Then, these layers were integrated with 4 layers of heights and intensities obtained from the LiDAR data. The fused layers (12 layers) were applied to a Random Forest (RF) algorithm to extract the boundaries of buildings. Finally, two morphological operators were applied to remove the holes on the buildings’ roofs and repair their boundaries. A comparison was also performed between the results obtained by the proposed method and the reference study in this field [Debes et al. 2014]. The proposed method demonstrated a better accuracy for building detection in a much shorter time compared to the refer ence method. The values of 97% and 96% were obtained for producer and user accuracies, respectively. Overall, the method presented in this study proved to have a high potential for building extraction.

Feature grouping-based multiple fuzzy classifier system for fusion of hyperspectral and LIDAR data

Interest in the joint use of different data from multiple sensors has been increased for classification applications. This is because the fusion of different information can produce a better understanding of the observed site. In this field of study, the fusion of light detection and ranging (LIDAR) and passive optical remote sensing data for classification of land cover has attracted much attention. This paper addressed the use of a combination of hyperspectral (HS) and LIDAR data for land cover classification. HS images provide a detailed description of the spectral signatures of classes, whereas LIDAR data give detailed information about the height but no information for the spectral signatures. This paper presents a multiple fuzzy classifier system for fusion of HS and LIDAR data. The system is based on the fuzzy K-nearest neighbor (KNN) classification of two data sets after application of feature grouping on them. Then a fuzzy decision fusion method is applied to fuse the results of fuzzy KNN classifiers. An experiment was carried out on the classification of HS and LIDAR data from Houston, USA. The proposed fuzzy classifier ensemble system for HS and LIDAR data provide interesting conclusions on the effectiveness and potentials of the joint use of these two data. Fuzzy classifier fusion on these two data sets improves the classification results when compared with independent single fuzzy classifiers on each data set. The fuzzy proposed method represented the best accuracy with a gain in overall accuracy of 93%.

Seamless Fusion of LiDAR and Aerial Imagery for Building Extraction

Although many efforts have been made on the fusion of Light Detection and Ranging (LiDAR) and aerial imagery for the extraction of houses, little research on taking advantage of a building's geometric features, properties, and structures for assisting the further fusion of the two types of data has been made. For this reason, this paper develops a seamless fusion between LiDAR and aerial imagery on the basis of aspect graphs, which utilize the features of houses, such as geometry, structures, and shapes. First, 3-D primitives, standing for houses, are chosen, and their projections are represented by the aspects. A hierarchical aspect graph is then constructed using aerial image processing in combination with the results of LiDAR data processing. In the aspect graph, the note represents the face aspect and the arc is described by attributes obtained by the formulated coding regulations, and the coregistration between the aspect and LiDAR data is implemented. As a consequence, the aspects and/or the aspect graph are interpreted for the extraction of houses, and then the houses are fitted using a planar equation for creating a digital building model (DBM). The experimental field, which is located in Wytheville, VA, is used to evaluate the proposed method. The experimental results demonstrated that the proposed method is capable of effectively extracting houses at a successful rate of 93%, as compared with another method, which is 82% effective when LiDAR spacing is approximately 7.3 by 7.3 ft $^{2}$ . The accuracy of 3-D DBM is higher than the method using only single LiDAR data.

A decision fusion method based on multiple support vector machine system for fusion of hyperspectral and LIDAR data

Fusion of remote sensing data from multiple sensors has been remarkably increased for classification. This is because, additional sources may provide more information, and fusion of different information can produce a better understanding of the observed site. In the field of data fusion, fusion of light detection and ranging (LIDAR) and optical remote sensing data for land cover classification has attracted more attention. This paper addressed the use of a decision fusion methodology for the combination of hyperspectral and LIDAR data in land cover classification. The proposed method applied a support vector machine (SVM)-based classifier fusion system for fusion of hyperspectral and LIDAR data in the decision level. First, feature spaces are extracted on LIDAR and hyperspectral data. Then, SVM classifiers are applied on each feature data. After producing multiple of classifiers, Naive Bayes as a classifier fusion method combines the results of SVM classifiers form two data sets. A co-registered hyperspectral and LIDAR data set from Houston, USA, was available to examine the effect of the proposed decision fusion methodology. Experimental results show that the proposed data fusion method improved the classification accuracy and kappa coefficient in comparison to the single data sets. The results revealed that the overall accuracies of SVM classification on hyperspectral and LIDAR data separately are 88% and 58% while our decision fusion methodology receive the accuracy up to 91%.

A Sensor-Fusion Drivable-Region and Lane-Detection System for Autonomous Vehicle Navigation in Challenging Road Scenarios

Autonomous vehicle navigation is challenging since various types of road scenarios in real urban environments have to be considered, particularly when only perception sensors are used, without position information. This paper presents a novel real-time optimal-drivable-region and lane detection system for autonomous driving based on the fusion of light detection and ranging (LIDAR) and vision data. Our system uses a multisensory scheme to cover the most drivable areas in front of a vehicle. We propose a feature-level fusion method for the LIDAR and vision data and an optimal selection strategy for detecting the best drivable region. Then, a conditional lane detection algorithm is selectively executed depending on the automatic classification of the optimal drivable region. Our system successfully handles both structured and unstructured roads. The results of several experiments are provided to demonstrate the reliability, effectiveness, and robustness of the system.

Mapping Savanna Tree Species at Ecosystem Scales Using Support Vector Machine Classification and BRDF Correction on Airborne Hyperspectral and LiDAR Data

Mapping the spatial distribution of plant species in savannas provides insight into the roles of competition, fire, herbivory, soils and climate in maintaining the biodiversity of these ecosystems. This study focuses on the challenges facing large-scale species mapping using a fusion of Light Detection and Ranging (LiDAR) and hyperspectral imagery. Here we build upon previous work on airborne species detection by using a two-stage support vector machine (SVM) classifier to first predict species from hyperspectral data at the pixel scale. Tree crowns are segmented from the lidar imagery such that crown-level information, such as maximum tree height, can then be combined with the pixel-level species probabilities to predict the species of each tree. An overall prediction accuracy of 76% was achieved for 15 species. We also show that bidirectional reflectance distribution (BRDF) effects caused by anisotropic scattering properties of savanna vegetation can result in flight line artifacts evident in species probability maps, yet these can be largely mitigated by applying a semi-empirical BRDF model to the hyperspectral data. We find that confronting these three challenges—reflectance anisotropy, integration of pixel- and crown-level data, and crown delineation over large areas—enables species mapping at ecosystem scales for monitoring biodiversity and ecosystem function.

Anticollision system PRORETA with automatic braking and steering

Methods and experimental results of a collision-avoidance, driver-assistance system are described with automatic object detection, trajectory prediction, and track following, with controlled braking and steering. The objects are detected by a fusion of Light Detection and Ranging (LIDAR) scanning and video camera pictures that show the location, size and speed of objects in front of the car. A desired trajectory is calculated depending on the distance, the width of a swerving action and differential speed. For the trajectory control, different control methods were designed and tested experimentally, such as velocity-dependent linear feedback and feedforward control, nonlinear asymptotic output tracking and nonlinear flatness-based control, using extended one-track models with vehicle state estimation for the sideslip angle and cornering stiffness. Automatic braking is realised with an electrohydraulic brake and automatic steering with an active front steering. Various control systems are compared by simulations and real test drives showing the behaviour of a Volkswagen Golf with automatic braking and/or automatic swerving to a free lane, thus avoiding hitting a suddenly appearing obstacle. The research project PRORETA was a four-year cooperation between Continental Automotive Systems and Darmstadt University of Technology.