Fusion Of LiDAR Data Research Articles

Optimized Factor Graph for Tightly-Coupled LiDAR/IMU Localization in Underground Parking Garages

Abstract. To address the challenge of intelligent vehicle localization in underground parking structures due to the loss of GNSS signals, this paper introduces a method to address this issue by developing a novel localization framework known as GF-LIO, which denotes a tightly-coupled fusion of LiDAR and IMU data, innovatively combining the Interactive Extended State Kalman Filter (IESKF) with a factor graph to enhance the localization process and solve the problem of GNSS signal loss in underground parking lots. The GF-LIO model commences with a strategic feature selection process, facilitated by a greedy algorithm that prioritizes environmental cues within the point cloud data. This method effectively filters out redundant features, thereby enhancing the saliency of retained features and subsequently improving the robustness of the localization process. Following feature selection, the model integrates LiDAR and IMU measurements utilizing the IESKF algorithm, ensuring a cohesive fusion of sensor data and bolstering attitude estimation accuracy. The culmination of the GF-LIO framework involves factor graph optimization, a sophisticated technique that synthesizes LiDAR odometry, IMU pre-integration factors, and loop closure detection factors. This optimization step enhances the overall precision and consistency of the localization process, resulting in superior performance compared to existing methodologies. Experimental evaluations conducted within underground parking environments corroborate the efficacy of the GF-LIO model. Comparative analyses against established approaches such as A-LOAM, LeGO-LOAM, LIO-LOAM, and FAST-LIO demonstrate a notable performance improvement exceeding 12.53%. The proposed model adeptly integrates domain-specific environmental characteristics with multi-sensor data, thereby facilitating precise localization and map construction tasks for intelligent vehicle navigating within the intricate confines of subterranean parking structures.

Data Fusion of Roadside Camera, LiDAR, and Millimeter-Wave Radar

Data Fusion of Roadside Camera, LiDAR, and Millimeter-Wave Radar

Occlusion-guided multi-modal fusion for vehicle-infrastructure cooperative 3D object detection

In autonomous driving, leveraging sensor data (e.g. camera, LiDAR data) from both the vehicle and the infrastructure significantly improves perception capabilities. However, this integration traditionally results in increased demands on communication bandwidth. To address these challenges, we introduce Fusion2comm, an occlusion-guided feature fusion approach designed to optimize vehicle-infrastructure cooperative 3D object detection. Our innovative strategy employs an intelligent fusion of camera and LiDAR data to enhance the expressiveness of features. Subsequently, it leverages a segmentation model to extract foreground features and utilizes an occlusion-based selection of communication content, effectively easing bandwidth constraints. We propose a multimodal foreground feature fusion architecture that selectively processes and transmits critical information, substantially reducing irrelevant background data transfer. An innovative occlusion confidence-aware communication technique dynamically adjusts communication regions based on occlusion levels, ensuring efficient data exchange. Fusion2comm sets a new benchmark in the DAIR-V2X dataset, achieving an average precision of 71.25% with minimal bandwidth usage of 221.04 bytes. Our comprehensive experimental evaluations confirm that Fusion2comm substantially advances detection precision while simultaneously improving communication efficiency.

CL-fusionBEV: 3D object detection method with camera-LiDAR fusion in Bird’s Eye View

In the wave of research on autonomous driving, 3D object detection from the Bird’s Eye View (BEV) perspective has emerged as a pivotal area of focus. The essence of this challenge is the effective fusion of camera and LiDAR data into the BEV. Current approaches predominantly train and predict within the front view and Cartesian coordinate system, often overlooking the inherent structural and operational differences between cameras and LiDAR sensors. This paper introduces CL-FusionBEV, an innovative 3D object detection methodology tailored for sensor data fusion in the BEV perspective. Our approach initiates with a view transformation, facilitated by an implicit learning module that transitions the camera’s perspective to the BEV space, thereby aligning the prediction module. Subsequently, to achieve modal fusion within the BEV framework, we employ voxelization to convert the LiDAR point cloud into BEV space, thereby generating LiDAR BEV spatial features. Moreover, to integrate the BEV spatial features from both camera and LiDAR, we have developed a multi-modal cross-attention mechanism and an implicit multi-modal fusion network, designed to enhance the synergy and application of dual-modal data. To counteract potential deficiencies in global reasoning and feature interaction arising from multi-modal cross-attention, we propose a BEV self-attention mechanism that facilitates comprehensive global feature operations. Our methodology has undergone rigorous evaluation on a substantial dataset within the autonomous driving domain, the nuScenes dataset. The outcomes demonstrate that our method achieves a mean Average Precision (mAP) of 73.3% and a nuScenes Detection Score (NDS) of 75.5%, particularly excelling in the detection of cars and pedestrians with high accuracies of 89% and 90.7%, respectively. Additionally, CL-FusionBEV exhibits superior performance in identifying occluded and distant objects, surpassing existing comparative methods.

A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

Extrinsic parameter calibration is the foundation and prerequisite for LiDAR and camera data fusion of the autonomous system. This technology is widely used in fields such as autonomous driving, mobile robots, intelligent surveillance, and visual measurement. The learning-based method is one of the targetless calibrating methods in LiDAR and camera calibration. Due to its advantages of fast speed, high accuracy, and robustness under complex conditions, it has gradually been applied in practice from a simple theoretical model in just a few years, becoming an indispensable and important method. This paper systematically summarizes the research and development of this type of method in recent years. According to the principle of calibration parameter estimation, learning-based calibration algorithms are divided into two categories: accurate calibrating estimation and relative calibrating prediction. The evolution routes and algorithm frameworks of these two types of algorithms are elaborated, and the methods used in the algorithms’ steps are summarized. The algorithm mechanism, advantages, limitations, and applicable scenarios are discussed. Finally, we make a summary, pointing out existing research issues and trends for future development.

An Advanced Approach to Object Detection and Tracking in Robotics and Autonomous Vehicles Using YOLOv8 and LiDAR Data Fusion

Accurately and reliably perceiving the environment is a major challenge in autonomous driving and robotics research. Traditional vision-based methods often suffer from varying lighting conditions, occlusions, and complex environments. This paper addresses these challenges by combining a deep learning-based object detection algorithm, YOLOv8, with LiDAR data fusion technology. The principle of this combination is to merge the advantages of these technologies: YOLOv8 excels in real-time object detection and classification through RGB images, while LiDAR provides accurate distance measurement and 3D spatial information, regardless of lighting conditions. The integration aims to apply the high accuracy and robustness of YOLOv8 in identifying and classifying objects, as well as the depth data provided by LiDAR. This combination enhances the overall environmental perception, which is critical for the reliability and safety of autonomous systems. However, this fusion brings some research challenges, including data calibration between different sensors, filtering ground points from LiDAR point clouds, and managing the computational complexity of processing large datasets. This paper presents a comprehensive approach to address these challenges. Firstly, a simple algorithm is introduced to filter out ground points from LiDAR point clouds, which are essential for accurate object detection, by setting different threshold heights based on the terrain. Secondly, YOLOv8, trained on a customized dataset, is utilized for object detection in images, generating 2D bounding boxes around detected objects. Thirdly, a calibration algorithm is developed to transform 3D LiDAR coordinates to image pixel coordinates, which are vital for correlating LiDAR data with image-based object detection results. Fourthly, a method for clustering different objects based on the fused data is proposed, followed by an object tracking algorithm to compute the 3D poses of objects and their relative distances from a robot. The Agilex Scout Mini robot, equipped with Velodyne 16-channel LiDAR and an Intel D435 camera, is employed for data collection and experimentation. Finally, the experimental results validate the effectiveness of the proposed algorithms and methods.

Cueing Flight Object Trajectory and Safety Prediction Based on SLAM Technology

With the rapid development of artificial intelligence and robot technology, SLAM technology, as a key component, has been paid more and more attention. SLAM technology enables robots to autonomously navigate, build maps, and achieve accurate positioning in unknown environments, providing strong support for the autonomy and intelligence of robots and unmanned vehicles. In this paper, the position prediction method of flying object based on SLAM technology and the application of EvolveGCN model in behavior prediction are introduced. First, through the fusion of SLAM technology and liDAR data, we can accurately predict the position and movement trajectory of flying objects, thereby improving the safety and efficiency of the system. Secondly, with the EvolveGCN model, we are able to capture dynamic changes in the environment and achieve accurate predictions of the behavior of flying objects. Through experimental verification, the prediction accuracy of our method has been significantly improved in both simulation and real environment, which indicates the feasibility and effectiveness of the method in practical application, and provides an important reference and technical support for the development of autonomous navigation, aerial surveillance and other fields.

Fusion of Dense Airborne LiDAR and Multispectral Sentinel-2 and Pleiades Satellite Imagery for Mapping Riparian Forest Species Biodiversity at Tree Level.

Multispectral and 3D LiDAR remote sensing data sources are valuable tools for characterizing the 3D vegetation structure and thus understanding the relationship between forest structure, biodiversity, and microclimate. This study focuses on mapping riparian forest species in the canopy strata using a fusion of Airborne LiDAR data and multispectral multi-source and multi-resolution satellite imagery: Sentinel-2 and Pleiades at tree level. The idea is to assess the contribution of each data source in the tree species classification at the considered level. The data fusion was processed at the feature level and the decision level. At the feature level, LiDAR 2D attributes were derived and combined with multispectral imagery vegetation indices. At the decision level, LiDAR data were used for 3D tree crown delimitation, providing unique trees or groups of trees. The segmented tree crowns were used as a support for an object-based species classification at tree level. Data augmentation techniques were used to improve the training process, and classification was carried out with a random forest classifier. The workflow was entirely automated using a Python script, which allowed the assessment of four different fusion configurations. The best results were obtained by the fusion of Sentinel-2 time series and LiDAR data with a kappa of 0.66, thanks to red edge-based indices that better discriminate vegetation species and the temporal resolution of Sentinel-2 images that allows monitoring the phenological stages, helping to discriminate the species.

Deep learning-based fusion networks with high-order attention mechanism for 3D object detection in autonomous driving scenarios

In the realm of autonomous driving, accurately detecting 3D objects is both vital and challenging. Recently, deep convolutional networks have been successfully implemented in the fusion of LiDAR and camera data, delivering impressive results. However, prevailing approaches tend to concentrate on basic architectural designs and the use of fixed 3D bounding boxes, overlooking the exploration of feature interrelations and the varying scales of 3D objects. In this paper, we propose High-order Attention Mechanism Fusion Networks (HAMFNs) for image expression and multi-scale learning, based on a novel high-order attention mechanism with multi-scale detection and scale linear regression. High-order convolution layers are built for tenser filtering with discriminative representations of the holistic image. Multi-scale query module further characterizes the saliency properties of the 3D objects. Our tests on the nuScenes dataset show that HAMFNs outperform the latest top-performing methods, achieving a 0.7% increase in mean Average Precision (mAP). We further integrated high-order convolutional layers into ResNet-50, ResNet-101, and ResNet-152 architectures, enhancing their performance with minimal parameter increase. The Top-1 error rates were reduced by 1.65%, 1.63%, and 1.60% for each network, respectively.

Multilevel Attention Dynamic-scale Network for HSI and LiDAR Data Fusion Classification

Multilevel Attention Dynamic-scale Network for HSI and LiDAR Data Fusion Classification

Comparative Study on BEV Vision and LiDAR Point Cloud Data Fusion Methods

With the gradual maturity of autonomous driving technology, efficient fusion and processing of multimodal sensor data has become an important research direction. This study mainly explores the strategy of integrating BEV (Bird's Eye View) vision with LiDAR point cloud data. We evaluated the performance and applicability of each of the three main data fusion methods through in-depth comparison: early fusion, mid-term fusion, and late fusion. First of all, we summarize the working principle and data characteristics of BEV vision and LiDAR, and emphasize their key roles in auto drive system. Then, the theoretical basis and implementation methods of the three fusion strategies were described in detail. The experimental results show that different fusion strategies exhibit their own advantages in different application scenarios and requirements. For example, early fusion performs well in high-precision tasks, but has a high demand for computing resources. And mid-term fusion is more suitable in scenarios with high real-time requirements. Overall, this study provides in-depth insights and practical suggestions on the fusion of BEV vision and LiDAR data in the field of autonomous driving, laying a solid foundation for future research and applications.

Monitoring the effects of weed management strategies on tree canopy structure and growth using UAV-LiDAR in a young almond orchard

The primary objective of this study was to assess the potential effect of integrated weed management (IWM) on canopy structure and growth in a young almond orchard using unmanned aerial vehicle (UAV) LiDAR point cloud data. The experiment took place in the Neve Ya’ar Model Farm, with four IWM strategies tested: (1) standard herbicide, (2) physical-mechanical, (3) cover crops, and (4) integrated management combining herbicide and mowing. One pre-treatment sessions in 2019 and two experiment sessions in 2020–2021 were conducted. During these three experimental sessions, manual measurements, including trunk diameter (TD), were taken for all trees in the orchard. Six canopy growth parameters (e.g., tree height and volume), and functionality parameters (e.g., leaf density, gap fraction profile (GFP) and entropy) were calculated for individual trees using UAV-LiDAR data and were compared to manual measurement (i.e., TD). The results indicated that the herbicide treatment enabled effectively maintained weed coverage at approximately 5–10 % throughout the experiment, demonstrating successful control over weed growth. In contrast, the integrated weed management strategy yielded the highest weed cover (∼50 %). With the view of replacing ground manual measurements with UAV-LiDAR-derived measurements, we explored the relationships between TD and UAV-LiDAR-derived volume and GFP. The relationships between TD and GFP were with R2 values of 0.43 and 0.57 for 2021, respectively. The relationship between GFP and volume showed an R2 value of 0.65 for both years. We observed significant differences between IWM for TD, volume, and GFP in both years, indicating on higher tree development in the herbicide and integrated management treatments. The standard herbicide-based management yielded the highest crop yield, while the cover crop strategy yielded the lowest per plot. Finally, random forest (RF) model was used to identify key factors that significantly influence tree volume, TD, and GFP. The RF results showed that IWM explained about 30 % of tree structural changes, while the rest were attributed to environmental factors (i.e., topographical and climatic). The RF model R2 values ranged between 0.79 and 0.86, representing the topographical and climatic effects on tree structural parameters and growth, reflecting the natural field variability. Our study designed and successfully implemented a framework for estimating canopy characterization in young almond orchards subjected to varied IWM treatments. Ensuring the precise calibration of the LiDAR system and validating data, especially in complex canopy structures, can be challenging since the accuracy of ground-truth data has its errors. For future study, the fusion of LiDAR data with multispectral imagery or terrestrial LiDAR data can be considered, as well as more frequent UAV-LiDAR re-visits, offering insights into phenological states and higher resolution of canopy characteristics in orchards. This research enhances our understanding of the advantages and challenges of using UAV-LiDAR point cloud data analysis to assess the impact of IWM on young almond tree growth.

Synergizing IFSAR and LiDAR data fusion using Delta Surface Fill for Hybrid DEMs accuracy in topographic mapping

Synergizing IFSAR and LiDAR data fusion using Delta Surface Fill for Hybrid DEMs accuracy in topographic mapping

Early detection of pine shoot beetle attack using vertical profile of plant traits through UAV-based hyperspectral, thermal, and lidar data fusion

Pine shoot beetle (PSB) is one of the most damaging forest insects of Yunnan pine plantations in southwest China. However, the subtle symptoms of heterogeneous tree crowns make it difficult to accurately detect at early stage of PSB attack. Here, we evaluated the potential of a combination of plant traits (PTs) and vegetation indices (VIs) to distinguish different levels of tree damage by integrating hyperspectral, thermal imagery, and light detection and ranging (lidar) data based on unmanned airborne vehicle (UAV) systems. A voxelization method was used to fuse hyperspectral reflectance, temperature, and lidar point cloud data. Subsequently, PTs such as pigments (i.e., chlorophyll, carotenoid, and anthocyanin contents) were retrieved from a radiative transfer model inversion, and structural, fluorescence, and thermal traits, as well as VIs, were derived from the fusion data. We developed a novel analytical approach using random forest (RF) algorithm with predictors from different spatial distributions (i.e., horizontal directions, vertical layers, and vertical clusters) to compare the performance of tree severity classification. The results showed that the difference of both PTs and VIs in vertical layers between different severity levels are more than that in horizontal directions. The performance of RF model with predictors of the vertical layers (OA = 74 %, kappa = 0.65) was better than that using the predictive variables in horizontal directions (OA = 69 % and kappa = 0.60) of tree crowns. Using the vertical clustering features RF model increased the accuracy (OA = 78 % and kappa = 0.70), especially for slightly and moderately damaged trees, with improvements of 10 % and 12 %, respectively. Among all variables analyzed, chlorophylls were the most important predictor, followed by photochemical reflectance index and structural traits. Our work demonstrates the effectiveness of using fused UAV-based multi-sensor data for early detection of PSB attack, and can be applied other potential forest diseases and insect monitoring.

Adaptive Feature Attention Module for Robust Visual–LiDAR Fusion-Based Object Detection in Adverse Weather Conditions

Object detection is one of the vital components used for autonomous navigation in dynamic environments. Camera and lidar sensors have been widely used for efficient object detection by mobile robots. However, they suffer from adverse weather conditions in operating environments such as sun, fog, snow, and extreme illumination changes from day to night. The sensor fusion of camera and lidar data helps to enhance the overall performance of an object detection network. However, the diverse distribution of training data makes the efficient learning of the network a challenging task. To address this challenge, we systematically study the existing visual and lidar features based on object detection methods and propose an adaptive feature attention module (AFAM) for robust multisensory data fusion-based object detection in outdoor dynamic environments. Given the camera and lidar features extracted from the intermediate layers of EfficientNet backbones, the AFAM computes the uncertainty among the two modalities and adaptively refines visual and lidar features via attention along the channel and the spatial axis. The AFAM integrated with the EfficientDet performs the adaptive recalibration and fusion of visual lidar features by filtering noise and extracting discriminative features for an object detection network under specific environmental conditions. We evaluate the AFAM on a benchmark dataset exhibiting weather and light variations. The experimental results demonstrate that the AFAM significantly enhances the overall detection accuracy of an object detection network.

Spatio-Temporal Fusion of LiDAR and Camera Data for Omnidirectional Depth Perception

Object recognition and depth perception are two tightly coupled tasks that are indispensable for situational awareness. Most autonomous systems are able to perform these tasks by processing and integrating data streaming from a variety of sensors. The multiple hardware and sophisticated software architectures required to operate these systems makes them expensive to scale and operate. This paper implements a fast, monocular vision system that can be used for simultaneous object recognition and depth perception. We borrow from the architecture of a start-of-the-art object recognition system, YOLOv3, and extend its architecture by incorporating distances and modifying its loss functions and prediction vectors to enable it to multitask on both tasks. The vision system is trained on a large database acquired through the coupling of LiDAR measurements with complementary 360-degree camera to generate a high-fidelity labeled dataset. The performance of the multipurpose network is evaluated on a test dataset consisting of a total of 7,634 objects collected on a different road network. When compared with ground truth LiDAR data, the proposed network achieves a mean absolute percentage error rate of 11% on the passenger car within 10 m and a mean error rate of 7% or 9% on the truck within 10 m and beyond 10 m, respectively. It was also observed that adding a second task (depth perception) to the modeling network improved the accuracy of object detection by about 3%. The proposed multipurpose model can be used for the development of automated alert systems, traffic monitoring, and safety monitoring.

A Feature-Level Point Cloud Fusion Method for Timber Volume of Forest Stands Estimation

Accurate diameter at breast height (DBH) and tree height (H) information can be acquired through terrestrial laser scanning (TLS) and airborne LiDAR scanner (ALS) point cloud, respectively. To utilize these two features simultaneously but avoid the difficulties of point cloud fusion, such as technical complexity and time-consuming and laborious efforts, a feature-level point cloud fusion method (FFATTe) is proposed in this paper. Firstly, the TLS and ALS point cloud data in a plot are georeferenced by differential global navigation and positioning system (DGNSS) technology. Secondly, point cloud processing and feature extraction are performed for the georeferenced TLS and ALS to form feature datasets, respectively. Thirdly, the feature-level fusion of LiDAR data from different data sources is realized through spatial join according to the tree trunk location obtained from TLS and ALS, that is, the tally can be implemented at a plot. Finally, the individual tree parameters are optimized based on the tally results and fed into the binary volume model to estimate the total volume (TVS) in a large area (whole study area). The results show that the georeferenced ALS and TLS point cloud data using DGNSS RTK/PPK technology can achieve coarse registration (mean distance ≈ 40 cm), which meets the accuracy requirements for feature-level point cloud fusion. By feature-level fusion of the two point cloud data, the tally can be achieved quickly and accurately in the plot. The proposed FFATTe method achieves high accuracy (with error of 3.09%) due to its advantages of combining different LiDAR data from different sources in a simple way, and it has strong operability when acquiring TVS over large areas.

Diverging Ozone Trends Above Western North America: Boundary Layer Decreases Versus Free Tropospheric Increases

AbstractThis study has produced an improved percentile and seasonal (median) trend estimate of free tropospheric ozone above western North America (WNA), through a data fusion of ozonesonde, lidar, commercial aircraft, and field campaign measurements. Our method combines heterogeneous data sets according to the consensus data characteristics and inherent uncertainty in order to produce our best fused product. In response to different data collection environments (in situ or ground‐based), we investigate the ozone variability based on a wide range of percentiles, which is preferable for trend detection due to tropospheric ozone's high degree of heteroscedasticity (i.e., inconsistent trends and variability between different ozone percentiles). We then compare the ozone trends and variability above the California sub‐domain to the full WNA region for better understanding of the correlations between different regional scales. In California, the 1995–2021 percentile (from the 5th to 95th) and seasonal trends are clearly positive in terms of high signal‐to‐noise ratios. The magnitude of the trends is generally weaker over WNA compared to California, but reliable positive trends can still be found between the 10th and 70th percentiles, as well as winter and summer, whereas autumn shows a negative trend over the same period. In addition, dozens of rural surface sites across the region are selected to represent the boundary layer variability. In contrast to increasing free tropospheric ozone, we find overall strong negative surface trends since 1995, with the greatest divergence found in summer. Throughout the analysis implications of the COVID‐19 economic downturn on ozone variability are discussed.

Forest Emissions Reduction Assessment Using Optical Satellite Imagery and Space LiDAR Fusion for Carbon Stock Estimation

Forests offer significant climate mitigation benefits, but existing emissions reduction assessment methodologies in forest-based mitigation activities are not scalable, which limits the development of carbon offset markets. In this study, we propose a measurement method using optical satellite imagery and space LiDAR data fusion to assess forest emissions reduction. Compared with the ALS-based carbon stock density estimation method, our approach presented a strong scalability for mapping 10 m-resolution carbon stock at a large scale. It was observed that dense canopy top height estimated by combining GEDI and Sentinel-2 could accurately predict forest carbon stock measurements estimated by the ALS-based method (R2 = 0.72). By conducting an on-site experiment of an ongoing forest carbon project in China, we found the consistency between the emissions reduction assessed by the data fusion measurement method (589,169 tCO2e) and the official ex post-monitored emissions reduction in the monitoring report (598,442 tCO2e). Our results demonstrated that forest carton stock estimation using optical satellite imagery and space LiDAR data fusion is efficient and economical for forest emissions reduction assessment. The acquisition of the data was more efficient over large areas with high frequencies using space-based technology. We further discussed the challenge of building a near-real-time monitoring system for forest-based mitigation activities by utilizing optical satellite imagery and space LiDAR data and pointed out that a quality control framework should be established to help us understand the sources of uncertainty in LiDAR-based models and improve carbon stock estimation from individual trees to forest carbon projects to meet the requirements of carbon standards better.

UAV Multisensory Data Fusion and Multi-Task Deep Learning for High-Throughput Maize Phenotyping.

Recent advances in unmanned aerial vehicles (UAV), mini and mobile sensors, and GeoAI (a blend of geospatial and artificial intelligence (AI) research) are the main highlights among agricultural innovations to improve crop productivity and thus secure vulnerable food systems. This study investigated the versatility of UAV-borne multisensory data fusion within a framework of multi-task deep learning for high-throughput phenotyping in maize. UAVs equipped with a set of miniaturized sensors including hyperspectral, thermal, and LiDAR were collected in an experimental corn field in Urbana, IL, USA during the growing season. A full suite of eight phenotypes was in situ measured at the end of the season for ground truth data, specifically, dry stalk biomass, cob biomass, dry grain yield, harvest index, grain nitrogen utilization efficiency (Grain NutE), grain nitrogen content, total plant nitrogen content, and grain density. After being funneled through a series of radiometric calibrations and geo-corrections, the aerial data were analytically processed in three primary approaches. First, an extended version normalized difference spectral index (NDSI) served as a simple arithmetic combination of different data modalities to explore the correlation degree with maize phenotypes. The extended NDSI analysis revealed the NIR spectra (750-1000 nm) alone in a strong relation with all of eight maize traits. Second, a fusion of vegetation indices, structural indices, and thermal index selectively handcrafted from each data modality was fed to classical machine learning regressors, Support Vector Machine (SVM) and Random Forest (RF). The prediction performance varied from phenotype to phenotype, ranging from R2 = 0.34 for grain density up to R2 = 0.85 for both grain nitrogen content and total plant nitrogen content. Further, a fusion of hyperspectral and LiDAR data completely exceeded limitations of single data modality, especially addressing the vegetation saturation effect occurring in optical remote sensing. Third, a multi-task deep convolutional neural network (CNN) was customized to take a raw imagery data fusion of hyperspectral, thermal, and LiDAR for multi-predictions of maize traits at a time. The multi-task deep learning performed predictions comparably, if not better in some traits, with the mono-task deep learning and machine learning regressors. Data augmentation used for the deep learning models boosted the prediction accuracy, which helps to alleviate the intrinsic limitation of a small sample size and unbalanced sample classes in remote sensing research. Theoretical and practical implications to plant breeders and crop growers were also made explicit during discussions in the studies.