Fusion Of Imagery Research Articles

Camphor tree detection in urban environments using RGB-DSM data fusion

ABSTRACT Individual tree detection in urban areas using unmanned air vehicles (UAVs) RGB imagery poses challenges due to the diverse shapes and structures of urban trees and the complexity of urban forests. The digital surface model (DSM) provides elevation data, and the fusion of UAV RGB imagery with elevation data has emerged as a promising approach for tree detection. Here, we constructed a novel network structure based on the faster region-based convolutional neural network (Faster R-CNN) to detect camphor trees in urban environments using RGB-DSM data. First, an attention fusion module was proposed to effectively fuse the RGB and DSM features by leveraging their complementarity. Second, the bidirectional feature pyramid network (BiFPN) was introduced to enhance the model performance in detecting camphor tree crowns of varying sizes. The results showed that our approach could effectively detect urban camphor trees and achieved an AP of 85.7%. Notably, our approach yielded an AP of 81.3% in urban green spaces. The analysis indicated that our approach was feasible for detecting camphor trees in urban areas and demonstrated its potential to facilitate urban forestry research and applications.

Generative artificial intelligence: a systematic review and applications

AbstractIn recent years, the study of artificial intelligence (AI) has undergone a paradigm shift. This has been propelled by the groundbreaking capabilities of generative models both in supervised and unsupervised learning scenarios. Generative AI has shown state-of-the-art performance in solving perplexing real-world conundrums in fields such as image translation, medical diagnostics, textual imagery fusion, natural language processing, and beyond. This paper documents the systematic review and analysis of recent advancements and techniques in Generative AI with a detailed discussion of their applications including application-specific models. Indeed, the major impact that generative AI has made to date, has been in language generation with the development of large language models, in the field of image translation and several other interdisciplinary applications of generative AI. Moreover, the primary contribution of this paper lies in its coherent synthesis of the latest advancements in these areas, seamlessly weaving together contemporary breakthroughs in the field. Particularly, how it shares an exploration of the future trajectory for generative AI. In conclusion, the paper ends with a discussion of Responsible AI principles, and the necessary ethical considerations for the sustainability and growth of these generative models.

Ensemble Machine Learning on the Fusion of Sentinel Time Series Imagery with High-Resolution Orthoimagery for Improved Land Use/Land Cover Mapping

In the United States, several land use and land cover (LULC) data sets are available based on satellite data, but these data sets often fail to accurately represent features on the ground. Alternatively, detailed mapping of heterogeneous landscapes for informed decision-making is possible using high spatial resolution orthoimagery from the National Agricultural Imagery Program (NAIP). However, large-area mapping at this resolution remains challenging due to radiometric differences among scenes, landscape heterogeneity, and computational limitations. Various machine learning (ML) techniques have shown promise in improving LULC maps. The primary purposes of this study were to evaluate bagging (Random Forest, RF), boosting (Gradient Boosting Machines [GBM] and extreme gradient boosting [XGB]), and stacking ensemble ML models. We used these techniques on a time series of Sentinel 2A data and NAIP orthoimagery to create a LULC map of a portion of Irion and Tom Green counties in Texas (USA). We created several spectral indices, structural variables, and geometry-based variables, reducing the dimensionality of features generated on Sentinel and NAIP data. We then compared accuracy based on random cross-validation without accounting for spatial autocorrelation and target-oriented cross-validation accounting for spatial structures of the training data set. Comparison of random and target-oriented cross-validation results showed that autocorrelation in the training data offered overestimation ranging from 2% to 3.5%. The XGB-boosted stacking ensemble on-base learners (RF, XGB, and GBM) improved model performance over individual base learners. We show that meta-learners are just as sensitive to overfitting as base models, as these algorithms are not designed to account for spatial information. Finally, we show that the fusion of Sentinel 2A data with NAIP data improves land use/land cover classification using geographic object-based image analysis.

Detection of Maize Crop Phenology Using Planet Fusion

Accurate identification of crop phenology timing is crucial for agriculture. While remote sensing tracks vegetation changes, linking these to ground-measured crop growth stages remains challenging. Existing methods offer broad overviews but fail to capture detailed phenological changes, which can be partially related to the temporal resolution of the remote sensing datasets used. The availability of higher-frequency observations, obtained by combining sensors and gap-filling, offers the possibility to capture more subtle changes in crop development, some of which can be relevant for management decisions. One such dataset is Planet Fusion, daily analysis-ready data obtained by integrating PlanetScope imagery with public satellite sensor sources such as Sentinel-2 and Landsat. This study introduces a novel method utilizing Dynamic Time Warping applied to Planet Fusion imagery for maize phenology detection, to evaluate its effectiveness across 70 micro-stages. Unlike singular template approaches, this method preserves critical data patterns, enhancing prediction accuracy and mitigating labeling issues. During the experiments, eight commonly employed spectral indices were investigated as inputs. The method achieves high prediction accuracy, with 90% of predictions falling within a 10-day error margin, evaluated based on over 3200 observations from 208 fields. To understand the potential advantage of Planet Fusion, a comparative analysis was performed using Harmonized Landsat Sentinel-2 data. Planet Fusion outperforms Harmonized Landsat Sentinel-2, with significant improvements observed in key phenological stages such as V4, R1, and late R5. Finally, this study showcases the method’s transferability across continents and years, although additional field data are required for further validation.

Vision transformer-enhanced thermal anomaly detection in building facades through fusion of thermal and visible imagery

ABSTRACT The thermal anomaly detection of building facades is critically important for the evaluation and upkeep of structures. While conventional methods are effective in numerous instances, their capability to discern flaws in complex environments is constrained. This research employs an optimized Vision Transformer (ViT) technology, enhanced by the Line Acquisition, Filtering, and Revision (LAFR) algorithm. By amalgamating the distinctive advantages of ViT’s attention-based mechanisms with the complementary data from thermal and visible imagery, this approach significantly bolsters the detection of thermal anomalies on building facades. The experimental outcomes reveal that our methodology, with an F1 score of 0.847, markedly surpasses traditional thermal imaging (F1 score of 0.466), showcasing its potential for precise anomaly detection in architectural contexts. This validates the method as a more robust and efficient solution for structural evaluation, promising to elevate the quality of architectural inspections and maintenance.

Fusion of multi-temporal PlanetScope data and very high-resolution aerial imagery for urban tree species mapping

Detailed assessment of the ecosystem services provided by urban green spaces requires data on urban tree species. While many approaches for mapping of urban trees from remotely sensed data have been proposed, the fusion of multi-temporal satellite imagery with very high resolution orthophotos remains relatively underexplored. In this research, we assess the potential of a multimodal deep learning approach with intermediate data fusion for classifying common tree species found in the Brussels Capital Region. Our method combines two image sources: (a) multi-temporal PlanetScope data and (b) high-resolution aerial imagery. To evaluate the contribution of each image source, we separately train and assess each branch of the network. Both image sources demonstrate the ability to predict prevalent tree species with high accuracy. However, the fusion of the two image sources yields the best results, achieving an overall accuracy of 0.88 for the five most common tree species in the region. Our approach is compared to two conventional machine learning methods: a random forest (RF) and a support vector machine classifier (SVM) and outperforms both with a 11 percentage point increase in overall accuracy over RF and a 10 percentage point increase over SVM. Increasing the number of species from five to thirteen, including all species with more than 500 tree samples, results in a marginal decrease in accuracy (from 0.88 to 0.84). Overall, our deep learning approach demonstrates its efficacy in classifying common tree species in urban settings and provides a foundation for a comprehensive quantification of ecosystem services offered by urban trees through remote sensing data.

Automated inpainting of Tex Willer comics: adapting manga techniques for western comic transformation

Comic inpainting is essential for language localization and adapting comics into animations, involving the removal of speech boxes, textboxes, and sound effects while maintaining artistic integrity. This paper introduces an automated approach leveraging the Manga inpainting method by Xie et al., adapted for western comics like Tex Willer. Tex Willer comics, known for their fusion of Italian art and American Western imagery, pose unique challenges due to their distinct styles. Customized Manga Restoration and Structural Line Extraction codes are employed, along with automated mask generation using deep CNN-based speech balloon detection and segmentation. Our contributions include the development of a precise model tailored to Tex Willer Comics, resulting in visually coherent inpainting results. The paper follows a structured format, covering State-of-the-Art, methodology, results, and conclusions, aiming to streamline the process of western comic inpainting for professionals in the comics and animation industries.

Identification and Positioning of Abnormal Maritime Targets Based on AIS and Remote-Sensing Image Fusion.

The identification of maritime targets plays a critical role in ensuring maritime safety and safeguarding against potential threats. While satellite remote-sensing imagery serves as the primary data source for monitoring maritime targets, it only provides positional and morphological characteristics without detailed identity information, presenting limitations as a sole data source. To address this issue, this paper proposes a method for enhancing maritime target identification and positioning accuracy through the fusion of Automatic Identification System (AIS) data and satellite remote-sensing imagery. The AIS utilizes radio communication to acquire multidimensional feature information describing targets, serving as an auxiliary data source to complement the limitations of image data and achieve maritime target identification. Additionally, the positional information provided by the AIS can serve as maritime control points to correct positioning errors and enhance accuracy. By utilizing data from the Jilin-1 Spectral-01 satellite imagery with a resolution of 5 m and AIS data, the feasibility of the proposed method is validated through experiments. Following preprocessing, maritime target fusion is achieved using a point-set matching algorithm based on positional features and a fuzzy comprehensive decision method incorporating attribute features. Subsequently, the successful fusion of target points is utilized for positioning error correction. Experimental results demonstrate a significant improvement in maritime target positioning accuracy compared to raw data, with over a 70% reduction in root mean square error and positioning errors controlled within 4 pixels, providing relatively accurate target positions that essentially meet practical requirements.

A satellite-field phenological bridging framework for characterizing community-level spring forest phenology using multi-scale satellite imagery

Forest phenology, as a sensitive indicator of a forest’s response to climate change and variability, has long been monitored using remote sensing, yet has seldom been interpreted or validated with spatially compatible, community-level field phenological observations. In temperate deciduous forests, multiple spring phenological phases are critical for modeling carbon storage and biogeochemical cycles. The simultaneous detection of all these critical phenological phases at the community level remains a long-standing challenge. To tackle these challenges, the objective of this study is to develop a novel satellite-field phenological bridging framework for characterizing all key spring phenological phases of a fragmented deciduous forest using multi-scale satellite time series. The framework consists of four key components: deep learning-based spatiotemporal image fusion, satellite-based forest phenology modeling, satellite-based forest phenological metric extraction, and field-based forest community phenological characterization. With the devised framework, we extract a total of 24 satellite phenological metrics of Trelease Woods, a forest fragment near Urbana, IL, USA. From weekly field phenological observations of 148 canopy trees of 15 common species of the forest community, we devise three summative field phenological indices to characterize community-level phenological states in spring. Under the satellite-field bridging framework, events during each key spring forest phenological phase (i.e., bud swell, budburst/leafing out, leaf expansion, and leaf maturity) are successfully detected using the fusion imagery (MAE from 1.1 to 2.9 days and bias from -2.4 to 1.5 days). However, the satellite detection of the earliest field events may be influenced by understory plants, soil background, and snow. The subsequent multi-scale, satellite phenological analysis underscores the importance of taking into account spatial scale and representation from both satellite and field phenological perspectives in building corresponding bridging relationships. Among the extracted pheno-metrics, bridging the threshold-based metrics to field phenological indices results in the highest accuracy (MAE less than 3 days and bias less than 2 days). The strong agreement among the field indices demonstrates the effectiveness of our field phenological surveying approach in generating community-wide forest phenological representation. Our study innovatively scales up the field phenological observations from the individual trees to the species to the community level, and the devised framework enables accurate retrieval of all key phenological events of community-wide, spring canopy development of the forest fragment.

A Joint Convolutional Cross ViT Network for Hyperspectral and Light Detection and Ranging Fusion Classification

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data for classification has received widespread attention and has led to significant progress in research and remote sensing applications. However, existing common CNN architectures suffer from the significant drawback of not being able to model remote sensing images globally, while transformer architectures are not able to capture local features effectively. To address these bottlenecks, this paper proposes a classification framework for multisource remote sensing image fusion. First, a spatial and spectral feature projection network is constructed based on parallel feature extraction by combining HSI and LiDAR data, which is conducive to extracting joint spatial, spectral, and elevation features from different source data. Furthermore, in order to construct local–global nonlinear feature mapping more flexibly, a network architecture coupling together multiscale convolution and a multiscale vision transformer is proposed. Moreover, a plug-and-play nonlocal feature token aggregation module is designed to adaptively adjust the domain offsets between different features, while a class token is employed to reduce the complexity of high-dimensional feature fusion. On three open-source remote sensing datasets, the performance of the proposed multisource fusion classification framework improves about 1% to 3% over other state-of-the-art algorithms.

YOLOv5n++: An edge-based improved YOLOv5n model to detect river floating debris

Rivers serve as vital water sources, maintain ecological equilibrium, and enhance landscapes. However, the looming issue of floating debris stemming from improper waste disposal and illegal discharge, poses an imminent threat to river ecosystems and their aesthetic appeal. Conventional human-led inspections prove labor-intensive, inefficient, and prone to errors. This study introduces an innovative approach for river debris detection, employing Unmanned Aerial Vehicles (UAVs) imagery in conjunction with a refined YOLOv5n model. This approach offers three key contributions. Primarily, the YOLOv5n model is bolstered by integrating the Efficient Channel Attention (ECA) module and reshaping the MobileNetV3 backbone to align with MobileNetV3S, thereby significantly streamlining computational demands and model intricacy. Additionally, precision and speed are augmented by eliminating the detection head for larger targets, while decreasing computational requirements. Subsequently, to counter dataset scarcity, we curate a UAV-derived river debris dataset, encompassing five prevalent debris types, serving as an indispensable resource for method refinement and assessment. Lastly, the upgraded model’s evaluation on Jetson Nano yields an mAP of 87.2%, merely 0.7% lower than the original YOLOv5n model. Remarkably, the refined model achieves substantial reductions of 57.1% in parameters, 52.6% in volume, and 54.8% in GFLOPs. Additionally, inference time is abbreviated to 57.3ms per Jetson Nano image, 13.4ms faster than the original. These findings underscore edge computing’s potential in river restoration. In conclusion, the fusion of deep learning object detection and UAV imagery empowers adept river debris detection.

A Comprehensive Assessment of the Pansharpening of the Nighttime Light Imagery of the Glimmer Imager of the Sustainable Development Science Satellite 1

The Sustainable Development Science Satellite 1 (SDGSAT-1) satellite, launched in November 2021, is dedicated to providing data detailing the “traces of human activities” for the implementation of the United Union’s 2030 Agenda for Sustainable Development and global scientific research. The glimmer imager (GI) that is equipped on SDGSAT-1 can provide nighttime light (NL) data with a 10 m panchromatic (PAN) band and red, green, and blue (RGB) bands of 40 m resolution, which can be used for a wide range of applications, such as in urban expansion, population studies of cities, and economics of cities, as well as nighttime aerosol thickness monitoring. The 10 m PAN band can be fused with the 40 m RGB bands to obtain a 10 m RGB NL image, which can be used to identify the intensity and type of night lights and the spatial distribution of road networks and to improve the monitoring accuracy of sustainable development goal (SDG) indicators related to city developments. Existing remote sensing image fusion algorithms are mainly developed for daytime optical remote sensing images. Compared with daytime optical remote sensing images, NL images are characterized by a large amount of dark (low-value) pixels and high background noises. To investigate whether daytime optical image fusion algorithms are suitable for the fusion of GI NL images and which image fusion algorithms are the best choice for GI images, this study conducted a comprehensive evaluation of thirteen state-of-the-art pansharpening algorithms in terms of quantitative indicators and visual inspection using four GI NL datasets. The results showed that PanNet, GLP_HPM, GSA, and HR outperformed the other methods and provided stable performances among the four datasets. Specifically, PanNet offered UIQI values ranging from 0.907 to 0.952 for the four datasets, whereas GSA, HR, and GLP_HPM provided UIQI values ranging from 0.770 to 0.856. The three methods based on convolutional neural networks achieved more robust and better visual effects than the methods using multiresolution analysis at the original scale. According to the experimental results, PanNet shows great potential in the fusion of SDGSAT-1 GI imagery due to its robust performance and relatively short training time. The quality metrics generated at the degraded scale were highly consistent with visual inspection, but those used at the original scale were inconsistent with visual inspection.

Land Suitability Assessment Based on Feature-Level Fusion of Sentinel-1 and Sentinel-2 Imagery: A Case Study of the Honam Region of Iran

Land Suitability Assessment Based on Feature-Level Fusion of Sentinel-1 and Sentinel-2 Imagery: A Case Study of the Honam Region of Iran

Utilizing Deep Learning-Based Fusion of Laser Point Cloud Data and Imagery for Digital Measurement in Steel Truss Member Applications

Utilizing Deep Learning-Based Fusion of Laser Point Cloud Data and Imagery for Digital Measurement in Steel Truss Member Applications

Research on Building AI Learning Dataset for Synthetic Aperture Radar Waterbody Detection through Optical Satellite Image Fusion

For the spatiotemporal analysis of water resources and disasters, water body detection using satellite imagery is crucial. Recently, AI-based methods have been widely employed in water body detection using satellite imagery. To use these AI techniques, a substantial amount of training data is required. When creating training data for water body detection, optical imagery and synthetic aperture radar (SAR) imagery have their respective strengths and weaknesses. To use the advantages of both, this study proposes a water body detection method through the fusion of optical and SAR imagery. The results of the proposed model show an Intersection over Union of 0.612 and an F1 score of 0.759, which is better compared to using either optical or SAR imagery alone. This research presents a method that can easily generate a large amount of water body data, making it promising for use as AI training data for water body detection.

Geospatial Artificial Intelligence (GeoAI) and Satellite Imagery Fusion for Soil Physical Property Predicting

This study aims to predict vital soil physical properties, including clay, sand, and silt, which are essential for agricultural management and environmental protection. Precision distribution of soil texture is crucial for effective land resource management and precision agriculture. To achieve this, we propose an innovative approach that combines Geospatial Artificial Intelligence (GeoAI) with the fusion of satellite imagery to predict soil physical properties. We collected 317 soil samples from Iran’s Golestan province for dependent data. The independent dataset encompasses 14 parameters from Landsat-8 satellite images, seven topographic parameters from the Shuttle Radar Topography Mission (SRTM) DEM, and two meteorological parameters. Using the Random Forest (RF) algorithm, we conducted feature importance analysis. We employed a Convolutional Neural Network (CNN), RF, and our hybrid CNN-RF model to predict soil properties, comparing their performance with various metrics. This hybrid CNN-RF network combines the strengths of CNN networks and the RF algorithm for improved soil texture prediction. The hybrid CNN-RF model demonstrated superior performance across metrics, excelling in predicting sand (MSE: 0.00003%, RMSE: 0.006%), silt (MSE: 0.00004%, RMSE: 0.006%), and clay (MSE: 0.00005%, RMSE: 0.007%). Moreover, the hybrid model exhibited improved precision in predicting clay (R2: 0.995), sand (R2: 0.992), and silt (R2: 0.987), as indicated by the R2 index. The RF algorithm identified MRVBF, LST, and B7 as the most influential parameters for clay, sand, and silt prediction, respectively, underscoring the significance of remote sensing, topography, and climate. Our integrated GeoAI-satellite imagery approach provides valuable tools for monitoring soil degradation, optimizing agricultural irrigation, and assessing soil quality. This methodology has significant potential to advance precision agriculture and land resource management practices.

Monitoring spring leaf phenology of individual trees in a temperate forest fragment with multi-scale satellite time series

Forest fragmentation has been increasingly exacerbated by deforestation, urbanization, and agricultural expansion. Monitoring the forest fragments via the lens of tree-crown scale leaf phenology is critical to understand tree species phenological responses to climate change and identify the fragment species vulnerable to environmental disturbance. Despite advances in remote sensing for phenology monitoring, detecting tree-crown scale leaf phenology in fragmented forests remains challenging. Simultaneous tracking of key spring phenological events that are crucial to ecosystem functions and climate change responses is also neglected. To address these challenges, we develop a novel tree-crown scale remote sensing phenological monitoring framework to characterize all the critical spring phenological events of individual trees of deciduous forest fragments, with Trelease Woods in Champaign, Illinois as a case study. The novel framework comprises four components: 1) generate high spatiotemporal resolution fusion imagery from multi-scale satellite time series with a hybrid deep learning fusion model; 2) calibrate PlanetScope imagery time series with fusion data using histogram matching; 3) model tree-crown scale phenology trajectory with a Beck logistic-based method; 4) detect a diversity of tree-crown scale phenological events using several phenological metric extraction methods (i.e., threshold- and curve feature-based methods). Combined with weekly in-situ phenological observations of 123 individual trees across 12 broadleaf species from 2017 to 2020, the framework effectively bridges the satellite- and field-based phenological measures for the key spring phenological events (i.e., budswell, budburst, leaf expansion, and leaf maturity events) at the tree-crown scale, particularly for large individuals (RMSE <1 week for most events). Calibration of PlanetScope imagery using multi-scale satellite fusion data in consideration of landscape fragmentation is critical for monitoring tree phenology of forest fragments. Compared to curve feature-based methods, threshold-based phenometric extraction methods demonstrate enhanced capability in detecting spring leaf phenological dynamics of individual trees. Among the phenological events, full leaf out and early leaf expansion events are retrieved with high accuracy using calibrated PlanetScope time series (RMSE from 3 to 5 days and R-squared higher than 0.8). With both intensive satellite and field phenological efforts, this novel framework is at the forefront of interpreting tree-crown scale remotely sensed phenological metrics in the context of biologically meaningful field phenological events in fragmented forest setting.

Consistent metropolitan boundaries for the remote sensing of urban land

This paper introduces an internationally consistent definition of metropolitan areas to the literature regarding the remote sensing of urban land use or land cover. In the cross-comparison of land use or land cover for explicitly bounded urban areas, the observed ‘economic’ definition is argued to hold distinct potential merits over administrative or agglomeration-based boundaries, which typically underpin other studies. To illustrate the proposed merits as well as their implications for the remote sensing literature, the empirical analysis considers the case of 687 European metropolitan areas. Across these metropolitan areas, whose boundaries are defined jointly by the OECD and the European Commission, land cover and land use are segmented in a fusion of imagery from radar and optical sensors in Sentinel satellites. Segmentation is achieved using deep learning in a well-established model architecture. The analytical focus is on built-up areas that are in a residential use or in a commercial or industrial use. Map classifications and accuracy measures are obtained for cities as well as their respective commuting zones as these together embody metropolitan areas. The results underline that not only land use area estimates but also map classification accuracy vary widely across individual metropolitan areas. Whereas classification accuracy to some degree varies for metropolitan areas within as well as between countries, classification accuracy is positively associated with population size and built-up area density as regression analysis confirms. Additionally, the extent of built-up areas in distinct uses is shown to vary across different types of metropolitan (sub-)areas. This study's findings highlight the typically unobserved role that study area definition and selection may play in affecting outcomes in remote sensing studies in urban settings, as relevant to both studies of single as well as multiple urban areas. The consistent comparison of remote sensing outcomes across metropolitan areas may further promote generalization in a growing and global field and potentially supports better-informed policy making processes.

Monitoring leaf nitrogen content in rice based on information fusion of multi-sensor imagery from UAV

Timely and accurately monitoring leaf nitrogen content (LNC) is essential for evaluating crop nutrition status. Currently, Unmanned Aerial Vehicles (UAV) imagery is becoming a potentially powerful tool of assessing crop nitrogen status in fields, but most of crop nitrogen estimates based on UAV remote sensing usually use single type imagery, the fusion information from different types of imagery has rarely been considered. In this study, the fusion images were firstly made from the simultaneously acquired digital RGB and multi-spectral images from UAV at three growth stages of rice, and then couple the selecting methods of optimal features with machine learning algorithms for the fusion images to estimate LNC in rice. Results showed that the combination with different types of features could improve the models’ accuracy effectively, the combined inputs with bands, vegetation indices (VIs) and Grey Level Co-occurrence Matrices (GLCMs) have the better performance. The LNC estimation of using fusion images was improved more obviously than multispectral those, and there was the best estimation at jointing stage based on Lasso Regression (LR), with R2 of 0.66 and RMSE of 11.96%. Gaussian Process Regression (GPR) algorithm used in combination with one feature-screening method of Minimum Redundancy Maximum Correlation (mRMR) for the fusion images, showed the better improvement to LNC estimation, with R2 of 0.68 and RMSE of 11.45%. It indicates that the information fusion from UAV multi-sensor imagery can significantly improve crop LNC estimates and the combination with multiple types of features also has a great potential for evaluating LNC in crops.

Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery

Field-scale crop yield prediction is critical to site-specific field management, which has been facilitated by recent studies fusing unmanned aerial vehicles (UAVs) based multimodal data. However, these studies equivalently stacked multimodal data and underused canopy spatial information. In this study, multimodal imagery fusion (MIF) attention was proposed to dynamically fuse UAV-based RGB, hyperspectral near-infrared (HNIR), and thermal imagery. Based on the MIF attention, a novel model termed MultimodalNet was proposed for field-scale yield prediction of winter wheat. To compare multimodal imagery-based and multimodal features-based methods, a stacking-based ensemble learning model was built using UAV-based canopy spectral, thermal, and texture features. The results showed that the MultimodalNet achieved accurate results at the reproductive stage and performed better than any single modality in the fusion. The MultimodalNet performed best at the flowering stage, with a coefficient of determination of 0.7411 and a mean absolute percentage error of 6.05%. The HNIR and thermal imagery were essential in yield prediction of winter wheat at the reproductive stage. Compared to equivalent stacking fusion, dynamic fusion through adaptively adjusting modality attention improved the model accuracy and adaptability across winter wheat cultivars and water treatments. Equivalently stacking more modalities did not necessarily yield improved performance than dynamically fusing fewer modalities. Methods using multimodal UAV imagery with rich spatial information were more applicable than methods using multimodal features to field-scale yield prediction. This study indicates that the MultimodalNet makes a powerful tool for field-scale yield prediction of winter wheat.