Flexible Spatiotemporal Data Fusion Research Articles

Comparison of change-based and shape-based data fusion methods in fine-resolution land surface phenology monitoring with Landsat and Sentinel-2 data

Fine-resolution land surface phenology (LSP) is urgently required for applications on agriculture management and vegetation-climate interaction, especially over heterogeneous areas, such as agricultural lands and fragmented forests. The critical challenge of fine-resolution LSP monitoring is how to reconstruct the spatiotemporal continuous vegetation index time series. To solve this problem, various data fusion methods have been devised; however, the comprehensive inter-comparison is lacking across different spatial heterogeneity, data quality, and vegetation types. We divide these methods into two main categories: the change-based methods fusing satellite observations with different spatiotemporal resolutions, and the shape-based methods fusing prior knowledge of shape models and satellite observations. We selected four methods to rebuilt two-band enhanced vegetation index (EVI2) series based on the harmonized Landsat and Sentinel-2 (HLS) data, including two change-based methods, namely the Spatial and temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), and two shape-based methods, namely the Multiple-year Weighting Shape-Matching (MWSM), and the Spatiotemporal Shape-Matching Model (SSMM). Four phenological transition dates were extracted, evaluated with PhenoCam observations and the 500 m Visible Infrared Imaging Radiometer Suite (VIIRS) phenology product. The 30 m transition dates show more spatial details and reveal more apparent intra-class and inter-class phenology variation compared with 500 m product. The four transition dates of SSMM and FSDAF (R2>0.74, MAD<15 days) show better agreement with PhenoCam-derived dates. The performance difference between fusion methods over various application scenarios are then analyzed. Fusion results are more robust when temporal frequency is higher than 15 observations per year. The shape-based methods are less sensitive to temporal sampling irregularity than change-based methods. Both change-based methods and shape-based methods cannot perform well when the region is heterogeneous. Among different vegetation types, SSMM-like methods have the highest overall accuracy. The findings in this paper can provide references for regional and global fine-resolution phenology monitoring.

MSISR-STF: Spatiotemporal Fusion via Multilevel Single-Image Super-Resolution

Due to technological limitations and budget constraints, spatiotemporal image fusion uses the complementarity of high temporal–low spatial resolution (HTLS) and high spatial–low temporal resolution (HSLT) data to obtain high temporal and spatial resolution (HTHS) fusion data, which can effectively satisfy the demand for HTHS data. However, some existing spatiotemporal image fusion models ignore the large difference in spatial resolution, which yields worse results for spatial information under the same conditions. Based on the flexible spatiotemporal data fusion (FSDAF) framework, this paper proposes a multilevel single-image super-resolution (SISR) method to solve this issue under the large difference in spatial resolution. The following are the advantages of the proposed method. First, multilevel super-resolution (SR) can effectively avoid the limitation of a single SR method for a large spatial resolution difference. In addition, the issue of noise accumulation caused by multilevel SR can be alleviated by learning-based SR (the cross-scale internal graph neural network (IGNN)) and then interpolation-based SR (the thin plate spline (TPS)). Finally, we add the reference information to the super-resolution, which can effectively control the noise generation. This method has been subjected to comprehensive experimentation using two authentic datasets, affirming that our proposed method surpasses the current state-of-the-art spatiotemporal image fusion methodologies in terms of performance and effectiveness.

STEPSBI: Quick spatiotemporal fusion with coarse- and fine-resolution scale transformation errors and pixel-based synthesis base image pair

Spatiotemporal fusion (STF) is crucial for reconciling the conflict between temporal and spatial resolutions of remote sensing observations. However, fusing images in heterogeneous areas remains challenging under continuous missing values. Moreover, most current STF methods only consider temporal errors and disregard the spatial scale error in time variation. Therefore, we proposed a Quick Spatiotemporal Fusion with Coarse- and Fine-Resolution Scale Transformation Errors and Pixel-Based Synthesis Base Image Pair (STEPSBI). First, the optimal pixel-based image synthesis strategy was designed using all available fine- and coarse-resolution images. Then, the scale transformation error (STE) of coarse-resolution downscaling to fine-resolution in temporal variation was quantified. And a residual term was introduced to reduce the prediction error from the temporal variation. Finally, the spatial scale and temporal errors were corrected using the results of super-pixel segmentation as spatial weights. This model has two strengths: (1) Pixel-based image synthesis alleviates the absence of base images under continuous missing values; and (2) STE correction restores spatial details of heterogeneous areas in rapid land cover change. In scenarios of continuous missing images, abrupt land cover changes, and high-resolution heterogeneity, we evaluated STEPSBI, the Gap Filling and Savitzky–Golay filtering method (GF-SG), the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), the cross-attention-based adaptive weighting fusion network (CAFE), and the Multi-scene Spatiotemporal Fusion Network (MUSTFN). The results indicate that STEPSBI yields better overall performance than other models in cropland, woodland, grassland, and other land cover types. Furthermore, ablation experiments demonstrated that each component improved the model’s performance. In addition, STEPSBI had higher fusion efficiency because it was developed on the Google Earth Engine cloud computing platform. Therefore, STEPSBI was feasible for advancing fine monitoring of spatiotemporal image fusion under continuous missing values and heterogeneous surfaces. The program of STEPBSI is freely available at: https://code.earthengine.google.com/684844aa42f64fa6b4eebe3bc0dd6483.

Bridging the Data Gap: Enhancing the Spatiotemporal Accuracy of Hourly PM2.5 Concentration through the Fusion of Satellite-Derived Estimations and Station Observations

Satellite-derived aerosol optical depth (AOD) has been extensively utilized for retrieving ground-level PM2.5 distributions. However, the presence of non-random missing data gaps in AOD poses a challenge to directly obtaining the gap-free AOD-derived PM2.5, thereby impeding accurate exposure risk assessment. Here, this study presents a novel and flexible framework that couples stacking and flexible spatiotemporal data fusion (FSDAF) approaches. By integrating multiple models and data sources, this framework aims to generate hourly (24-h) gap-free PM2.5 estimates for the Beijing–Tianjin–Hebei (BTH) region in 2018. This study effectively reconstructed data at least three times more effectively than the original AOD-derived PM2.5, achieving the Pearson coefficient (r), the coefficient determination (R2), root mean squared error (RMSE), and mean absolute error (MAE) values of 0.91, 0.84, 19.38 µg/m3, and 12.17 µg/m3, respectively, based on entire samples. Such strong predictive performance was also exhibited in spatial-based (r: 0.92–0.93, R2: 0.85–0.87, RMSE: 18.13 µg/m3–20.18 µg/m3, and MAE: 11.21 µg/m3–12.52 µg/m3) and temporal-based (r: 0.91–0.98, R2: 0.82–0.96, RMSE: 3.8 µg/m3–21.89 µg/m3, and MAE: 2.71 µg/m3–14.00 µg/m3) validations, indicating the robustness of this framework. Additionally, this framework enables the assessment of annual and seasonal PM2.5 concentrations and distributions, revealing that higher levels are experienced in the southern region, while lower levels prevail in the northern part. Winter exhibits the most severe levels, followed by spring and autumn, with comparatively lower levels in summer. Notably, the proposed framework effectively mitigates bias in calculating population-weighted exposure risk by filling data gaps with calculated values of 51.04 µg/m3, 54.17 µg/m3, 56.24 µg/m3, and 55.00 µg/m3 in Beijing, Tianjin, Hebei, and the BTH region, respectively.

Adaptability Evaluation of the Spatiotemporal Fusion Model in the Summer Maize Planting Area of the Southeast Loess Plateau

Precise regional crop yield estimates based on the high-spatiotemporal-resolution remote sensing data are essential for directing agronomic practices and policies to increase food security. This study used the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM), the flexible spatiotemporal data fusion (FSADF), and the spatial and temporal non-local filter based fusion model (STNLFFM) to calculate the normalized differential vegetation index (NDVI) of the summer maize planting area in the Southeast Loess Plateau based on the Sentinel-2 and MODIS data. The spatiotemporal resolution was 10 m and 1 d, respectively. Then, we evaluated the adaptability of the ESTARFM, FSADF, and STNLFFM fusion models in the field from the perspectives of spatial and textural characteristics of the data, summer maize NDVI growing curves, and yield estimation accuracy through qualitative visual discrimination and quantitative statistical analysis. The results showed that the fusion of ESTARFM–NDVI, FSDAF–NDVI, and STNLFFM–NDVI could precisely represent the variation tendency and local mutation information of NDVI during the growth period of summer maize, compared with MODIS–NDVI. The correlation between STNLFFM–NDVI and Sentinel-2–NDVI was favorable, with large correlation coefficients and a small root mean square error (RMSE). In the NDVI growing curve simulation of summer maize, STNLFFM introduced overall weights based on non-local mean filtering, which could significantly improve the poor fusion results at seedling and maturity stages caused by the long gap period of the high-resolution data in ESTARFM. Moreover, the accuracy of yield estimation was as follows (from high to low): STNLFFM (R = 0.742, mean absolute percentage error (MAPE) = 6.22%), ESTARFM (R = 0.703, MAPE = 6.80%), and FSDAF (R = 0.644, MAPE = 10.52%). The FADSF fusion model was affected by the spatial heterogeneity in the semi-humid areas, and the yield simulation accuracy was low. In the semi-arid areas, the FADSF fusion model had the advantages of less input data and a faster response.

High-Resolution Ratoon Rice Monitoring under Cloudy Conditions with Fused Time-Series Optical Dataset and Threshold Model

Ratoon rice, an effective rice cultivation system, allows paddy rice to be harvested twice from the same stubble, playing an important role in ensuring food security and adapting to climate change with its unique growth characteristics. However, there is an absence of research related to remote-sensing monitoring of ratoon rice, and the presence of other rice cropping systems (e.g., double-season rice) with similar characteristics poses a hindrance to the accurate identification of ratoon rice. Furthermore, cloudy and rainy regions have limited available remote-sensing images, meaning that remote-sensing monitoring is limited. To address this issue, taking Yongchuan District, a typical cloud-prone region in Chongqing, China, as an example, this study proposed the construction of a time-series optical dataset using the Modified Neighborhood Similar Pixel Interpolator (MNSPI) method for cloud-removal interpolation and the Flexible Spatiotemporal DAta Fusion (FSDAF) model for fusing multi-source optical remote-sensing data, in combination with vegetation index features and phenological information to build a threshold model to map ratoon rice at high-resolution (10 m). The mapping performance of ratoon rice was evaluated using independent field samples to obtain the overall accuracy and kappa coefficient. The findings indicate that the combination of the MNSPI method and FSDAF model had a stable and effective performance, characterized by high correlation coefficient (r) values and low root mean square error (RMSE) values between the restored/predicted images and the true images. Notably, it was possible to effectively capture the distinct characteristics of ratoon rice in cloudy and rainy regions using the proposed threshold model. Specifically, the identified area of ratoon rice in the study region was 194.17 km2, which was close to the official data (158–180 km2), and the overall accuracy and kappa coefficient of ratoon rice identification result were 90.73% and 0.81, respectively. These results demonstrate that our proposed threshold model can effectively distinguish ratoon rice during vital phenological stages from other crop types, enrich the technical system of rice remote-sensing monitoring, and provide a reference for agricultural remote-sensing applications in cloudy and rainy regions.

Revealing the ecological impact of low-speed mountain wind power on vegetation and soil erosion in South China: A case study of a typical wind farm in Yunnan

Due to the limited local electricity consumption in northeastern, northwestern, and northern China, developers are committed to unlocking wind resource-rich areas in the mountainous regions of southern China, leading to the southward movement of wind power. Construction processes destroy vegetation (especially forests) and may cause subsequent ecological impacts. However, due to the scarcity of cloudless remote sensing images in southern China, the ecosystem impact of low-speed wind farm construction in southern China remains unclear, which is not conducive to decision-making. This study used the flexible spatiotemporal data fusion (FSDAF) model to fuse the normalized difference vegetation index (NDVI) value obtained by Landsat 5 satellite Thematic Mapper (TM)/Landsat 8 satellite Operational Land Imager (OLI) and the NDVI value obtained by Moderate-resolution Imaging Spectroradiometer (MODIS) images. Cloudless and high-resolution (30 m) fusion results were used to establish a system framework for evaluating and quantifying the impact of low-speed mountain wind farms on vegetation and soil erosion. The results showed that wind farm construction led to a significant decline in mountain vegetation growth. The impact ranges of wind farm roads were mainly within the 0–30 m buffer zone (the reduction range of NDVI was 0.04–0.22), and the impact ranges of wind turbine foundations were mainly within the 0–60 m buffer zone (the reduction range of NDVI was 0.00–0.26). In addition, the NDVI in regions damaged by mountain wind farm construction was more sensitive to climate change. After vegetation damage, the synergistic effect of factors such as rain and terrain may lead to the doubling of soil erosion (compared to 2010, the increase in soil erosion could reach 1195.96%). In the study area, roads should be scientifically planned to reduce their length, and vegetation restoration and environmental supervision should be strengthened.

Fine-scale mapping of Spartina alterniflora-invaded mangrove forests with multi-temporal WorldView-Sentinel-2 data fusion

Mangrove forests, a critical component of blue carbon ecosystems, make a significant contribution to mitigating climate change and maintaining biodiversity. But in China, they have been suffering from severe invasion by Spartina alterniflora (S. alterniflora). Accurate information on spatial distribution of mangroves and S. alterniflora is a prerequisite to protecting indigenous mangrove ecosystems and managing S. alterniflora invasion. However, the classification performance is often constrained by two major issues: (1) A single-date image, regardless of acquisition time, hardly provides optimal spectral separability among evergreen mangroves, deciduous S. alterniflora and mudflats; (2) Due to the gradual invasion process, the mixtures between land cover classes are prevalent in moderate-resolution images. The use of high-resolution (finer than 2 m) image time series could effectively solve these two issues. However, due to frequent cloud cover in coastal areas and long revisit time of high-resolution satellite sensors, the collection of high-resolution image time series is still a great challenge, let alone substantial expenses. In this study, we proposed a new multi-temporal classification approach to fine-scale mapping of exotic S. alterniflora and native mangroves using synthetic high-resolution image time series derived from spatiotemporal fusion of WorldView-2 and Sentinel-2 data. A total of six synthetic WorldView-2-like images in different phenological periods were produced by the Flexible Spatiotemporal DAta Fusion (FSDAF) method. With the combination of the original and synthetic images, both types of image-based multi-temporal features and pixel-based image compositing features were extracted and evaluated in the following Random Forest classifications. Monte Carlo cross-validation was adopted to provide robust accuracy estimates, and a spatially explicit confidence map was generated to show classification uncertainty and potential errors. In order to further demonstrate the necessity of generating synthetic WorldView-2 images, the entire classification process was repeated using Sentinel-2 images at 10-m spatial resolution and the comparison between fine- and moderate-resolution classification results was made in terms of classification accuracies, classification maps and fractional abundances. Our results show that the spectral profiles of S. alterniflora and mangroves generated from the synthetic WorldView-2 images were consistent with the ones from the paired Sentinel-2 images. Meanwhile, the spatial details of high landscape heterogeneity, unrecognizable in the Sentinel-2 images, were revealed by the synthetic WorldView-2 images. Compared to the single-date classification with the original WorldView-2 image, most of multi-temporal classifications using both the original and synthetic images provided more accurate results, with the overall accuracy raised by up to 3.9%. Among various temporal features adopted in this study, the use of the image-based, stacked spectral reflectance and vegetation index features of all original and synthetic images produced the best classification performance, increasing the producer's and user's accuracies of mangroves, S. alterniflora and mudflats by 2.5%–7.1%. Finally, the Sentinel-2 classification accuracies were generally lower than the WorldView-2 counterparts across all temporal features, with a 6% difference in the highest overall accuracy between the two. Prediction errors of the moderate-resolution classification were often found in heterogeneous landscapes, where S. alterniflora was mixed with mangroves or mudflats. Considering the availability of moderate-resolution data with high revisit frequencies, such as Sentinel-2A/B and Landsat 8/9, future fine-scale mapping exercises might benefit from the multi-temporal classification approach with spatiotemporal fusion of moderate- and high-resolution data, alleviating the problem of the scarcity of high-resolution data in cloudy areas.

Adaptability Evaluation of the Spatiotemporal Fusion Model of Sentinel-2 and MODIS Data in a Typical Area of the Three-River Headwater Region

The study of surface vegetation monitoring in the “Three-River Headwaters” Region (TRHR) relies on satellite data with high spatial and temporal resolutions. The spatial and temporal fusion method for multiple data sources can effectively overcome the limitations of weather, the satellite return period, and funding on research data to obtain data higher spatial and temporal resolutions. This paper explores the spatial and temporal adaptive reflectance fusion model (STARFM), the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM), and the flexible spatiotemporal data fusion (FSDAF) method applied to Sentinel-2 and MODIS data in a typical area of the TRHR. In this study, the control variable method was used to analyze the parameter sensitivity of the models and explore the adaptation parameters of the Sentinel-2 and MODIS data in the study area. Since the spatiotemporal fusion model was directly used in the product data of the vegetation index, this study used NDVI fusion as an example and set up a comparison experiment (experiment I first performed the band spatiotemporal fusion and then calculated the vegetation index; experiment II calculated the vegetation index first and then performed the spatiotemporal fusion) to explore the feasibility and applicability of the two methods for the vegetation index fusion. The results showed the following. (1) The three spatiotemporal fusion models generated high spatial resolution and high temporal resolution data based on the fusion of Sentinel-2 and MODIS data, the STARFM and FSDAF model had a higher fusion accuracy, and the R2 values after fusion were higher than 0.8, showing greater applicability. (2) The fusion accuracy of each model was affected by the model parameters. The errors between the STARFM, ESTARFM, and FSDAF fusion results and the validation data all showed a decreasing trend with an increase in the size of the sliding window or the number of similar pixels, which stabilized after the sliding window became larger than 50 and the similar pixels became larger than 80. (3) The comparative experimental results showed that the spatiotemporal fusion model can be directly fused based on the vegetation index products, and higher quality vegetation index data can be obtained by calculating the vegetation index first and then performing the spatiotemporal fusion. The high spatial and temporal resolution data obtained using a suitable spatial and temporal fusion model are important for the identification and monitoring of surface cover types in the TRHR.

STFRDN: a residual dense network for remote sensing image spatiotemporal fusion

ABSTRACT Convolutional Neural Networks (CNN) are useful models for spatiotemporal fusion, especially under strong temporal changes. Convolutional layers with receptive fields of various sizes produce hierarchical features that can improve prediction performance. In this study, we aimed to develop a CNN model for spatiotemporal fusion, which can extract hierarchical features and fully use them. The proposed network is composed of residual dense blocks with local dense connections that effectively utilize the hierarchical features. We conducted the experiments in the newly proposed Kansas dataset, which includes Sentinel-2 and Sentinel-3 image pairs with large resolution differences and strong temporal changes. Based on both quantitative and qualitative evaluation, experiments revealed that the STFRDN model outperformed the Flexible Spatiotemporal DAta Fusion (FSDAF) 2.0, Reliable and Adaptive Spatiotemporal Data Fusion (RASDF), Fit-FC, and DMNet methods in all of the test groups. The second major finding was that the STFRDN model was able to produce sufficiently successful predictions against cases with such large resolution differences. To get more accurate prediction results when there is a significant resolution difference, it is important to extract the right features that can model the discrepancy. Loss of spatial detail can be avoided by making full use of hierarchical features without any manipulation after extraction. The results of this study provide insights into the significance of extracting and fully utilizing the hierarchical features.

An Advanced Spatiotemporal Fusion Model for Suspended Particulate Matter Monitoring in an Intermontane Lake

Ebinur Lake is the largest brackish-water lake in Xinjiang, China. Strong winds constantly have an impact on this shallow water body, causing high variability in turbidity of water. Therefore, it is crucial to continuously monitor suspended particulate matter (SPM) for water quality management. This research aims to develop an advanced spatiotemporal fusion model based on the inversion technique that enables time-continuous and detailed monitoring of SPM over an intermontane lake. The findings shows that: (1) the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM) fusion in blue, green, red, and near infrared (NIR) bands was better than the flexible spatiotemporal data fusion (FSDAF) model in extracting SPM information; (2) the inversion model constructed by random forest (RF) outperformed the support vector machine (SVM) and partial least squares (PLS) algorithms; and (3) the SPM concentrations acquired from the fused images of Landsat 8 OLI and ESTARFM matched with the actual data of Ebinur Lake based on the visual perspective and accuracy assessment.

Reconstruction of Sentinel Images for Suspended Particulate Matter Monitoring in Arid Regions

Missing data is a common issue in remote sensing. Data reconstruction through multiple satellite data sources has become one of the most powerful ways to solve this issue. Continuous monitoring of suspended particulate matter (SPM) in arid lakes is vital for water quality solutions. Therefore, this research aimed to develop and evaluate the performance of two image reconstruction strategies, spatio-temporal fusion reflectance image inversion SPM and SPM spatio-temporal fusion, based on the measured SPM concentration data with Sentinel-2 and Sentinel-3. The results show that (1) ESTARFM (Enhanced Spatio-temporal Adaptive Reflection Fusion Model) performed better than FSDAF (Flexible Spatio-temporal Data Fusion) in the fusion image generation, particularly the red band, followed by the blue, green, and NIR (near-infrared) bands. (2) A single-band linear and non-linear regression model was constructed based on Sentinel-2 and Sentinel-3. Analysis of the accuracy and stability of the model led us to the conclusion that the red band model performs well, is fast to model, and has a wide range of applications (Sentinel-2, Sentinel-3, and fused high-accuracy images). (3) By comparing the two data reconstruction strategies of spatio-temporal fused image inversion SPM and spatio-temporal fused SPM concentration map, we found that the fused SPM concentration map is more effective and more stable when applied to multiple fused images. The findings can provide an important scientific reference value for further expanding the inversion research of other water quality parameters in the future and provide a theoretical basis as well as technical support for the scientific management of Ebinur Lake’s ecology and environment.

Spatiotemporal fusion of multi-source remote sensing data for estimating aboveground biomass of grassland

Accurate estimation of aboveground biomass of grasslands is key to sustainable grassland utilization. However, most satellites cannot provide high temporal and spatial resolution data. Patterns of grassland dynamics associated with variability in climate conditions across spatiotemporal scales are yet to be adequately quantified. A spatiotemporal fusion model offers the opportunity to combine the resolution advantages of different remote sensing data to achieve a high frequency and high precision monitoring of vegetation. We test a flexible spatiotemporal data fusion (FSDAF) methodology to generate synthetic normalized difference vegetation index (NDVI) data from Moderate-Resolution Imaging Spectroradiometer (MODIS) and Landsat data sets. The methodology is tested for the semi-arid grassland of the Xilin River Basin, China. Based on NDVI data fusion and field measured aboveground biomass an aboveground biomass estimation model is established for the watershed. Exploring the temporal and spatial changes of biomass and its relationship with environmental factors. The results show that: (1) The FSDAF model performs well (R2 = 0.75) and has clear textural features. (2) The established Support Vector Machine Aboveground Biomass model not only ensured the accuracy of estimation (R2 = 0.78, RMSE = 15.43 g/m2), but also generated spatiotemporal maps of biomass with higher spatial (30 m) and temporal resolution (8 days). (3) The grassland aboveground biomass in this area decreases from southeast to northwest, and the grassland biomass reaches its peak at the end of July. The average biomass of different grasslands decreases in the order of meadow grassland > typical grassland > desert grassland. (4) Aboveground biomass increased linearly with increasing water content, organic carbon and total nitrogen, and was most sensitive to soil water content. During the early growing and rapid growing period, aboveground biomass is mainly affected by both air temperature and precipitation, while the effects of temperature and human activities gradually dominate in the middle and late growing periods. This study helps to improve the spatial and temporal resolution of dynamic monitoring of grassland biomass, and provides a scientific basis for grassland protection and management in arid and semi-arid regions.

Tree Species Classification over Cloudy Mountainous Regions by Spatiotemporal Fusion and Ensemble Classifier

Accurate mapping of tree species is critical for the sustainable development of the forestry industry. However, the lack of cloud-free optical images makes it challenging to map tree species accurately in cloudy mountainous regions. In order to improve tree species identification in this context, a classification method using spatiotemporal fusion and ensemble classifier is proposed. The applicability of three spatiotemporal fusion methods, i.e., the spatial and temporal adaptive reflectance fusion model (STARFM), the flexible spatiotemporal data fusion (FSDAF), and the spatial and temporal nonlocal filter-based fusion model (STNLFFM), in fusing MODIS and Landsat 8 images was investigated. The fusion results in Helong City show that the STNLFFM algorithm generated the best fused images. The correlation coefficients between the fusion images and actual Landsat images on May 28 and October 19 were 0.9746 and 0.9226, respectively, with an average of 0.9486. Dense Landsat-like time series at 8-day time intervals were generated using this method. This time series imagery and topography-derived features were used as predictor variables. Four machine learning methods, i.e., K-nearest neighbors (KNN), random forest (RF), artificial neural networks (ANNs), and light gradient boosting machine (LightGBM), were selected for tree species classification in Helong City, Jilin Province. An ensemble classifier combining these classifiers was constructed to further improve the accuracy. The ensemble classifier consistently achieved the highest accuracy in almost all classification scenarios, with a maximum overall accuracy improvement of approximately 3.4% compared to the best base classifier. Compared to only using a single temporal image, utilizing dense time series and the ensemble classifier can improve the classification accuracy by about 20%, and the overall accuracy reaches 84.32%. In conclusion, using spatiotemporal fusion and the ensemble classifier can significantly enhance tree species identification in cloudy mountainous areas with poor data availability.

Applicability Analysis of GF-2PMS and PLANETSCOPE Data for Ground Object Recognition in Karst Region

Remote sensing image with high spatial and temporal resolution is very important for rational planning and scientific management of land resources. However, due to the influence of satellite resolution, revisit period, and cloud pollution, it is difficult to obtain high spatial and temporal resolution images. In order to effectively solve the “space–time contradiction” problem in remote sensing application, based on GF-2PMS (GF-2) and PlanetSope (PS) data, this paper compares and analyzes the applicability of FSDAF (flexible spatiotemporal data fusion), STDFA (the spatial temporal data fusion approach), and Fit_FC (regression model fitting, spatial filtering, and residual compensation) in different terrain conditions in karst area. The results show the following. (1) For the boundary area of water and land, the FSDAF model has the best fusion effect in land boundary recognition, and provides rich ground object information. The Fit_FC model is less effective, and the image is blurry. (2) For areas such as mountains, with large changes in vegetation coverage, the spatial resolution of the images fused by the three models is significantly improved. Among them, the STDFA model has the clearest and richest spatial structure information. The fused image of the Fit_FC model has the highest similarity with the verification image, which can better restore the coverage changes of crops and other vegetation, but the actual spatial resolution of the fused image is relatively poor, the image quality is fuzzy, and the land boundary area cannot be clearly identified. (3) For areas with dense buildings, such as cities, the fusion image of the FSDAF and STDFA models is clearer and the Fit_FC model can better reflect the changes in land use. In summary, compared with the Fit_FC model, the FSDAF model and the STDFA model have higher image prediction accuracy, especially in the recognition of building contours and other surface features, but they are not suitable for the dynamic monitoring of vegetation such as crops. At the same time, the image resolution of the Fit_FC model after fusion is slightly lower than that of the other two models. In particular, in the water–land boundary area, the fusion accuracy is poor, but the model of Fit_FC has unique advantages in vegetation dynamic monitoring. In this paper, three spatiotemporal fusion models are used to fuse GF-2 and PS images, which improves the recognition accuracy of surface objects and provides a new idea for fine classification of land use in karst areas.

MUSTFN: A spatiotemporal fusion method for multi-scale and multi-sensor remote sensing images based on a convolutional neural network

• A deep learning spatiotemporal fusion method for large area remote sensing images. • Multi-constraint loss function to improve model accuracy. • Multi-scale features to avoid the influence of different sensor registration errors on fusion results. • Model is suitable for drastically changing remote sensing image fusion scenarios. • Model has strong transferability for remote sensing images at different times and locations. Spatiotemporal data fusion is a commonly-used and well-proven technique to enhance the application potential of multi-source remote sensing images. However, most existing methods have trouble in generating quality fusion results when areas covered by the images undergoes rapid land cover changes or images have substantial registration errors. While deep learning algorithms have demonstrated their capabilities for imagery fusion, it is challenging to apply deep-learning-based fusion methods in regions that experiences persistent cloud covers and have limited cloud-free imagery observations. To address these challenges, we developed a Multi-scene Spatiotemporal Fusion Network (MUSTFN) algorithm based on a Convolutional Neural Network (CNN). Our approach uses multi-level features to fuse images at different resolutions acquired by multiple sensors. Furthermore, MUSTFN uses the multi-scale features to overcome the effects of geometric registration errors between different images. Additionally, a multi-constrained loss function is proposed to improve the accuracy of imagery fusion over large areas and solve fusion and gap-filling problems simultaneously by utilizing cloud-contaminated images with the fine-tuning method. Compared with several commonly-used methods, our proposed MUSTFN performs better in fusing the 30-m Landsat-7 images and 500-m MODIS images over a small area that has undergone large changes (the average relative Mean Absolute Errors (rMAE) of the first four bands are 6.8% by MUSTFN as compared to 14.1% by the Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM), 12.8% by the Flexible Spatiotemporal Data Fusion (FSDAF), 8.4% by the Extended Super-Resolution Convolutional Neural Network (ESRCNN), 8.1% by the Spatiotemporal Fusion Using a Generative Adversarial Network (STFGAN)). In particularly for images at different resolutions with different registration accuracies (e.g., 16-m Chinese GaoFen-1 and 500-m MODIS), MUSTFN achieved fusion results of good quality with an average rMAE of 9.3% in spectral reflectance at the first four bands. Finally, we demonstrated the applicability of MUSTFN (average rMAE of 9.18%) when fusing long-term Landsat-8 composite images and MODIS images over a large region (830 km × 600 km). Overall, our results suggest the effectiveness of MUSTFN to address the challenges in imagery fusion, including rapid land cover changes between image acquisition dates, geometric misregistration between images and limited availabilities of cloud-free images. The program of MUSTFN is freely available at: https://github.com/qpyeah/MUSTFN .

High-Resolution Monitoring of the Snow Cover on the Moroccan Atlas through the Spatio-Temporal Fusion of Landsat and Sentinel-2 Images

Mapping seasonal snow cover dynamics provides essential information to predict snowmelt during spring and early summer. Such information is vital for water supply management and regulation by national stakeholders. Recent advances in remote sensing have made it possible to reliably estimate and quantify the spatial and temporal variability of snow cover at different scales. However, because of technological constraints, there is a compromise between the temporal, spectral, and spatial resolutions of available satellites. In addition, atmospheric conditions and cloud contamination may increase the number of missing satellite observations. Therefore, data from a single satellite is insufficient to accurately capture snow dynamics, especially in semi-arid areas where snowfall is extremely variable in both time and space. Considering these limitations, the combined use of the next generation of multispectral sensor data from the Landsat-8 (L8) and Sentinel-2 (S2), with a spatial resolution ranging from 10 to 30 m, provides unprecedented opportunities to enhance snow cover mapping. Hence, the purpose of this study is to examine the effectiveness of the combined use of optical sensors through image fusion techniques for capturing snow dynamics and producing detailed and dense normalized difference snow index (NDSI) time series within a semi-arid context. Three different models include the enhanced spatial and temporal adaptive reflectance fusion model (ESTARFM), the flexible spatio-temporal data fusion model (FSDAF), and the pre-classification flexible spatio-temporal data fusion model (pre-classification FSDAF) were tested and compared to merge L8 and S2 data. The results showed that the pre-classification FSDAF model generates the most accurate precise fused NDSI images and retains spatial detail compared to the other models, with the root mean square error (RMSE = 0.12) and the correlation coefficient (R = 0.96). Our results reveal that, the pre-classification FSDAF model provides a high-resolution merged snow time series and can compensate the lack of ground-based snow cover data.

VSDF: A variation-based spatiotemporal data fusion method

The fusion of spatiotemporal data provides the possibility to improve both the spatial and temporal resolution of remote sensing data. Nevertheless, the performance of current spatiotemporal data fusion methods is affected by several aspects, e.g., (1) retrieval of abrupt land cover changes, (2) recovery of detailed spatial information, and (3) the need to reduce side effects related to the performance differences between sensors. Concerning the above aspects, this study proposes the use of a Variation-based Spatiotemporal Data Fusion (VSDF) method. In VSDF, an abundant variation classification (AVC) is used to identify explicitly the land cover changes for spectral unmixing. In addition, feature-level fusion is introduced to strengthen the spatial structures, i.e., the edges and texture. Furthermore, a relative reliability index (RRI) is proposed to guide the prediction process to lower the uncertainty of the input datasets. With reference to the all-round performance assessment (APA) metrics, the performance of VSDF was compared with five popular methods, i.e., Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), Flexible Spatiotemporal DAta Fusion (FSDAF), FSDAF 2.0, Reliable and Adaptive Spatiotemporal Data Fusion (RASDF), and Fit-FC (regression model Fitting, spatial Filtering and residual Compensation). The experimental results demonstrated that VSDF can realize a more accurate prediction of temporal land cover changes with better spatial detail than the benchmark methods. Consequently, VSDF has the potential to generate accurate high-spatiotemporal-resolution simulations for global remote sensing studies.

An Object-Based Weighting Approach to Spatiotemporal Fusion of High Spatial Resolution Satellite Images for Small-Scale Cropland Monitoring

Continuous crop monitoring often requires a time-series set of satellite images. Since satellite images have a trade-off in spatial and temporal resolution, spatiotemporal image fusion (STIF) has been applied to construct time-series images at a consistent scale. With the increased availability of high spatial resolution images, it is necessary to develop a new STIF model that can effectively reflect the properties of high spatial resolution satellite images for small-scale crop field monitoring. This paper proposes an advanced STIF model using a single image pair, called high spatial resolution image fusion using object-based weighting (HIFOW), for blending high spatial resolution satellite images. The four-step weighted-function approach of HIFOW includes (1) temporal relationship modeling, (2) object extraction using image segmentation, (3) weighting based on object information, and (4) residual correction to quantify temporal variability between the base and prediction dates and also represent both spectral patterns at the prediction date and spatial details of fine-scale images. The specific procedures tailored for blending fine-scale images are the extraction of object-based change and structural information and their application to weight determination. The potential of HIFOW was evaluated from the experiments on agricultural sites using Sentinel-2 and RapidEye images. HIFOW was compared with three existing STIF models, including the spatial and temporal adaptive reflectance fusion model (STARFM), flexible spatiotemporal data fusion (FSDAF), and Fit-FC. Experimental results revealed that the HIFOW prediction could restore detailed spatial patterns within crop fields and clear crop boundaries with less spectral distortion, which was not represented in the prediction results of the other three models. Consequently, HIFOW achieved the best prediction performance in terms of accuracy and structural similarity for all the spectral bands. Other than the reflectance prediction, HIFOW also yielded superior prediction performance for blending normalized difference vegetation index images. These findings indicate that HIFOW could be a potential solution for constructing high spatial resolution time-series images in small-scale croplands.

Remote Monitoring of NH3-N Content in Small-Sized Inland Waterbody Based on Low and Medium Resolution Multi-Source Remote Sensing Image Fusion

In applying quantitative remote sensing in water quality monitoring for small inland rivers, the time-frequency of monitoring dramatically impacts the accuracy of time-spatial changes estimates of the water quality parameters. Due to the limitation of satellite sensor design and the influence of atmospheric conditions, the number of spatiotemporal dynamic monitoring images of water quality parameters is insufficient. Meanwhile, MODIS and other high temporal resolution images’ spatial resolution is too low to effectively extract small inland river boundaries. To solve the problem, many researchers used Spatio-temporal fusion models in multisource data remote sensing monitoring of ground features. The wildly used Spatio-temporal fusion models, such as FSDAF (flexible spatial-temporal data fusion), have poor performance in heterogeneous changes of ground objects. We proposed a spatiotemporal fusion algorithm SR-FSDAF (Super-resolution based flexible spatiotemporal data fusion) to solve the problem. Based on the FSDAF, it added ESPCN to reconstruct the spatial change prediction image, so as to obtain better prediction results for heterogeneous changes. Both qualitative and quantitative evaluation results showed that our fusion algorithm obtained better results. We compared the band sensitivity of the images before and after fusion to find out that the sensitive band combination of NH3-N has not changed, which proved that the fusion method can be used to improve the time-frequency of NH3-N inversion. After the fusion, we compared the accuracy of linear regression and random forest inversion models and selected the random forest model with better accuracy to predict the NH3-N concentration. The inversion accuracy of NH3-N was as follows: the R2 was 0.75, the MAPE was 23.7% and the RMSE was 0.15. The overall concentration change trend of NH3-N in the study area was high-water period &lt; water-stable period &lt; low water period. NH3-N pollution was serious in some reaches.