Spatiotemporal Fusion Method Research Articles

Efficient fusion of spatio-temporal saliency for frame wise saliency identification

<p>Video saliency detection is a rapidly growing subject that has seen very few contributions. The most common technique used nowadays is to perform frame-by-frame saliency detection. The modified Spatio-temporal fusion method presented in this paper offers a novel approach to saliency detection and mapping. It uses frame-wise overall motion color saliency as well as pixel-based consistent Spatio-temporal diffusion for its temporal uniformity. Additionally, a variety of techniques is advocated as a way to increase the saliency maps' overall accuracy and precision. The video is divided into groups of frames, and each frame temporarily goes through diffusion and integration in order to compute the color saliency mapping, as covered in the proposed method section. Then, with the aid of a permutation matrix, the inter-group frame is used to format the pixel-based saliency fusion, after which the features, or the fusion of pixel saliency and color information, direct the diffusion of the spatiotemporal saliency. The result is tested using five publicly accessible global saliency evaluation metrics, and it is determined that the proposed algorithm outperforms numerous saliency detection techniques with an improvement in accuracy margin. The robustness, dependability, adaptability, and precision are all demonstrated by the results.</p>

A fast adaptive spatio-temporal fusion method to enhanced Fit-FC.

Space-time fusion is an economical and efficient way to solve "space-time contradiction". Among all kinds of space-time fusion methods, Fit-FC space-time fusion method based on weight Function is widely used. However, this method is based on the linear model to depict the phase change, but the phase change in the real scene is complicated, and the linear model is difficult to accurately capture the phase change, resulting in the spectral distortion of the fusion image. In addition, pixel-by-pixel scanning with moving Windows leads to inefficiency issues, limiting its use in large-scale and long-term tasks. To overcome these limitations, this paper developed a simple and fast adaptive remote sensing image Spatio-Temporal fusion method based on Fit-FC, called Adapt Lasso-Fit-FC (AL-FF). Firstly, the sparse characteristics of time phase change between images are explored, and a time phase change estimation model based on sparse regression is constructed, which overcomes the fuzzy problem of fusion image caused by the failure of linear regression to capture complex nonlinear time phase transition in the weighted Function method, making the algorithm better at capturing details. Secondly, an adaptive window selection Function is established to overcome the problem of manually setting parameters on different data sets, improve the convenience of the algorithm and robustness of the application on different data sets, and make the algorithm simpler and more efficient. Finally, the improved AL-FF algorithm is compared with other algorithms to verify the performance improvement. Compared with the current advanced Spatio-Temporal fusion methods, AL-FF algorithm has stronger detail capture ability and can generate more accurate fusion results. In addition, the computational efficiency is significantly improved, and the efficiency is increased by more than 20 times compared with the current mainstream method.

An Improved Gap-Filling Method for Reconstructing Dense Time-Series Images from LANDSAT 7 SLC-Off Data

Over recent decades, Landsat satellite data has evolved into a highly valuable resource across diverse fields. Long-term satellite data records with integrity and consistency, such as the Landsat series, provide indispensable data for many applications. However, the malfunction of the Scan Line Corrector (SLC) on the Landsat 7 satellite in 2003 resulted in stripping in subsequent images, compromising the temporal consistency and data quality of Landsat time-series data. While various methods have been proposed to improve the quality of Landsat 7 SLC-off data, existing gap-filling methods fail to enhance the temporal resolution of reconstructed images, and spatiotemporal fusion methods encounter challenges in managing large-scale datasets. Therefore, we propose a method for reconstructing dense time series from SLC-off data. This method utilizes the Neighborhood Similar Pixel Interpolator to fill in missing values and leverages the time-series information to reconstruct high-resolution images. Taking the blue band as an example, the surface reflectance verification results show that the Mean Absolute Error (MAE) and BIAS reach minimum values of 0.0069 and 0.0014, respectively, with the Correlation Coefficient (CC) and Structural Similarity Index Metric (SSIM) reaching 0.93 and 0.94. The proposed method exhibits advantages in repairing SLC-off data and reconstructing dense time-series data, enabling enhanced remote sensing applications and reliable Earth’s surface reflectance data reconstruction.

A Hybrid Spatio-Temporal Fusion Method for High Spatial Resolution Imagery: Fusion of Gaofen-1 and Sentinel-2 over Agricultural Landscapes

A Hybrid Spatio-Temporal Fusion Method for High Spatial Resolution Imagery: Fusion of Gaofen-1 and Sentinel-2 over Agricultural Landscapes

Filling gaps in cloudy Landsat LST product by spatial-temporal fusion of multi-scale data

Land surface temperature (LST) is an important factor in studies of surface energy fluxes between the Earth's surface and atmosphere. The Landsat LST product has been applied widely due to its fine spatial resolution and high data quality. Frequent cloud coverage, however, results in different degrees of gaps in the Landsat LST images, restricting greatly their application in practical cases requiring continuous LST time-series. Gap filling is an important technique used to reconstruct the missing data in the Landsat LST images. Using only temporally neighboring data from the same sensor, however, current gap filling methods struggle to deal with large spatial gaps and large temporal changes in LST. To solve these challenging issues, in this paper, multi-scale data (i.e., the 2 km, all-sky, hourly GOES LST product) were proposed for gap filling of Landsat LST images, and a spatial-temporal random forest (STRF) method was developed. The GOES LST product can provide effective information on LST in the gaps of the target Landsat LST data, with great potential to decreasing the uncertainty in gap filling when large LST change exists or the gap size is large. Through experiments in six regions of the USA, the proposed STRF method was shown to be more accurate than three typical gap filling and spatio-temporal fusion methods, and particularly advantageous for cases involving large temporal changes or large spatial gaps in LST. Specifically, STRF produces an average root mean square error (RMSE) of 1.6444 K and average correlation coefficient (CC) of 0.8653 for the six regions, which are 0.1133 K smaller and 0.0176 larger than the most competitive benchmark method (i.e., the spatial random forest (SRF) method). Conceptually, by spatial-temporal fusion of multi-scale data in gap filling, STRF provides a new paradigm for both spatial-temporal fusion and gap filling. Furthermore, the proposed STRF has the potential to be applied to various gap filling tasks in other situations.

Video saliency-recognition by applying custom spatio temporal fusion technique

<span lang="EN-US">Video saliency detection is a major growing field with quite few contributions to it. The general method available today is to conduct frame wise saliency detection and this leads to several complications, including an incoherent pixel-based saliency map, making it not so useful. This paper provides a novel solution to saliency detection and mapping with its custom spatio-temporal fusion method that uses frame wise overall motion colour saliency along with pixel-based consistent spatio-temporal diffusion for its temporal uniformity. In the proposed method section, it has been discussed how the video is fragmented into groups of frames and each frame undergoes diffusion and integration in a temporary fashion for the colour saliency mapping to be computed. Then the inter group frame are used to format the pixel-based saliency fusion, after which the features, that is, fusion of pixel saliency and colour information, guide the diffusion of the spatio temporal saliency. With this, the result has been tested with 5 publicly available global saliency evaluation metrics and it comes to conclusion that the proposed algorithm performs better than several state-of-the-art saliency detection methods with increase in accuracy with a good value margin. All the results display the robustness, reliability, versatility and accuracy.</span>

Separable ConvNet Spatiotemporal Mixer for Action Recognition

Video action recognition is vital in the research area of computer vision. In this paper, we develop a novel model, named Separable ConvNet Spatiotemporal Mixer (SCSM). Our goal is to develop an efficient and lightweight action recognition backbone that can be applied to multi-task models to increase the accuracy and processing speed. The SCSM model uses a new hierarchical spatial compression, employing the spatiotemporal fusion method, consisting of a spatial domain and a temporal domain. The SCSM model maintains the independence of each frame in the spatial domain for feature extraction and fuses the spatiotemporal features in the temporal domain. The architecture can be adapted to different frame rate requirements due to its high scalability. It is suitable to serve as a backbone for multi-task video feature extraction or industrial applications with its low prediction and training costs. According to the experimental results, SCSM has a low number of parameters and low computational complexity, making it highly scalable with strong transfer learning capabilities. The model achieves video action recognition accuracy comparable to state-of-the-art models with a smaller parameter size and fewer computational requirements.

Effectiveness of Spatiotemporal Data Fusion in Fine-Scale Land Surface Phenology Monitoring: A Simulation Study

Spatiotemporal data fusion technologies have been widely used for land surface phenology (LSP) monitoring since it is a low-cost solution to obtain fine-resolution satellite time series. However, the reliability of fused images is largely affected by land surface heterogeneity and input data. It is unclear whether data fusion can really benefit LSP studies at fine scales. To explore this research question, this study designed a sophisticated simulation experiment to quantify effectiveness of 2 representative data fusion algorithms, namely, pair-based Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and time series-based Spatiotemporal fusion method to Simultaneously generate Full-length normalized difference vegetation Index Time series (SSFIT) by fusing Landsat and Moderate Resolution Imaging Spectroradiometer (MODIS) data in extracting pixel-wise spring phenology (i.e., the start of the growing season, SOS) and its spatial gradient and temporal variation. Our results reveal that: (a) STARFM can improve the accuracy of pixel-wise SOS by up to 74.47% and temporal variation by up to 59.13%, respectively, compared with only using Landsat images, but it can hardly improve the retrieval of spatial gradient. For SSFIT, the accuracy of pixel-wise SOS, spatial gradient, and temporal variation can be improved by up to 139.20%, 26.36%, and 162.30%, respectively; (b) the accuracy improvement introduced by fusion algorithms decreases with the number of available Landsat images per year, and it has a large variation with the same number of available Landsat images, and (c) this large variation is highly related to the temporal distributions of available Landsat images, suggesting that fusion algorithms can improve SOS accuracy only when cloud-free Landsat images cannot capture key vegetation growth period. This study calls for caution with the use of data fusion in LSP studies at fine scales.

Understanding the Potential, Uncertainties, and Limitations of Spatiotemporal Fusion for Monitoring Chlorophyll a Concentration in Inland Eutrophic Lakes

The tradeoffs between the spatial and temporal resolutions for the remote sensing instruments limit their capacity to monitor the eutrophic status of inland lakes. Spatiotemporal fusion (STF) provides a cost-effective way to obtain remote sensing data with both high spatial and temporal resolutions by blending multisensor observations. However, remote sensing reflectance ( R rs ) over water surface with a relatively low signal-to-noise ratio is prone to be contaminated by large uncertainties in the fusion process. To present a comprehensive analysis on the influence of processing and modeling errors, we conducted an evaluation study to understand the potential, uncertainties, and limitations of using STF for monitoring chlorophyll a (Chla) concentration in an inland eutrophic water (Chaohu Lake, China). Specifically, comparative tests were conducted on the Sentinel-2 and Sentinel-3 image pairs. Three typical STF methods were selected for comparison, i.e., Fit-FC, spatial and temporal nonlocal filter-based fusion model, and the flexible spatiotemporal data fusion. The results show as follows: (a) among the influencing factors, atmospheric correction uncertainties and geometric misregistration have larger impacts on the fusion results, compared with radiometric bias between the imaging sensors and STF modeling errors; and (b) the machine-learning-based Chla inversion accuracy of the fusion data [ R 2 = 0.846 and root mean square error (RMSE) = 17.835 μg/l] is comparable with that of real Sentinel-2 data ( R 2 = 0.856 and RMSE = 16.601 μg/l), and temporally dense Chla results can be produced with the integrated Sentinel-2 and fusion image datasets. These findings will help to provide guidelines to design STF framework for monitoring aquatic environment of inland waters with remote sensing data.

STEPSBI: Quick spatiotemporal fusion with coarse- and fine-resolution scale transformation errors and pixel-based synthesis base image pair

Spatiotemporal fusion (STF) is crucial for reconciling the conflict between temporal and spatial resolutions of remote sensing observations. However, fusing images in heterogeneous areas remains challenging under continuous missing values. Moreover, most current STF methods only consider temporal errors and disregard the spatial scale error in time variation. Therefore, we proposed a Quick Spatiotemporal Fusion with Coarse- and Fine-Resolution Scale Transformation Errors and Pixel-Based Synthesis Base Image Pair (STEPSBI). First, the optimal pixel-based image synthesis strategy was designed using all available fine- and coarse-resolution images. Then, the scale transformation error (STE) of coarse-resolution downscaling to fine-resolution in temporal variation was quantified. And a residual term was introduced to reduce the prediction error from the temporal variation. Finally, the spatial scale and temporal errors were corrected using the results of super-pixel segmentation as spatial weights. This model has two strengths: (1) Pixel-based image synthesis alleviates the absence of base images under continuous missing values; and (2) STE correction restores spatial details of heterogeneous areas in rapid land cover change. In scenarios of continuous missing images, abrupt land cover changes, and high-resolution heterogeneity, we evaluated STEPSBI, the Gap Filling and Savitzky–Golay filtering method (GF-SG), the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), the Flexible Spatiotemporal DAta Fusion (FSDAF), the cross-attention-based adaptive weighting fusion network (CAFE), and the Multi-scene Spatiotemporal Fusion Network (MUSTFN). The results indicate that STEPSBI yields better overall performance than other models in cropland, woodland, grassland, and other land cover types. Furthermore, ablation experiments demonstrated that each component improved the model’s performance. In addition, STEPSBI had higher fusion efficiency because it was developed on the Google Earth Engine cloud computing platform. Therefore, STEPSBI was feasible for advancing fine monitoring of spatiotemporal image fusion under continuous missing values and heterogeneous surfaces. The program of STEPBSI is freely available at: https://code.earthengine.google.com/684844aa42f64fa6b4eebe3bc0dd6483.

Agri-Fuse: A novel spatiotemporal fusion method designed for agricultural scenarios with diverse phenological changes

Images of key phenological periods play a vital role in agricultural applications as they capture the unique spectral characteristics of crops. Unfortunately, acquiring high-spatial-resolution images of the key phenological period from a single satellite platform remains challenging due to its short duration and synchronization with the rainy season. Spatiotemporal fusion (STF) is an effective tool for the prediction of missing high spatial resolution images on the required date, however, most STF methods assume a uniform relationship between the reflectance on the base date and the predicted date for the same land-cover type, which fails to hold in agricultural scenarios because of the diverse phenological changes among different crop types or different growth processes, even within a single crop type. To address these challenges, we propose a novel spatiotemporal fusion method for agricultural scenarios called Agri-Fuse. Agri-Fuse emphasizes change information and classifies change types based on the difference image derived from the fine image on the base date and the coarse image on the predicted date. Considering the various regression relationships between reflectance on the base date and the predicted date for different change types, a category-based linear regression model was built, and we introduced an unmixing model to solve these regression coefficients. We validated the performance of Agri-Fuse at two experimental sites where diverse phenological changes occurred during the key phenological periods of the same and different crop types. Compared to the four typical fusion methods (FSDAF, RASDF, STARFM, and Fit-FC), Agri-Fuse achieved the best performance in preserving both spectral fidelity and spatial details. Ablation experiments further confirmed the effectiveness of the proposed change-type classification and category-based regression in capturing diverse phenological change information. Thus, Agri-Fuse is expected to precisely reconstruct high-spatial-resolution images of key phenological periods necessary for agricultural applications.

Applying deep-learning enhanced fusion methods for improved NDVI reconstruction and long-term vegetation cover study: A case of the Danjiang River Basin

The Normalized Difference Vegetation Index (NDVI) is an essential metric in vegetation monitoring for remote sensing applications. While there are numerous long-term low-resolution NDVI datasets available there remains an unmet need for high-resolution NDVI reconstructions over extended time frames. Existing research has not comprehensively assessed the efficacy of various temporal fusion techniques for NDVI reconstruction at large regional scales. Traditional Spatiotemporal Image Fusion (TSTIF) methods often suffer from limited fusion accuracy due to input data quality constraints. To address these limitations this study introduces an innovative Deep Learning-Enhanced Spatiotemporal Fusion Method. Deep learning algorithms are employed to refine the spatial resolution of input data thereby facilitating the separation of complex image elements located at feature boundaries. This improved approach significantly enhances fusion accuracy as validated against five established TSTIF techniques through empirical analysis. As a case study we generate long-term Fractional Vegetation Cover (FVC) datasets to investigate the ecological dynamics of the Danjiang River Basin. Our findings reveal substantial gains in NDVI reconstruction accuracy through the incorporation of deep learning into traditional fusion techniques. Among the methods tested the STRUM algorithm showed the greatest improvement with its R2 value increasing from 0.872 to 0.894 and a consistent reduction in Root Mean Square Error (RMSE). The FSDAF technique emerged as the most effective boasting an R2 value of 0.953 and an RMSE of 0.012. These enhanced NDVI datasets provide more accurate representations of the basin's vegetation cover thus enriching long-term observational archives and facilitating further quantitative remote sensing analyses. Moreover we report a 21-year mean FVC of 0.702 for the Danjiang River Basin characterized by a “north-high south-low” distribution that varies seasonally. Improvements in vegetation cover were observed in Xichuan and Neixiang counties while environmental degradation is evident in the broader Danjiang River watershed. Key factors influencing these spatial variations include elevation and soil composition in addition to climate variables such as rainfall and temperature. Infrastructure projects like the South-to-North Water Diversion Project and targeted ecological conservation policies have also positively impacted vegetation within the basin.

Spatio-temporal fusion methods for spectral remote sensing: a comprehensive technical review and comparative analysis

Spatio-temporal fusion methods for spectral remote sensing: a comprehensive technical review and comparative analysis

Generating daily 100 m resolution land surface temperature estimates continentally using an unbiased spatiotemporal fusion approach

Fine spatial resolution (i.e., ≤ 100 m) land surface temperature (LST) data are crucial to study heterogeneous landscapes (e.g., agricultural and urban). Some well-known spatiotemporal fusion methods like the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and the Enhanced STARFM (ESTARFM), which were originally developed to fuse surface reflectance data, may not be suitable for direct application in LST studies due to the high sub-diurnal dynamics of LST. Furthermore, the effectiveness of spatiotemporal fusion methods for LST data has not been thoroughly evaluated in previous studies that only focused on relatively small spatiotemporal extents. To address these limitations, we proposed a variant of ESTARFM, referred to as the unbiased ESTARFM (ubESTARFM), specifically designed to accommodate the high temporal dynamics of LST to generate fine-resolution LST estimates. We evaluated ubESTARFM and ESTARFM against in-situ LST and the ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station (ECOSTRESS) LST across 12 regions throughout Australia, encompassing various land covers and environments. Independent validation showed that ubESTARFM had a bias of 2.55 K, unbiased root mean squared error (ubRMSE) of 2.57 K, and Pearson correlation coefficient (R) of 0.95 against the in-situ LST over 11,290 observations at the 12 sites, all of which were considerably better than those calculated for ESTARFM, being a bias of 4.73 K, ubRMSE of 3.80 K and R of 0.92. When compared to ECOSTRESS data, ubESTARFM LST had a bias of −1.69 K, ubRMSE of 2.00 K, and R of 0.70 over 43 near clear-sky scenes, while ESTARFM LST had a bias of 1.79 K, ubRMSE of 2.68 K, and R of 0.59. Overall, our results demonstrated that ubESTARFM can avoid systematic bias accumulation, substantially reduce uncertainty deviation, and maintain a good level of correlation with validation datasets when compared to ESTARFM. A further assessment underscored the potential of ubESTARFM for application using LST data acquired from geostationary platforms (e.g., Himawari-8), with a mean ubRMSE (R) of 2.22 K (0.97) against in-situ LST over 1327 observations at 3 sites from southeast Australia at the overpass time of MODIS/Terra. This promising method leverages reliable numeric values from coarse-resolution LST while borrowing spatial heterogeneity from fine-resolution LST and has the potential to be coupled with energy balance and/or radiative transfer models thus enabling better farm and/or regional-scale water management strategies to be implemented. Furthermore, both the input and generated LST data, encompassing a comprehensive spatial extent over diverse land covers and climatic conditions, are publicly available for benchmarking future algorithmic refinements.

A Method for Spatiotemporally Merging Multi-Source Precipitation Based on Deep Learning

Reliable precipitation data are essential for studying water cycle patterns and climate change. However, there are always temporal or spatial errors in precipitation data from various sources. Most precipitation fusion methods are influenced by high-dimensional input features and do not make good use of the spatial correlation between precipitation and environmental variables. Thus, this study proposed a novel multi-source precipitation spatiotemporal fusion method for improving the spatiotemporal accuracy of precipitation. Specifically, the attention mechanism was used to first select critical input information to dimensionalize the inputs, and the Convolutional long-short-term memory network (ConvLSTM) was used to merge precipitation products and environmental variables spatiotemporally. The Yalong River in the southeastern part of the Tibetan Plateau was used as the case study area. The results show that: (1) Compared with the original precipitation products (IMERG, ERA5 and CHIRPS), the proposed method has optimal accuracy and good robustness, and its correlation coefficient (CC) reaches 0.853, its root mean square coefficient (RMSE) decreases to 3.53 mm/d and its mean absolute error (MAE) decreases to 1.33 mm/d. (2) The proposed method can reduce errors under different precipitation intensities and greatly improve the detection capability for strong precipitation. (3) The merged precipitation generated by the proposed method can be used to describe the rainfall–runoff relationship and has good applicability. The proposed method may greatly improve the spatiotemporal accuracy of precipitation in complex terrain areas, which is important for scientific management and the allocation of water resources.

A spatio-temporal fusion method based on Landsat and MODIS normalized vegetation index data

Abstract. Normalized difference vegetation index (NDVI) can simply and effectively reflect the growth status of plants, and has a linear relationship with vegetation coverage, thus it is an important indicator to identify vegetation growth status and coverage. However, the commonly used MODIS NDVI or Landsat NDVI cannot simultaneously achieve the high temporal and spatial resolution. To this end, we proposed a generic and automatic method to fuse MODIS and Landsat satellite vegetation index (NDVI) into daily high-resolution products which takes time series NDVI data from MODIS/Landsat as input, filters noise pixels, and generates high-resolution and high-frequency products through a fusion process of temporal-linearly interpolation to increase the temporal resolution and spatial filtering to alleviate the blocky artifacts from MODIS pixel border. The fused NDVI maps are visually compared to evaluate the performance of this method, and more quantitative assessment will be continued in the next step. This study can provide important support for large-scale long-term surface vegetation monitoring.

Supervised and self-supervised learning-based cascade spatiotemporal fusion framework and its application

Spatiotemporal fusion (STF) is considered an effective way to address the mutual constraints of the spatiotemporal resolutions of the remote sensing images from a single satellite sensor. Although the deep learning (DL)-based STF methods have shown great potential so far, there are still some deficiencies. Supervised DL-based methods usually train the network on the original scale data with richer spatial information, but they utilize only the data at auxiliary dates to generate the fusion data at prediction dates, which may perform poorly when significant spatiotemporal changes occur between the auxiliary and prediction dates. For self-supervised DL-based methods, the training can focus more on the data at prediction dates, but the network is generally trained on the down-sampled data with less spatial information, causing insufficient spatial structures of the fusion results. Based on the above, we innovatively combine supervised and self-supervised learning and propose a dual-stage cascade STF framework. In the first stage, a model based on supervised learning is trained on the data at auxiliary dates to extract abundant spatial features and obtain the initial fusion results. In the second stage, we utilize a self-supervised strategy to excavate the spatiotemporal features of prediction dates based on the initial fusion results and the observed data at prediction dates. In addition, to alleviate the insufficient consideration of the temporal correlation in existing STF methods, we design a temporal consistency loss function to fully utilize the temporal correlation information between multiple prediction dates to generate more accurate fusion results. The proposed framework can not only realize the STF of remote sensing images but also be well applied to the STF of remote sensing products such as land surface temperature (LST). Based on the quantitative results of three datasets, the proposed method improves the quantitative indices of root mean square error (RMSE), structural similarity (SSIM), erreur relative global adimensionnelle de synthese (ERGAS), peak signal to noise ratio (PSNR), and spectral angle mapper (SAM) by up to 0.2014, 0.0279, 0.6797, 0.7629, and 0.2006 at most compared with the best comparison method, which fully shows the superiority of our method.

A task decoupled framework for enhancing the deep learning-based spatiotemporal fusion method

ABSTRACT Spatiotemporal fusion (STF) is a cost-effective way to reconstruct time-series images. In recent years, deep learning-based (DL-based) STF methods have received substantial attention. However, two limitations of DL-based STF methods still remain: (1) existing methods require simultaneous learning of both the multi-source images correction model and the STF model, which complicates the training task. The high complexity poses a challenge for the network to accurately learn the underlying mathematical principles of STF, thereby reducing the method’s reliability and generalization ability; (2) existing methods tend to generate blurry predictions. To address these limitations, this study proposes a task decoupled (TD) framework that offers a simple yet effective solution for enhancing DL-based STF method. The framework consists of a correction model and a STF model, which are trained using actual and simulated image pairs, respectively, to learn the multi-source images correction model and STF mathematical model. The loss of edge feature is added in the loss function of the STF model to ameliorate its detailed information preservation ability. The proposed framework is evaluated on three DL-based STF methods in five different sites using root-mean-square error (RMSE) and Robert’s edge (Edge) to assess spectral and spatial accuracy. The experimental results indicate that the proposed framework can significantly improve the models’ ability to predict spectral information (average increase rate = 5.3% in spectral accuracy), preserve spatial information (average increase rate = 16.2% in spatial accuracy), retrieve land cover change, and generalize to new data. These findings demonstrate the effectiveness of the proposed framework in addressing the limitations of existing DL-based STF methods and its potential for advancing STF applications.

A spatiotemporal fusion method based on interpretable deep networks

A spatiotemporal fusion method based on interpretable deep networks

ROBOT: A spatiotemporal fusion model toward seamless data cube for global remote sensing applications

Dense time-series high-resolution satellite images are extremely valuable for long-term monitoring of land dynamics. Spatiotemporal fusion (STF) techniques have been developed to integrate multi-resolution satellite images to produce data with high spatial resolution and temporal frequency. Due to the large volume and diversity of higher resolution global Earth Observation (EO) data, large-scale data processing methods need to be computationally efficient, free of parameter fine-tuning, and adaptive to various data structures without process customization. These requirements are especially critical for the production of global Seamless Data Cube (SDC). Considering the limitations of existing STF methods, we propose a ROBust OpTimization-based (ROBOT) fusion model that exploits time-series information to obtain more accurate predictions. ROBOT maintains stable under varied data conditions by adopting a temporal-coherence regularization term. And being free of parameter tuning, ROBOT can be applied to arbitrary spatiotemporally distributed data without repetitive process customization. Its performance was compared with eight representative STF methods. Results show that ROBOT outperforms existing STF methods in most cases and is computationally efficient, about 4000-fold faster than ESTARFM and 600-fold faster than FSDAF. The proposed method has demonstrated its potential for global-scale SDC generation to support subsequent remote sensing applications.