Spatiotemporal Fusion Research Articles

Spatiotemporal Fusion Model of Remote Sensing Images Combining Single-Band and Multi-Band Prediction

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels’ convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies.

STF-Trans: A two-stream spatiotemporal fusion transformer for very high resolution satellites images

Spatiotemporal satellite image fusion is regarded as an effective approach to address the limitations of a single optical sensor and generate data with high spatial and temporal resolutions. Thanks to the recent advances of Deep Learning, spatiotemporal fusion has gained a significant performance improvement. However, most models require at least three inputs, including a high resolution one at a prior date, to make the prediction at the desired date. Such a requirement is not always guaranteed in practice, in particular due to the high cost of high resolution image and bad atmospheric conditions. In the last few years, Transformers have achieved tremendous performance on several computer vision tasks compared to convolutional neural networks. Inspired by the new trend, we proposed an end-to-end two-stream SpatioTemporal fusion technique based on encoder–decoder transformer architecture, termed STF-Trans. To deal with a complexity and high-cost of data preparation, the proposed approach aims to fuse a coarse-resolution image with high temporal resolution, and a very high resolution Google Earth image, to produce a very high resolution image at the desired date. The proposed technique employs a two-stream convolutional network to capture both temporal and spatial features from the inputs, and then the encoder–decoder transformer captures long-range dependency of the latter and generates a more efficient image representation from the shallow features. Finally, a shallow network maps the features into the desired fused image. The experimental results show that the proposed method outperforms the current state-of-the-art techniques at both quantitative and qualitative evaluations.

Improving agricultural field parcel delineation with a dual branch spatiotemporal fusion network by integrating multimodal satellite data

Accurate spatial information for agricultural field parcels is important for agricultural production management and understanding agro-industrialization and intensification. However, traditional remote sensing methods that rely on single-modal or single-date data struggle to identify heterogeneous field parcels, particularly in regions dominated by smallholder farming systems. To address this challenge, we proposed a Dual branch Spatiotemporal Fusion Network (DSTFNet) that integrated very high-resolution (VHR) images and medium-resolution satellite image time series (MRSITS) to extract agricultural field parcels over various landscapes. The DSTFNet consisted of two branches: a spatial branch that extracted spatial features from VHR images and a temporal branch that explored seasonal spectral dynamics from MRSITS data by using ConvLSTM units and an attention module. We evaluated the DSTFNet in four regions across China by using GF-2 and Sentinel-2 data. The results showed that DSTFNet performed well in delineating agricultural field parcels, achieving the highest Matthew’s correlation coefficient (MCC) = 0.823 for the field extent, the highest F1-score of edge (Fedge) = 0.865 for field boundary, and the lowest errors of segmentation evaluation index (SEI) = 0.191 for the vectorized field parcels in Hubei province. In addition, DSTFNet significantly outperformed two single-branch models that used VHR or MRSITS alone, as well as existing BsiNet, ResUNet_a, UNet and RAUNet models. DSTFNet also showed good spatial transferability in distinct regions without training data (on average, MCC = 0.728, Fedge = 0.729, and SEI = 0.281 for three target regions). Using limited training data to fine-tune the DSTFNet can further improve its ability to delineate field parcels over complex regions. The visualization analysis of temporal attention weights demonstrated that DSTFNet can well capture cropland spectral dynamics, making it advantageous in extracting diverse cropland parcels. By exploiting important spectral, spatial and temporal information from multimodal satellite data, DSTFNet provided an effective, robust, and transferable solution for accurately delineating agricultural field parcels across heterogeneous farming systems.

Spatio-temporal fusion of meteorological factors for multi-site PM2.5 prediction: A deep learning and time-variant graph approach

In the field of environmental science, traditional methods for predicting PM2.5 concentrations primarily focus on singular temporal or spatial dimensions. This approach presents certain limitations when it comes to deeply mining the joint influence of multiple monitoring sites and their inherent connections with meteorological factors. To address this issue, we introduce an innovative deep-learning-based multi-graph model using Beijing as the study case. This model consists of two key modules: firstly, the 'Meteorological Factor Spatio-Temporal Feature Extraction Module'. This module deeply integrates spatio-temporal features of hourly meteorological data by employing Graph Convolutional Networks (GCN) and Long Short-Term Memory (LSTM) for spatial and temporal encoding respectively. Subsequently, through an attention mechanism, it retrieves a feature tensor associated with air pollutants. Secondly, these features are amalgamated with PM2.5 concentration values, allowing the 'PM2.5 Concentration Prediction Module' to predict with enhanced accuracy the joint influence across multiple monitoring sites. Our model exhibits significant advantages over traditional methods in processing the joint impact of multiple sites and their associated meteorological factors. By providing new perspectives and tools for the in-depth understanding of urban air pollutant distribution and optimization of air quality management, this model propels us towards a more comprehensive approach in tackling air pollution issues.

Construction of customer-aware grid interactive service system based on neural network model

Abstract Neural network models have advantages in spatial and temporal information processing, which provide new ideas for the construction of customer-aware grid interactive service system. In this paper, firstly, the spatio-temporal fusion deep neural network model based on the attention mechanism is used to extract spatial and temporal information by convolutional neural network and gated recurrent neural network, respectively. The attention mechanism is also introduced to assign weights to various types of features to improve prediction accuracy. Secondly, the global information of the attention mechanism of the channel on the feature map is fully utilized to solve the problem of inadequate extraction of feature information from a single model, and then the grid interactive service system is constructed. The results show that the model proposed in this paper has higher prediction accuracy with root mean square error, coefficient of determination, and prediction accuracy of 0.972, 1.742, and 0.935, respectively. To verify the performance of the cache-based curve service in handling customer-perceived grid interactions, minute samples are created, where the maximum number of minute sample records is 3.5 million. Thus, it is shown that by introducing the attention mechanism can effectively improve the computational efficiency of the model, get better prediction results, and meet the demand for efficient and stable customer-aware interactive service system.

Improving Reliability in Reconstruction of Landsat EVI Seasonal Trajectory over Cloud-Prone, Fragmented, and Mosaic Agricultural Landscapes

Although the Landsat 30 m Enhanced Vegetation Index (EVI) products are important input variables in land surface models, recurring Landsat 5/7 EVI time series over cloud-prone, fragmented, and mosaic agricultural landscapes is still a great challenge. In this study, we put forward a simple, but effective “Light and Temperature-Driven Growth model and Double Logistic function fusion algorithm” (LTDG_DL). The empirical basis of the LTDG_DL algorithm was traced from the de Wit crop growth simulation model and the commonly observed nonlinear correlation between the EVI and the Leaf Area Index (LAI). It assimilates the ground daily solar radiation and air temperature to generate seasonal profiles of the empirical LAI and EVI and conducts the within-season calibration of the empirical EVI by adjusting crop growth using cloud-free Landsat EVI observations. The initial date of seedling emergence (DOYini) and the accumulated Growing Degree Days for completion of the vegetative and Flowering stage (FGDDs) were variables to which the algorithm’s accuracy was most sensitive. The variable-constrained optimization of the LTDG_DL algorithm was performed by loading the seedling emergence calendar of local prevailing crops and establishing an FGDD lookup table with an exhaustive sampling without replication method. Compared to temporal interpolation functions and Landsat–MODIS spatiotemporal fusion algorithms, the LTDG_DL algorithm had superior performance in the predictions of the EVI increment slope at the vegetative growth stage, the timing of the peak EVI, and the protection of key Landsat EVI observations over cloud-contaminated and complex landscape agricultural systems. Finally, the advantages and limitations of the LTDG_DL algorithm are discussed.

Hourly emissions of air pollutants and greenhouse gases from open biomass burning in China during 2016–2020

Open biomass burning (OBB) is a significant source of air pollutants and greenhouse gases that have contributed to air pollution episodes in China in recent years. An accurate emission inventory is critical for the precise control of OBB. Existing OBB emission datasets are commonly based on MODIS observations, and most only have a daily-scale temporal resolution. Daily OBB emissions, however, might not accurately represent diurnal variations, peak hours, or any open burning processes. The China Hourly Open Biomass Burning Emissions (CHOBE) dataset for mainland China from 2016 to 2020 was developed in this study using the spatiotemporal fusion of multiple active fires from MODIS, VIIRS S-NPP and Himawari-8 AHI detections. At a spatial resolution of 2 km, CHOBE provided gridded CO, NOx, SO2, NH3, VOCs, PM2.5, CO2, CH4 and N2O emissions from OBB. CHOBE will enhance insight into OBB spatiotemporal variability, improves air quality and climate modelling and forecasting, and aids in the formulation of precise OBB preventive and control measures.

Multi-stage deep residual collaboration learning framework for complex spatial–temporal traffic data imputation

Performing accurate and efficient traffic data repair has become an essential task before proceeding with other applications of intelligent transportation systems. However, existing repair models often impose assumptions on the underlying data generation process or fail to sufficiently capture the complex spatial–temporal dependencies between several nodes at different intervals. To address these issues, we propose a novel multi-stage deep residual collaboration learning framework (named Multi-DRCF) to tackle the complex task of repairing missing traffic data. In particular, a spatiotemporal fusion network (i.e., ST-imputator) incorporating the graph-based convolutions with recurrent structures is introduced for modeling the spatial topology and dynamic temporal coherences of the traffic graph network in turn. Moreover, to further achieve both higher accuracy and efficiency, we exploit a stackable bi-directional residual optimization (i.e., Bi-RO) structure to enhance the role of ST-imputator in the utilization of the spatiotemporal properties of the residuals. Indeed, this combination of ST-imputator and Bi-RO structure can serve as an iterable unit for Multi-DRCF, and the whole computation process is highly modular and flexible in operation. We evaluate Multi-DRCF using two real-world large-scale traffic speed datasets, i.e., Seattle-Loop and METR-LA. Experimental results demonstrate that Multi-DRCF can effectively extract the spatial and temporal properties from traffic data, resulting in an excellent repair performance with an acceptable computational cost. Compared with the state-of-the-art baseline models, Multi-DRCF performs more competitively for three types of missing patterns, and also provides steadier repair results with missing rates ranging from 10% to 90%. Finally, the ablation experiments and visualization analysis also present well insights for better understanding the superiority of the Multi-DRCF for complex spatial–temporal traffic data imputation.

An unsupervised spatiotemporal fusion network augmented with random mask and time-relative information modulation for anomaly detection of machines with multiple measuring points

In industrial environments, individual sensor is easily affected by background noise, etc. In order to improve the reliability of anomaly detections, sensors are arranged at multiple measuring points to collect monitoring data of machines. However, under the coupling of vibration responses of multiple components of machines, the complex nonlinear relationship between monitoring data of multiple measuring points makes it difficult to achieve the best feature extraction and fusion effect, which reduces the accuracy of anomaly detection. To solve this problem, an unsupervised spatiotemporal fusion network augmented with random mask and time-relative information modulation is proposed. Firstly, we creatively propose random mask and modulated signal generation method based on mask index to learn the dependence of waveform and time dimension and achieve temporal dimension fusion of signals. Based on end-to-end training, modulated signals are also more conducive to spatial fusion. Then, to fully exploit the correlation between monitoring data of multiple measuring points and obtain the best spatial dimension fusion effect, a multi-head graph neural network based on self-attention weight matrix is carried out. Finally, we use transformer encoder to reconstruct the signal of each measuring point and obtain reconstruction error. Based on exponentially weighted moving average, anomaly detection threshold is obtained. Two anomaly detection experiments are conducted, and accuracy of 99.78%, 99% are achieved.

Refined assessment of space-time changes, influencing factors and socio-economic impacts of the terrestrial ecosystem quality: A case study of the GBA

The terrestrial ecosystem is the cradle of energy and material basis for human survival and development. However, there are large research deficits in accurately and finely depicting the quality of the terrestrial ecosystem (QTE) and assessing its changing triggers' contribution. Here, we summarized three major principles for selecting image sources in remote sensing data fusion. A continuous 30-m net vegetation productivity (NPP) dataset during 2000–2019 for the Guangdong-Hong Kong-Macao Greater Bay Area (GBA) was derived by using the Carnegie-Ames-Stanford approach model and pre-fused normalized difference vegetation index. The factors’ contributions to the QTE changes were quantitatively assessed. The role of the QTE in affecting the socio-economic and its behind mechanisms was quantitatively investigated. The results showed that: (1) High-quality images sources are the preference for spatio-temporal fusion of remote sensing data. Images with close month, the same season and year, and sensors should be then selected. Images of different sensors with similar spectral bandwidth, the ones from adjacent years and seasons, can be alternately considered. (2) Fine-resolution NPP has higher accuracy than coarse-resolution NPP and has marked advantages in finely characterizing the QTE. In the past 20 years, the QTE in the GBA has shown a fluctuating increasing trend (0.20 Tg C/yr). (3) Human activities contributed 54.19% of the QTE changes in the GBA, and dominates the QTE changes in the central rapidly urbanizing areas. Residual factors accounted for an overall contribution ratio of 35.71%. Climate change dominants the peripheral forest variations in the GBA. (4) In the GBA, the improvement of QTE has a significant positive socio-economic impact, it contributes to the GDP increment firstly then the GDP aggregate indirectly. Our results highlight that it is of great urgent to estimate long-term continuous NPP with high spatio-temporal resolution globally. Controlling strategies should be implemented to reduce factitious impacts on QTE. High level of ecological and environmental protection promotes the sustainable development.

Spatio-temporal fusion methods for spectral remote sensing: a comprehensive technical review and comparative analysis

Spatio-temporal fusion methods for spectral remote sensing: a comprehensive technical review and comparative analysis

Generating daily 100 m resolution land surface temperature estimates continentally using an unbiased spatiotemporal fusion approach

Fine spatial resolution (i.e., ≤ 100 m) land surface temperature (LST) data are crucial to study heterogeneous landscapes (e.g., agricultural and urban). Some well-known spatiotemporal fusion methods like the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM) and the Enhanced STARFM (ESTARFM), which were originally developed to fuse surface reflectance data, may not be suitable for direct application in LST studies due to the high sub-diurnal dynamics of LST. Furthermore, the effectiveness of spatiotemporal fusion methods for LST data has not been thoroughly evaluated in previous studies that only focused on relatively small spatiotemporal extents. To address these limitations, we proposed a variant of ESTARFM, referred to as the unbiased ESTARFM (ubESTARFM), specifically designed to accommodate the high temporal dynamics of LST to generate fine-resolution LST estimates. We evaluated ubESTARFM and ESTARFM against in-situ LST and the ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station (ECOSTRESS) LST across 12 regions throughout Australia, encompassing various land covers and environments. Independent validation showed that ubESTARFM had a bias of 2.55 K, unbiased root mean squared error (ubRMSE) of 2.57 K, and Pearson correlation coefficient (R) of 0.95 against the in-situ LST over 11,290 observations at the 12 sites, all of which were considerably better than those calculated for ESTARFM, being a bias of 4.73 K, ubRMSE of 3.80 K and R of 0.92. When compared to ECOSTRESS data, ubESTARFM LST had a bias of −1.69 K, ubRMSE of 2.00 K, and R of 0.70 over 43 near clear-sky scenes, while ESTARFM LST had a bias of 1.79 K, ubRMSE of 2.68 K, and R of 0.59. Overall, our results demonstrated that ubESTARFM can avoid systematic bias accumulation, substantially reduce uncertainty deviation, and maintain a good level of correlation with validation datasets when compared to ESTARFM. A further assessment underscored the potential of ubESTARFM for application using LST data acquired from geostationary platforms (e.g., Himawari-8), with a mean ubRMSE (R) of 2.22 K (0.97) against in-situ LST over 1327 observations at 3 sites from southeast Australia at the overpass time of MODIS/Terra. This promising method leverages reliable numeric values from coarse-resolution LST while borrowing spatial heterogeneity from fine-resolution LST and has the potential to be coupled with energy balance and/or radiative transfer models thus enabling better farm and/or regional-scale water management strategies to be implemented. Furthermore, both the input and generated LST data, encompassing a comprehensive spatial extent over diverse land covers and climatic conditions, are publicly available for benchmarking future algorithmic refinements.

A novel fusion framework embedded with zero-shot super-resolution and multivariate autoregression for precipitable water vapor across the continental Europe

Precipitable water vapor (PWV), as the most abundant greenhouse gas, significantly impacts the evapotranspiration process and thus the global climate. However, the applicability of mainstream satellite PWV products is limited by the tradeoff between spatial and temporal resolutions, as well as some external factors such as cloud contamination. In this study, we proposed a novel PWV spatio-temporal fusion framework based on the zero-shot super-resolution and the multivariate autoregression models (ZSSR-ARF) to improve the accuracy and continuity of PWV. The framework is implemented in a way that the satellite-derived observations (MOD05) are fused with the reanalysis data (ERA5) to generate accurate and seamless PWV of high spatio-temporal resolution (0.01°, daily) across the European continent from 2001 to 2021. Firstly, the ZSSR approach is used to enhance the spatial resolution of ERA5 PWV based on the internal recurrence of image information. Secondly, the optimal ERA5-MOD05 image pairs are selected based on the image similarity as inputs to improve the fusion accuracy. Thirdly, the framework develops a multivariate autoregressive fusion approach to allocate weights adaptively for the high-resolution image prediction, which primely addresses the non-stationarity and autocorrelation of PWV. The results reveal that the accuracies of fused PWV are consistent with those of the GPS retrievals (r = 0.82–0.95 and RMSE = 2.21–4.01 mm), showing an enhancement in the accuracy and continuity compared to the original MODIS PWV. The ZSSR-ARF fusion framework outperforms the other methods with R2 improved by over 24% and RMSE reduced by over 0.61 mm. Furthermore, the fused PWV exhibits similar temporal consistency (mean difference of 0.40 mm and DSTD of 3.22 mm) to the reliable ERA5 products, and substantial increasing trends (mean of 0.057 mm/year and over 0.1 mm/year near the southern and western coasts) are observed over the European continent. As the accuracy and continuity of PWV are improved, the outcome of this paper has potential for climatic analyses during the land-atmosphere cycle process.

A Method for Spatiotemporally Merging Multi-Source Precipitation Based on Deep Learning

Reliable precipitation data are essential for studying water cycle patterns and climate change. However, there are always temporal or spatial errors in precipitation data from various sources. Most precipitation fusion methods are influenced by high-dimensional input features and do not make good use of the spatial correlation between precipitation and environmental variables. Thus, this study proposed a novel multi-source precipitation spatiotemporal fusion method for improving the spatiotemporal accuracy of precipitation. Specifically, the attention mechanism was used to first select critical input information to dimensionalize the inputs, and the Convolutional long-short-term memory network (ConvLSTM) was used to merge precipitation products and environmental variables spatiotemporally. The Yalong River in the southeastern part of the Tibetan Plateau was used as the case study area. The results show that: (1) Compared with the original precipitation products (IMERG, ERA5 and CHIRPS), the proposed method has optimal accuracy and good robustness, and its correlation coefficient (CC) reaches 0.853, its root mean square coefficient (RMSE) decreases to 3.53 mm/d and its mean absolute error (MAE) decreases to 1.33 mm/d. (2) The proposed method can reduce errors under different precipitation intensities and greatly improve the detection capability for strong precipitation. (3) The merged precipitation generated by the proposed method can be used to describe the rainfall–runoff relationship and has good applicability. The proposed method may greatly improve the spatiotemporal accuracy of precipitation in complex terrain areas, which is important for scientific management and the allocation of water resources.

Spatial enhanced spatiotemporal reflectance fusion model for heterogeneous regions with land cover change

Numerous spatiotemporal fusion models have been developed to fuse dense time-series data with a high spatial resolution for monitoring land surface dynamics. Nonetheless, enhancing spatial details of fused images, eliminating the obvious ‘plaque’ phenomenon and image blurring in fused images, and developing relatively simple and 1 easy-to-implement algorithms remain a challenge for spatiotemporal fusion algorithms. Therefore, this paper presents a newly proposed spatial enhanced spatiotemporal reflectance fusion model (SE-STRFM) for image fusions in heterogeneous regions with land cover change. The SE-STRFM model predicts temporal changes of reflectance in sub-pixel details based on the spectral unmixing theory, and allocates reflectance changes caused by abrupt land cover change in fine-resolution images with a relatively simple algorithm and easy implementation. SE-STRFM only needs one pair of input data, comprising one fine-resolution image and one coarse-resolution image, to achieve high-precision reflectance prediction with spatial details. To verify the reliability and applicability of the SE-STRFM, we use Landsat image and simulated MODIS-like image to fuse high spatial and temporal resolution images and select two study areas with heterogeneous landscape and land cover type change for fusion experiments and accuracy evaluation. The results show that the images fused by SE-STRFM have clearer spatial details and a more accurate spectral distribution compared with those fused by the most widely used STARFM, ESTARFM and FSDAF. In two study areas with heterogeneous landscape and land cover type change, compared with STARFM, ESTARFM and FSDAF, the RMSE of SE-STRFM is 10.52%, 28.39% and 6.58% lower on average, respectively; r is 3.67%, 10.33% and 1.65% higher on average, respectively; AAD is 9.05%, 24.58% and 7.29% lower on average, respectively; and SSIM is 3.16%, 10.16% and 1.92% higher on average, respectively. SE-STRFM can accurately capture temporal changes with spatial details and effectively predict abrupt land-cover changes.

Spatiotemporal fusion target detection and location based on depth optical flow and YOLOv3

Spatiotemporal fusion target detection and location based on depth optical flow and YOLOv3

Fast-MFQE: A Fast Approach for Multi-Frame Quality Enhancement on Compressed Video.

For compressed images and videos, quality enhancement is essential. Though there have been remarkable achievements related to deep learning, deep learning models are too large to apply to real-time tasks. Therefore, a fast multi-frame quality enhancement method for compressed video, named Fast-MFQE, is proposed to meet the requirement of video-quality enhancement for real-time applications. There are three main modules in this method. One is the image pre-processing building module (IPPB), which is used to reduce redundant information of input images. The second one is the spatio-temporal fusion attention (STFA) module. It is introduced to effectively merge temporal and spatial information of input video frames. The third one is the feature reconstruction network (FRN), which is developed to effectively reconstruct and enhance the spatio-temporal information. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods in terms of lightweight parameters, inference speed, and quality enhancement performance. Even at a resolution of 1080p, the Fast-MFQE achieves a remarkable inference speed of over 25 frames per second, while providing a PSNR increase of 19.6% on average when QP = 37.

Identification and correction of abnormal measurement data in power system based on graph convolutional network and gated recurrent unit

Power system measurement data is used as the basis for various power system analyses. Its accuracy and reliability are particularly important. However, due to many problems, there are abnormal data in the measurement data. An identification method based on the spatial-temporal characteristics of measurement data is proposed to resolve the problems of poor identification and low efficiency. The Pre-identification (PI) process used to improve efficiency is introduced, and the sliding window based on the slope is used to realize rapid interval targeting of abnormal data and generate suspicious data sets. A spatiotemporal fusion model based on Graph Convolutional Network (GCN) and Gated Recurrent Unit (GRU) has been proposed. By aggregating the spatial-temporal characteristics of measurement data, precise reconstruction of measurement data is achieved. Then, by setting a threshold to separate the real abnormal data and normal data, the suspicious data set can be cleaned. By simulation experiments, under the cases of different ratios and different types of abnormal data, the proposed method is proved to be better in identification performances and higher in efficiency. By testing actual measurement data, the proposed method can accurately identify abnormal data under the interference of fluctuating data, indicating the proposed method has good robustness.

Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion for remote sensing image

ABSTRACT Remote sensing image (RSI) with concurrently high spatial, temporal, and spectral resolutions cannot be produced by a single sensor. Multisource RSI fusion is a convenient technique to realize high spatial resolution multispectral (MS) images (spatial spectral fusion, i.e. SSF) and high temporal and spatial resolution MS images (spatiotemporal fusion, i.e. STF). Currently, deep learning-based fusion models can only implement SSF or STF, lacking models that perform both SSF and STF. Multiresolution generative adversarial networks with bidirectional adaptive-stage progressive guided fusion (BAPGF) for RSI are proposed to implement both SSF and STF, namely BPF-MGAN. A bidirectional adaptive-stage feature extraction architecture in fine-scale-to-coarse-scale and coarse-scale-to-fine-scale modes is introduced. The designed BAPGF introduces a previous fusion result-oriented cross-stage-level dual-residual attention fusion strategy to enhance critical information and suppress superfluous information. Adaptive resolution U-shaped discriminators are implemented to feed multiresolution context into the generator. A generalized multitask loss function unlimited by no-reference images is developed to strengthen the model via constraints on the multiscale feature, structural, and content similarities. The BPF-MGAN model is validated on SSF datasets and STF datasets. Compared with the state-of-the-art SSF and STF models, results demonstrate the superior performance of the proposed BPF-MGAN model in both subjective and objective evaluations.

A spatio-temporal fusion method based on Landsat and MODIS normalized vegetation index data

Abstract. Normalized difference vegetation index (NDVI) can simply and effectively reflect the growth status of plants, and has a linear relationship with vegetation coverage, thus it is an important indicator to identify vegetation growth status and coverage. However, the commonly used MODIS NDVI or Landsat NDVI cannot simultaneously achieve the high temporal and spatial resolution. To this end, we proposed a generic and automatic method to fuse MODIS and Landsat satellite vegetation index (NDVI) into daily high-resolution products which takes time series NDVI data from MODIS/Landsat as input, filters noise pixels, and generates high-resolution and high-frequency products through a fusion process of temporal-linearly interpolation to increase the temporal resolution and spatial filtering to alleviate the blocky artifacts from MODIS pixel border. The fused NDVI maps are visually compared to evaluate the performance of this method, and more quantitative assessment will be continued in the next step. This study can provide important support for large-scale long-term surface vegetation monitoring.