WDBSTF: A Weighted Dual-Branch Spatiotemporal Fusion Network Based on Complementarity between Super-Resolution and Change Prediction

Abstract

Spatiotemporal fusion (STF) is a solution to generate satellite images with both high-spatial and high-temporal resolutions. The deep learning-based STF algorithms focus on spatial dimensions to build a super-resolution (SR) model or the temporal dimensions to build a change prediction (CP) model, or the task itself to build a data-driven end-to-end model. The multi-source images used for STF usually have large spatial scale gaps and temporal spans. The large spatial scale gaps lead to poor spatial details based on a SR model; the large temporal spans make it difficult to accurately reconstruct changing areas based on a CP model. We propose a weighted dual-branch spatiotemporal fusion network based on complementarity between super-resolution and change prediction (WDBSTF), which includes the SR branch and CP branch, and a weight module representing the complementarity of the two branches. The SR branch makes full use of edge information and high-resolution reference images to obtain high-quality spatial features for image reconstruction. The CP branch decomposes complex problems via a two-layer cascaded network, changes features from the difference image, and selects high-quality spatial features through the attention mechanism. The fusion result of the CP branch has rich image details, but the fusion accuracy in the changing area is low due to the lack of detail. The SR branch has consistent and excellent fusion performances in the changing and no-changing areas, but the image details are not rich enough compared with the CP branch due to the large amplification factor. Next, a weighted network was designed to combine the advantages of the two branches to produce improved fusion results. We evaluated the performance of the WDBSTF in three representative scenarios, and both visual and quantitative evaluations demonstrate the state-of-the-art performance of our algorithm. (On the LGC dataset, our method outperforms the suboptimal method by 2.577% on SSIM. On the AHB dataset, our method outperforms the suboptimal method by 1.684% on SSIM. On the CIA dataset, our method outperforms the suboptimal method by 5.55% on SAM).