MLFF-GAN: A Multilevel Feature Fusion With GAN for Spatiotemporal Remote Sensing Images

Abstract

Due to the limitation of technology and budget, it is often difficult for sensors of a single remote sensing satellite to have both high temporal resolution and high spatial (HTHS) resolution at the same time. In this paper, we proposed a new Multi-level Feature Fusion with Generative Adversarial Network (MLFF-GAN) for generating fusion HTHS images. MLFF-GAN mainly uses U-net-like architecture and its generator is composed of three stages: feature extraction, feature fusion, and image reconstruction. In feature extraction and reconstruction stage, the generator employs the encoding and decoding structure to extract three groups of multi-level features, which can cope with the huge difference of resolution between high-resolution images and low-resolution images. In the feature fusion stage, Adaptive Instance Normalization (AdaIN) block is designed to learn the global distribution relationship between multi-temporal images, and an attention module (AM) is used to learn the local information weights for the change of small areas. The proposed MLFF-GAN was tested on two Landsat and MODIS datasets. Some state-of-the-art algorithms are comprehensively compared with MLFF-GAN. We also carried on the ablation experiment to test the effectiveness of different sub-module in MLFF-GAN. The experiment results and ablation analysis show the better performances of the proposed method when compared with other methods. The code is available at https://github.com/songbingze/MLFF-GAN.