Satellite Image Multi-Frame Super Resolution Using 3D Wide-Activation Neural Networks

Francisco Dorr

doi:10.3390/rs12223812

Abstract

The small satellite market continues to grow year after year. A compound annual growth rate of 17% is estimated during the period between 2020 and 2025. Low-cost satellites can send a vast amount of images to be post-processed at the ground to improve the quality and extract detailed information. In this domain lies the resolution enhancement task, where a low-resolution image is converted to a higher resolution automatically. Deep learning approaches to Super Resolution (SR) reached the state-of-the-art in multiple benchmarks; however, most of them were studied in a single-frame fashion. With satellite imagery, multi-frame images can be obtained at different conditions giving the possibility to add more information per image and improve the final analysis. In this context, we developed and applied to the PROBA-V dataset of multi-frame satellite images a model that recently topped the European Space Agency’s Multi-frame Super Resolution (MFSR) competition. The model is based on proven methods that worked on 2D images tweaked to work on 3D: the Wide Activation Super Resolution (WDSR) family. We show that with a simple 3D CNN residual architecture with WDSR blocks and a frame permutation technique as the data augmentation, better scores can be achieved than with more complex models. Moreover, the model requires few hardware resources, both for training and evaluation, so it can be applied directly on a personal laptop.

Highlights

In the past, the satellite market was reserved for a few companies and governments, which had the capacity to build and deploy large machinery in space, and the data obtained afterwards were used just by only a few research teams worldwide
We used the set of images from the vegetation observation satellite PROBA-V of the European Space Agency (ESA) [20] provided in the context of the ESA’s super resolution competition PROBA-V, which took place between 1 November 2018 and 31 May 2019 [25]
Thereafter, all possible MSE scores are calculated for each HRu,v patch, and the minimum score is taken. We found that this loss works quite well for the problem, but based on Zhao et al [32], we follow their recommendation to use the Mean Absolute Error loss (MAE) as a substitute

Summary

Introduction

The satellite market was reserved for a few companies and governments, which had the capacity (technical and monetary) to build and deploy large machinery in space, and the data obtained afterwards were used just by only a few research teams worldwide. A compound annual growth rate of 17% has been estimated for the small satellite market (forecast from 2020 to 2025) [1]. This expansion brings with it new challenges because of the vast amount of new data available. Satellite images are used in many different fields to accomplish a wide spectrum of tasks. Xu et al [2] investigated vegetation growth trends over time; Martinez et al [3] tracked tree growth through soil moisture monitoring; Ricker et al [4] studied Arctic ice growth decay; and Liu et al [5] developed a technique to extract deep features from high-resolution images for scene classification

Methods

Results

Discussion

Conclusion