Deep Internal Learning for Inpainting of Cloud-Affected Regions in Satellite Imagery

Mikolaj Czerkawski,Christopher Davison,Christos Tachtatzis,Javier Cardona,Robert Atkinson,Priti Upadhyay,Craig Michie,Ivan Andonovic,Astrid Werkmeister,Malcolm Macdonald

doi:10.3390/rs14061342

Abstract

Cloud cover remains a significant limitation to a broad range of applications relying on optical remote sensing imagery, including crop identification/yield prediction, climate monitoring, and land cover classification. A common approach to cloud removal treats the problem as an inpainting task and imputes optical data in the cloud-affected regions employing either mosaicing historical data or making use of sensing modalities not impacted by cloud obstructions, such as SAR. Recently, deep learning approaches have been explored in these applications; however, the majority of reported solutions rely on external learning practices, i.e., models trained on fixed datasets. Although these models perform well within the context of a particular dataset, a significant risk of spatial and temporal overfitting exists when applied in different locations or at different times. Here, cloud removal was implemented within an internal learning regime through an inpainting technique based on the deep image prior. The approach was evaluated on both a synthetic dataset with an exact ground truth, as well as real samples. The ability to inpaint the cloud-affected regions for varying weather conditions across a whole year with no prior training was demonstrated, and the performance of the approach was characterised.

Highlights

Cloud occlusion reduces the temporal availability and, in turn, the usability of optical satellite data, imposing a significant obstacle to applications where frequent sampling of the ground surface is necessary
MT and Multi-Source and Multi-Temporal (MS-MT) variants that utilised multi-temporal data consistently outperformed the other variants for both datasets and for both the full image and inpainted regions
It is clearly evident that the synthesis modes adjusted the content of the inpainted region based on the information available in the clear region, and the informing reference acted as a guide for generating the static structure of the image

Summary

Introduction

Cloud occlusion reduces the temporal availability and, in turn, the usability of optical satellite data, imposing a significant obstacle to applications where frequent sampling of the ground surface is necessary. In the field of precision agriculture, for tasks such as monitoring crop growth [1], crop classification [2], or crop yield prediction [3], gaps in data result in challenges related to the development of accurate models, variability in prediction performance, and the need for the use of data imputation. Open data sources, such as those generated by the EU Copernicus Sentinel missions [4], have been instrumental to the accelerated development of applications, enabling easy and wide access to large sets of high-resolution data at relatively short revisit periods.

Methods

Results

Conclusion