Cloud Removal with Fusion of High Resolution Optical and SAR Images Using Generative Adversarial Networks

Jianhao Gao,Jie Li,Qiangqiang Yuan,Xin Su,Hai Zhang

doi:10.3390/rs12010191

Abstract

The existence of clouds is one of the main factors that contributes to missing information in optical remote sensing images, restricting their further applications for Earth observation, so how to reconstruct the missing information caused by clouds is of great concern. Inspired by the image-to-image translation work based on convolutional neural network model and the heterogeneous information fusion thought, we propose a novel cloud removal method in this paper. The approach can be roughly divided into two steps: in the first step, a specially designed convolutional neural network (CNN) translates the synthetic aperture radar (SAR) images into simulated optical images in an object-to-object manner; in the second step, the simulated optical image, together with the SAR image and the optical image corrupted by clouds, is fused to reconstruct the corrupted area by a generative adversarial network (GAN) with a particular loss function. Between the first step and the second step, the contrast and luminance of the simulated optical image are randomly altered to make the model more robust. Two simulation experiments and one real-data experiment are conducted to confirm the effectiveness of the proposed method on Sentinel 1/2, GF 2/3 and airborne SAR/optical data. The results demonstrate that the proposed method outperforms state-of-the-art algorithms that also employ SAR images as auxiliary data.

Highlights

Great numbers of remote sensing data have been acquired and played an even more important role in Earth observation and land monitoring in recent years
Taking into consideration the advantages and disadvantages of approaches mentioned above, we propose a novel framework to reconstruct the missing parts of optical images with single-temporal synthetic aperture radar (SAR) images as auxiliary data based on the latest development of generative adversarial network (GAN) in this paper
Inspired by the great success in style transfer work achieved by deep learning, we introduced a classicIanlsdpeireepdlbeyartnhienggrmeaotdseulcUce-snsetin[3s5ty],lewthriacnhsfsehrowwosriktsascuhpieevreiodribtyy idneesepmleaanrtnicinsge,gwmeeninttartoiodnuctaesdkas aIgancaIgfsrrunleosrattnouhhtptshuchnseirtseiniodicssopdvsaitmpniirlrmdorutdauroectuerulhcdeeaeltesahtpadpssitnso,ierlo,adewontnnawavaneirdinlecndeewceitdwaeuntwidcwrsgiicoanshdouhmtrreterkthkoattoooatetdtoboeiolofejbeoetlbotrtdclrUatabltaaoi-nij-nnnetiwnsnocslset-tliahsoa-inttmtitebeg[moe3j.-ufea5aouocl]bSlaltS,laAjtomAwetewecRdRhadtipiionicmmopmphgpia.atnastipggigchcpeae.oaliTwttlnooihimgsmaae.ianntasTgstgoorheseuppsuescttpwiitwsccuetaairrirtltliuehhoiicmomrstsifiutiamatmyrggheiieeielnlaaoininrnsrfeesatsamtpwpnhneeaeoococnrbttbnktrrjijaeeaceactllcnwstiti-edn-ntogtoffotromo-hk-orroeemmbbanljojneaateasctdtcsiitttooimfotmnnuhnanettantoocnatnlnstotieohhkesrneessr

Summary

Introduction

Great numbers of remote sensing data have been acquired and played an even more important role in Earth observation and land monitoring in recent years. A large proportion of remote sensing data are destructed due to the unavoidable existence of thick/thin clouds, which enormously increases the difficulties of processing and restrains further applications. As is demonstrated in [2], traditional reconstruction approaches could be classified into three main types according to the difference of homogeneous auxiliary data source: spatial-based approaches, spectral-based approaches and multitemporal-based approaches. Some novel approaches based on the heterogeneous auxiliary data source, mainly synthetic aperture radar (SAR) data, have been developed in recent years and proved their effectiveness in practice, which are termed as SAR-based approaches for convenience in this paper. A compendious review of three varieties of traditional reconstruction approaches and SAR-based approaches is presented below

Methods

Results

Conclusion