Two-step ResUp&Down generative adversarial network to reconstruct multispectral image from aerial RGB image

Abstract

Convolutional neural network has brought breakthroughs on multispectral image reconstruction research. Previous work has largely focused on reconstructing MSI using the R-G-B channels from the MSI as inputs of the model. However, it’s image manipulation rather than practical use. In real application, to reconstruct multispectral image using images from RGB camera is a research that has hardly been studied. In this research, high resolution aerial RGB images are collected by drone with RGB camera and multispectral images are collected by drone with RedEdge-M multispectral Camera. Then a new two-step Generative Adversarial Network (GAN)-based reconstruction method was proposed as follows: At first, MSI and RGB images are carefully registered to make sure that pixels are one–one correspondent. Then two data sources are cropped to form dataset. After that, a novel R-MSI GAN using is proposed. It uses a ResUp&Down block to replace the ResNet block of the Generator network and it outperforms ResNet-based GAN. The experimental results show that: (1) the combination of Mean Square Error and Discriminator (MSE-D) can alleviate the problem of the high-frequency loss of generated images. (2) The root means square error (RMSE), mean relative absolute error (MRAE) and Structural Similarity (SSIM) can only reflect overall error but can’t reflect details in reconstructed images, while different bands' statistical histogram can present the total high-frequency loss of generated bands. (3) 3 indexes, which are intersection over union (IoU) based normalized difference vegetation index (NDVI)-IoU, normalized difference red edge (NDRE)-IoU and enhance vegetation index (EVI)-IoU, were defined to verify the effect of the generated MSI and they show good consistence with vegetation index map. 4 In comparisons among ResNet-based GAN, single step ResUp&Down GAN and two-step ResUp&Down GAN(T-GAN) with 3 loss functions (L1, MSE, Discriminator), the two-step ResUp&Down GAN(T-GAN) with MSE-D loss function performs best in reconstructing RGB bands. The T-GAN with L1loss-D (mean absolute error loss) performs best in reconstructing NIR and rededge bands. In summary, the proposed methods can effectively reconstruct MSI using images from RGB camera at drone based remote sensing.