Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry

Michael J. Starek,Hamid Kamangir,Mohammad Pashaei,Jacob Berryhill

doi:10.3390/rs12111757

Michael J. Starek, Hamid Kamangir + Show 2 more

Open Access

https://doi.org/10.3390/rs12111757

Copy DOI

Journal: Remote Sensing	Publication Date: May 29, 2020
Citations: 21	License type: CC BY 4.0

Affiliation: Texas A&M University – Corpus Christi

Abstract

The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor × 4 . Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively.

Highlights

In most remote sensing (RS) applications, high-resolution (HR) images are usually more demanding in a wide range of image analysis tasks leading to more precise and accurate RS-derived products [1,2,3]
For quantitative evaluation of the single image super-resolution (SISR) performance, in this experiment with enhanced SISR generative adversarial network (GAN) (SRGAN) (ESRGAN) model, peak signal-to-noise ratio (PSNR) value and structural similarity (SSIM) index were calculated for the test image set and enhanced HR (HRenh) image set
The range of values for both PSNR and SSIM index in Table 2, resulting from evaluating ESRGAN performance on SRpre image set, is comparable in values reported for those image quality measures (IQMs) when ESRGAN, or any other high-performance deep convolutional neural network (DCNN)-based SISR model, is applied on standard SISR image sets [23,25,32]

Summary

Introduction

In most remote sensing (RS) applications, high-resolution (HR) images are usually more demanding in a wide range of image analysis tasks leading to more precise and accurate RS-derived products [1,2,3]. Regardless of the other factors contributing to the spatial resolution of imagery, such as focal length and the distance from sensor to the target, GSD of an image and the quality of its high-frequency contents deteriorate mainly due to some manufacturing limitations and imperfections of an imaging sensor. Those recovered fine details in SR images make predicted HR images more appealing to a human, and have a great impact on the accuracy and reliability of imaging geometry and scene details when they are retrieved by the SfM phtotogrammetry process. The SRGAN model has shown significant improvement on overall visual quality of SR images over all previously introduced PSNR-oriented methods [23,32].

Methods

Results

Discussion

Conclusion