Adding Noise to Super-Resolution Training Set: Method to Denoise Super Resolution for Structure from Motion Preprocessing

Abstract

The resolution and noise levels of input images directly affect the three-dimensional (3D) structure-from-motion (SfM) reconstruction performance. Conventional super-resolution (SR) methods focus too little on denoising, and latent image noise becomes worse when resolution is improved. This study proposes two SR denoising training algorithms to simultaneously improve resolution and noise: add-noise-before-downsampling and downsample-before-adding-noise. These portable methods preprocess low-resolution training images using real-world noise samples instead of altering the basic neural network. Hence, they concurrently improve resolution while reducing noise for an overall cleaner SfM performance. We applied these methods to the existing SR network: super-resolution convolutional neural network, enhanced deep residual super-resolution, residual channel attention network, and efficient super-resolution transformer, comparing their performances with those of conventional methods. Impressive peak signal-to-noise and structural similarity improvements of 0.12 dB and 0.56 were achieved on the noisy images of Smartphone Image Denoising Dataset, respectively, without altering the network structure. The proposed methods caused a very small loss (<0.01 dB) on clean images. Moreover, using the proposed SR algorithm makes the 3D SfM reconstruction more complete. Upon applying the methods to non-preprocessed and conventionally preprocessed models, the mean projection error was reduced by a maximum of 27% and 4%, respectively, and the number of 3D densified points was improved by 310% and 7%, respectively.