Recurrently-Trained Super-Resolution

Abstract

We are motivated by the observation that for problems where inputs and outputs are in the same form such as in image enhancement, deep neural networks can be reinforced by retraining the network using a new target set to the output for the original target. As an example, we introduce a new learning strategy for super-resolution by recurrently training the same simple network. Unlike the existing self-trained SR, which involves a single stage of learning with multiple runs at test time, our method trains the same SR network multiple times with increasingly better targets requiring only a single inference at test time. At each stage of the proposed learning scheme, a new target for training is obtained by applying the most recently trained SR network to the original image and downscaling the resultant SR image to normalize the size. Even if downscaling is involved, we argue that the downscaled SR image acts as a better target compared to the old target. We could mathematically demonstrate that this process is similar to unsharp masking when it is linearly approximated and that this process makes the image sharper. However, unlike unsharp masking, the proposed recurrent learning tends to converge to a specific target. By retraining the existing network aiming at a more enhanced target, the proposed method can achieve a similar effect of applying SR multiple times without increasing implementation cost and inference time. To objectively verify the supremacy of our approach by experiments, we propose to use VIQET MOS, which does not require a reference image as a measure of image quality. As far as we know, our work of using an objective quality measure in image enhancement is the first one whose validity was verified by showing similar results to the actual user's subjective evaluation. The proposed recurrent learning scheme makes existing SR algorithms more useful by clearly improving the effect of SR. Codes are available at https://github.com/rtsr82/rtsr.git.