Human face super-resolution on poor quality surveillance video footage

Abstract

Most super-resolution (SR) methods proposed to date do not use real ground-truth high-resolution (HR) and low-resolution (LR) image pairs; instead, the vast majority of methods use synthetic LR images generated from the HR images. This approach yields excellent performance on synthetic datasets, but on real-world poor quality surveillance video footage, they suffer from performance degradation. A promising alternative is to apply recent advances in style transfer for unpaired datasets, but state-of-the-art work along these lines has used LR images and HR images from completely different datasets, introducing more variation between the HR and LR domains than necessary. In this paper, we propose methods that overcome both of these shortcomings, applying unpaired style transfer learning methods to face SR but using HR and LR datasets that share important properties. The key is to acquire roughly paired training data from a high-quality main stream and a lower-quality sub-stream of the same IP camera. Based on this principle, we have constructed four datasets comprising more than 400 people, with 1–15 weakly aligned real HR–LR pairs for each subject. We adopt a cycle generative adversarial networks (Cycle GANs) approach that produces impressive super-resolved images for low-quality test images never seen during training. Experiments prove the efficacy of the method. The approach to face SR advocated for in this paper makes possible many real-world applications requiring the extraction of high-quality face images from low-resolution video streams such as those produced by security cameras. Developers of diverse applications such as face recognition, 3D face reconstruction, face alignment, face parsing, human–computer interaction, remote sensing, and access control will benefit from the methods introduced in this work.