Supervised Pixel-Wise GAN for Face Super-Resolution

Abstract

For many face-related multimedia applications, low-resolution face images may greatly degrade the face recognition performance and necessitate face super-resolution (SR). Among the current SR methods, MSE-oriented SR methods often produce over-smoothed outputs and could miss some texture details while GAN-oriented SR methods may generate artifacts which are harmful to face recognition. To resolve the above issues, this paper presents a supervised pixel-wise Generative Adversarial Network (SPGAN) that can resolve a very low-resolution face image of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$16\times 16$</tex-math></inline-formula> or smaller pixel-size to its larger version of multiple scaling factors ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$2\times$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$4\times$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$8\times$</tex-math></inline-formula> and even <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$16\times$</tex-math></inline-formula> ) in a unified framework. Being different from traditional unsupervised discriminators which generate a single number to represent the likelihood whether the input image is real or fake, the proposed supervised pixel-wise discriminator mainly focus on whether each pixel of the generated SR face image is as photo-realistic as its corresponding pixel in the ground-truth HR (high-resolution) face image. To further improve the face recognition performance of SPGAN, we take advantage of the face identity prior by sending two inputs to the discriminator, including an input face image (either a real HR face image or its corresponding SR face image) and its face features which are extracted from a pre-trained face recognition model. Due to the introduced face identity prior, the identity-based discriminator can pay more attention to texture details which are closely related to face recognition. Extensive experiments demonstrate that the proposed SPGAN can achieve more photo-realistic SR images and higher face recognition accuracy than some state-of-the-art methods.