Facial Image Super-resolution Research Articles

Facial image super-resolution using progressive network interleaved correlation filter

Facial image super-resolution using progressive network interleaved correlation filter

Does Deep Learning-Based Super-Resolution Help Humans With Face Recognition?

The last decade witnessed a renaissance of machine learning for image processing. Super-resolution (SR) is one of the areas where deep learning techniques have achieved impressive results, with a specific focus on the SR of facial images. Examining and comparing facial images is one of the critical activities in forensic video analysis; a compelling question is thus whether recent SR techniques could help face recognition (FR) made by a human operator, especially in the challenging scenario where very low resolution images are available, which is typical of surveillance recordings. This paper addresses such a question through a simple yet insightful experiment: we used two state-of-the-art deep learning-based SR algorithms to enhance some very low-resolution faces of 30 worldwide celebrities. We then asked a heterogeneous group of more than 130 individuals to recognize them and compared the recognition accuracy against the one achieved by presenting a simple bicubic-interpolated version of the same faces. Results are somehow surprising: despite an undisputed general superiority of SR-enhanced images in terms of visual appearance, SR techniques brought no considerable advantage in overall recognition accuracy.

Robust Facial Image Super-Resolution by Kernel Locality-Constrained Coupled-Layer Regression

Super-resolution methods for facial image via representation learning scheme have become very effective methods due to their efficiency. The key problem for the super-resolution of facial image is to reveal the latent relationship between the low-resolution ( LR ) and the corresponding high-resolution ( HR ) training patch pairs. To simultaneously utilize the contextual information of the target position and the manifold structure of the primitive HR space, in this work, we design a robust context-patch facial image super-resolution scheme via a kernel locality-constrained coupled-layer regression (KLC2LR) scheme to obtain the desired HR version from the acquired LR image. Here, KLC2LR proposes to acquire contextual surrounding patches to represent the target patch and adds an HR layer constraint to compensate the detail information. Additionally, KLC2LR desires to acquire more high-frequency information by searching for nearest neighbors in the HR sample space. We also utilize kernel function to map features in original low-dimensional space into a high-dimensional one to obtain potential nonlinear characteristics. Our compared experiments in the noisy and noiseless cases have verified that our suggested methodology performs better than many existing predominant facial image super-resolution methods.

Features Guided Face Super-Resolution via Hybrid Model of Deep Learning and Random Forests.

Face hallucination or super-resolution is a practical application of general image super-resolution which has been recently studied by many researchers. The challenge of good face hallucination comes from a variety of poses, illuminations, facial expressions, and other degradations. In many proposed methods, researchers resolve it by using a generative neural network to reduce the perceptual loss so we can generate a photo-realistic image. The problem is that researchers usually overlook the fidelity of the super-resolved image which could affect further facial image processing. Meanwhile, many CNN based approaches cascade multiple networks to extract facial prior information to improve super-resolution quality. Because of the end-to-end design, the details are missing for investigation. In this paper, we combine new techniques in convolutional neural network and random forests to a Hierarchical CNN based Random Forests (HCRF) approach for face super-resolution in a coarse-to-fine manner. In the proposed approach, we focus on a general approach that can handle facial images with various conditions without pre-processing. To the best of our knowledge, this is the first paper that combines the advantages of deep learning with random forests for face super-resolution. To achieve superior performance, we propose two novel CNN models for coarse facial image super-resolution and segmentation and then apply new random forests to target on local facial features refinement making use of the segmentation results. Extensive benchmark experiments on subjective and objective evaluation show that HCRF can achieve comparable speed and competitive performance compared with state-of-the-art super-resolution approaches for very low-resolution images.

Facial image super-resolution guided by adaptive geometric features

This paper addresses the traditional issue of restoring a high-resolution (HR) facial image from a low-resolution (LR) counterpart. Current state-of-the-art super-resolution (SR) methods commonly adopt the convolutional neural networks to learn a non-linear complex mapping between paired LR and HR images. They discriminate local patterns expressed by the neighboring pixels along the planar directions but ignore the intrinsic 3D proximity including the depth map. As a special case of general images, the face has limited geometric variations, which we believe that the relevant depth map can be learned and used to guide the face SR task. Motivated by it, we design a network including two branches: one for auxiliary depth map estimation and the other for the main SR task. Adaptive geometric features are further learned from the depth map and used to modulate the mid-level features of the SR branch. The whole network is implemented in an end-to-end trainable manner under the extra supervision of depth map. The supervisory depth map is either a paired one from RGB-D scans or a reconstructed one by a 3D prior model of faces. The experiments demonstrate the effectiveness of the proposed method and achieve improved performance over the state of the arts.

Joint Face Super-Resolution and Deblurring Using Generative Adversarial Network

Facial image super-resolution (SR) is an important aspect of facial analysis, and it can contribute significantly to tasks such as face alignment, face recognition, and image-based 3D reconstruction. Recent convolutional neural network (CNN) based models have exhibited significant advancements by learning mapping relations using pairs of low-resolution (LR) and high-resolution (HR) facial images. However, because these methods are conventionally aimed at increasing the PSNR and SSIM metrics, the reconstructed HR images might be blurry and have an overall unsatisfactory perceptual quality even when state-of-the-art quantitative results are achieved. In this study, we address this limitation by proposing an adversarial framework intended to reconstruct perceptually high-quality HR facial images while simultaneously removing blur. To this end, a simple five-layer CNN is employed to extract feature maps from LR facial images, and this feature information is provided to two-branch encoder-decoder networks that generate HR facial images with and without blur. In addition, local and global discriminators are combined to focus on the reconstruction of HR facial structures. Both qualitative and quantitative results demonstrate the effectiveness of the proposed method for generating photorealistic HR facial images from a variety of LR inputs. Moreover, it was also verified, through a use case scenario that the proposed method can contribute more to the field of face recognition than existing approaches.

On potentials of regularized Wasserstein generative adversarial networks for realistic hallucination of tiny faces

Super-resolution of facial images, a.k.a. face hallucination, has been intensively studied in the past decades due to the increasingly emerging analysis demands in video surveillance, e.g., face detection, verification, identification. However, the actual performance of most previous hallucination approaches will drop dramatically when a very low-res tiny face is provided, due to the challenging multimodality of the problem as well as lack of an informative prior as a strong semantic guidance. Inspired by the latest progress in deep unsupervised learning, this paper focuses on tiny faces of size 16 × 16 pixels, hallucinating them to their 8 × upsampling versions by exploring the potentials of Wasserstein generative adversarial networks (WGAN). Besides a pixel-wise L2 regularization term imposed to the generative model, it is found that our advocated autoencoding generator with both residual and skip connections is a critical component for WGAN representing the facial contour and semantic content to a reasonable precision. With the additional Lipschitz penalty and architectural considerations for the critic in WGAN, the proposed approach finally achieves state-of-the-art hallucination performance in terms of both visual perception and objective assessment. The cropped CelebA face dataset is primarily used to aid the tuning and analysis of the new method, termed as tfh-WGAN. Experimental results demonstrate that the proposed approach not only achieves realistic hallucination of tiny faces, but also adapts to pose, expression, illuminance and occluded variations to a great degree.

A robust facial image super-resolution model via mirror-patch based neighbor representation

Many patch-based facial image super-resolution (or hallucination) techniques have been proposed in literature but all of them fail in the presence of high-density impulse noise and occlusion. A novel mirror-patch based neighbor representation (MPNR) model is proposed here which uses mirror-patch based data fidelity along with the input-patch based fidelity in low-resolution (LR) space to address the above problem. The computation of mirror-patch based data fidelity helps in compensating the corrupted features of an input patch through its mirror-patch. The objective function of the proposed model is designed in such a way that it hallucinate the input LR faces and takes care of occlusion/heavy noise effect simultaneously in the reconstruction process. It is conspicuous from experimental results attained on FEI and CAS-PEAL-R1 databases that the proposed MPNR model has outperformed all the comparative methods.

Grey wolf optimization algorithm for facial image super-resolution

Face image super-resolution (FSR) algorithms are capable of providing a high-resolution image from an input low-resolution (LR) image. Various FSR algorithms use a set of training examples to reconstruct the input LR image. For the purpose, proper weights need to be calculated for each training image. In general, the least square estimation approach is used for obtaining optimal reconstruction weights, known as least square representation (LSR) problem. In this paper, to minimize LSR problem more effectively, a grey wolf optimizer (GWO) based FSR algorithm (FSR-GWO) is proposed. To make search process of GWO algorithm suitable to FSR, a new formulation for upper-bound and lower-bound is introduced. Performance comparison with state-of-the-art nature-inspired algorithms and several super-resolution methods on FEI public face database shows the effectiveness of the proposed FSR-GWO algorithm.

Expanding Training Data for Facial Image Super-Resolution.

The quality of training data is very important for learning-based facial image super-resolution (SR). The more similarity between training data and testing input is, the better SR results we can have. To generate a better training set of low/high resolution training facial images for a particular testing input, this paper is the first work that proposes expanding the training data for improving facial image SR. To this end, observing that facial images are highly structured, we propose three constraints, i.e., the local structure constraint, the correspondence constraint and the similarity constraint, to generate new training data, where local patches are expanded with different expansion parameters. The expanded training data can be used for both patch-based facial SR methods and global facial SR methods. Extensive testings on benchmark databases and real world images validate the effectiveness of training data expansion on improving the SR quality.

Locally nonlinear transformation for facial image superresolution

Reconstruction of a high-resolution face image, from a low-res- olution observation based on a set of high- and low-resolution training image pairs, is an important problem for optical engineering applications. In this paper, we study this facial superresolution problem and propose a novel locally nonlinear transformation based approach. Multiple locally nonlinear transformation are utilized to approximate the global nonlinear connections between low resolution (LR)/high resolution (HR) images. LR/ HR images are initially divided into multiple pairs of patches with the cor- responding position information. As facial images are highly structured, patches at the same position spanned a subspace. Since the curse of dimensionality is avoided in these subspaces (patches in the same posi- tion), the Euclidean distance can express the intrinsic between samples in the same subspace. Therefore, multiple radial basis functions are utilized to approximate the nonlinear mapping between LR/HR pairs at each position from training examples. The proposed locally nonlinear transformation (LNT)-based reconstruction is achieved by applying the learned nonlinear transformation to each position patch of an LR input. The final SR results are obtained by refining the LNT reconstruction by the projection onto a convex sets algorithm using the consistency con- straint. Extensive experiments on benchmark databases and real world images validate the superiority of the proposed method. © 2013 Society of

Facial Image Super-Resolution Reconstruction Based on Separated Frequency Components

Facial Image Super-Resolution Reconstruction Based on Separated Frequency Components

Manifold learning-based sample selection method for facial image super-resolution

Learning efficiency, related to not only the size of training set but also to the usage of samples, is an important issue of learning-based super-resolution (SR). We propose a face hallucination method with adaptive sample selection ability by learning from the facial manifold. Employing locality preserving projections (LPP) to analyze the intrinsic features on the local facial manifold, our method searches out image patches dynamically in the LPP subspace, which makes the training set tailored to the input patch. Using the selected training set, we develop a patch-based eigen-transformation algorithm to efficiently restore the lost high-frequency components of the low-resolution face image. Experiments on synthetic and real-life images fully demonstrate that the proposed adaptive sample selection SR method can achieve better performance than some state-of-the-art learning-based SR techniques with less computational cost by utilizing a relative small sample, especially under the case of low quality input images.

Fast Facial Image Super-Resolution via Local Linear Transformations for Resource-Limited Applications

Most popular learning-based super-resolution (SR) approaches suffer from complicated learning structures and highly intensive computation, especially in resource-limited applications. We propose a novel frontal facial image SR approach by using multiple local linear transformations to approximate the nonlinear mapping between low-resolution (LR) and high-resolution (HR) images in the pixel domain. We adopt Procrustes analysis to obtain orthogonal matrices representing the learned linear transformations, which cannot only well capture appearance variations in facial patches but also greatly simplify the transformation computation to matrices manipulation. An HR image can be directly reconstructed from a single LR image without need of the large training data, thus avoiding the use of a large redundant LR and HR patch database. Experimental results show that our approach is computationally fast as well the SR quality compares favorably with the state-of-the-art approaches from both subjective and objective evaluations. Besides, our approach is insensitive to the size of training data and robust to a wide range of facial variations like occlusions. More importantly, the proposed method is also much more effective than other comparative methods to reconstruct real-world images captured from the Internet and webcams.