Hallucination Approach Research Articles

On potentials of regularized Wasserstein generative adversarial networks for realistic hallucination of tiny faces

Super-resolution of facial images, a.k.a. face hallucination, has been intensively studied in the past decades due to the increasingly emerging analysis demands in video surveillance, e.g., face detection, verification, identification. However, the actual performance of most previous hallucination approaches will drop dramatically when a very low-res tiny face is provided, due to the challenging multimodality of the problem as well as lack of an informative prior as a strong semantic guidance. Inspired by the latest progress in deep unsupervised learning, this paper focuses on tiny faces of size 16 × 16 pixels, hallucinating them to their 8 × upsampling versions by exploring the potentials of Wasserstein generative adversarial networks (WGAN). Besides a pixel-wise L2 regularization term imposed to the generative model, it is found that our advocated autoencoding generator with both residual and skip connections is a critical component for WGAN representing the facial contour and semantic content to a reasonable precision. With the additional Lipschitz penalty and architectural considerations for the critic in WGAN, the proposed approach finally achieves state-of-the-art hallucination performance in terms of both visual perception and objective assessment. The cropped CelebA face dataset is primarily used to aid the tuning and analysis of the new method, termed as tfh-WGAN. Experimental results demonstrate that the proposed approach not only achieves realistic hallucination of tiny faces, but also adapts to pose, expression, illuminance and occluded variations to a great degree.

Face hallucination through differential evolution parameter map learning with facial structure prior

Current learning based face hallucination approaches mainly focus on how to design a reasonable objective function, such as using different assumptions and incorporating different regularization terms, but do not give a reasonable way of selecting the model parameters. In this paper, we propose to exploit the facial structure prior to learn a parameter map based on differential evolution. Specifically, we claim that different position patches have different parameter settings because of their different statistical properties, and patches from the same position of different face images should have similar parameter settings. As a result, we first learn a parameter map for each training sample by leveraging an evolutionary algorithm based on differential evolution, and then fuse these learned parameter maps to an optimal parameter map for testing via mean-pooling strategy. Finally, we use the predicted parameter map to guide the co-occurrence relationship modeling in different regions of the input low-resolution (LR) face image. Experimental results demonstrate that, even without seeing the ground truth, results of proposed parameter map learning method are comparable to or better than those traditional unified parameter setting methods and some recently proposed deep learning methods.

Context-Patch Face Hallucination Based on Thresholding Locality-Constrained Representation and Reproducing Learning.

Face hallucination is a technique that reconstructs high-resolution (HR) faces from low-resolution (LR) faces, by using the prior knowledge learned from HR/LR face pairs. Most state-of-the-arts leverage position-patch prior knowledge of the human face to estimate the optimal representation coefficients for each image patch. However, they focus only the position information and usually ignore the context information of the image patch. In addition, when they are confronted with misalignment or the small sample size (SSS) problem, the hallucination performance is very poor. To this end, this paper incorporates the contextual information of the image patch and proposes a powerful and efficient context-patch-based face hallucination approach, namely, thresholding locality-constrained representation and reproducing learning (TLcR-RL). Under the context-patch-based framework, we advance a thresholding-based representation method to enhance the reconstruction accuracy and reduce the computational complexity. To further improve the performance of the proposed algorithm, we propose a promotion strategy called reproducing learning. By adding the estimated HR face to the training set, which can simulate the case that the HR version of the input LR face is present in the training set, it thus iteratively enhances the final hallucination result. Experiments demonstrate that the proposed TLcR-RL method achieves a substantial increase in the hallucinated results, both subjectively and objectively. In addition, the proposed framework is more robust to face misalignment and the SSS problem, and its hallucinated HR face is still very good when the LR test face is from the real world. The MATLAB source code is available at https://github.com/junjun-jiang/TLcR-RL.

Face Hallucination with Linear Regression Model in Semi-Orthogonal Multilinear PCA Method

In this paper, we propose a new face hallucination technique, face images reconstruction in HSV color space with a semi-orthogonal multilinear principal component analysis method. This novel hallucination technique can perform directly from tensors via tensor-to-vector projection by imposing the orthogonality constraint in only one mode. In our experiments, we use facial images from FERET database to test our hallucination approach which is demonstrated by extensive experiments with high-quality hallucinated color faces. The experimental results assure clearly demonstrated that we can generate photorealistic color face images by using the SO-MPCA subspace with a linear regression model.

Super-resolution for face image with an improved K-NN search strategy

Recently, neighbor embedding based face super-resolution (SR) methods have shown the ability for achieving high-quality face images, those methods are based on the assumption that the same neighborhoods are preserved in both low-resolution (LR) training set and high-resolution (HR) training set. However, due to the “one-to-many” mapping between the LR image and HR ones in practice, the neighborhood relationship of the LR patch in LR space is quite different with that of the HR counterpart, that is to say the neighborhood relationship obtained is not true. In this paper, we explore a novel and effective re-identified K-nearest neighbor (RIKNN) method to search neighbors of LR patch. Compared with other methods, our method uses the geometrical information of LR manifold and HR manifold simultaneously. In particular, it searches K-NN of LR patch in the LR space and refines the searching results by re-identifying in the HR space, thus giving rise to accurate K-NN and improved performance. A statistical analysis of the influence of the training set size and nearest neighbor number is given, experimental results on some public face databases show the superiority of our proposed scheme over state-of-the-art face hallucination approaches in terms of subjective and objective results as well as computational complexity.

A CORRELATIVE TWO-STEP APPROACH TO HALLUCINATING FACES

Face hallucination is to synthesize high-resolution face image from the input low-resolution one. Although many two-step learning-based face hallucination approaches have been developed, they suffer from the expensive computational cost due to the separate calculation of the global and local models. To overcome this problem, we propose a correlative two-step learning-based face hallucination approach which bridges the gap between the global model and the local model. In the global phase, we build a global face hallucination framework by combining the steerable pyramid decomposition and the reconstruction. In the residue compensation phase, based on the combination weights and constituent samples obtained in the global phase, a residue face image is synthesized by the neighbor reconstruction algorithm to compensate the hallucinated global face image with subtle facial features. The ultimate hallucinated result is synthesized by adding the residue face image to the global face image. Compared with existing methods, in the global phase, our global face image is more similar to the original high-resolution face image. Furthermore, in the residue compensation phase, we use the combination weights and constituent samples obtained in the global phase to compute the residue face image, by which the computational efficiency can be greatly improved without compromising the quality of facial details. The experimental results and comparisons demonstrate that our approach can not only generate convincible high-resolution face images efficiently, but also has high computational efficiency. Furthermore, our proposed approach can be used to restore the damaged face images in image inpainting. The efficacy of our approach is validated by recovering the damaged face images with visually good results.

Noise Robust Face Hallucination via Locality-Constrained Representation

Recently, position-patch based approaches have been proposed to replace the probabilistic graph-based or manifold learning-based models for face hallucination. In order to obtain the optimal weights of face hallucination, these approaches represent one image patch through other patches at the same position of training faces by employing least square estimation or sparse coding. However, they cannot provide unbiased approximations or satisfy rational priors, thus the obtained representation is not satisfactory. In this paper, we propose a simpler yet more effective scheme called Locality-constrained Representation (LcR). Compared with Least Square Representation (LSR) and Sparse Representation (SR), our scheme incorporates a locality constraint into the least square inversion problem to maintain locality and sparsity simultaneously. Our scheme is capable of capturing the non-linear manifold structure of image patch samples while exploiting the sparse property of the redundant data representation. Moreover, when the locality constraint is satisfied, face hallucination is robust to noise, a property that is desirable for video surveillance applications. A statistical analysis of the properties of LcR is given together with experimental results on some public face databases and surveillance images to show the superiority of our proposed scheme over state-of-the-art face hallucination approaches.

Face Hallucination Via Weighted Adaptive Sparse Regularization

Sparse representation-based face hallucination approaches proposed so far use fixed l1 norm penalty to capture the sparse nature of face images, and thus hardly adapt readily to the statistical variability of underlying images. Additionally, they ignore the influence of spatial distances between the test image and training basis images on optimal reconstruction coefficients. Consequently, they cannot offer a satisfactory performance in practical face hallucination applications. In this paper, we propose a weighted adaptive sparse regularization (WASR) method to promote accuracy, stability and robustness for face hallucination reconstruction, in which a distance-inducing weighted lq norm penalty is imposed on the solution. With the adjustment to shrinkage parameter q , the weighted lq penalty function enables elastic description ability in the sparse domain, leading to more conservative sparsity in an ascending order of q . In particular, WASR with an optimal q > 1 can reasonably represent the less sparse nature of noisy images and thus remarkably boosts noise robust performance in face hallucination. Various experimental results on standard face database as well as real-world images show that our proposed method outperforms state-of-the-art methods in terms of both objective metrics and visual quality.

Super-Resolution for Surveillance Facial Images via Shape Prior and Residue Compensation

Super-resolution (SR) methods are widely used to enhance the resolution of input images such as low-resolution (LR) remote or surveillance images. The key problem of SR algorithm is to introduce a proper prior for high-resolution (HR) image reconstruction. Usually pixel feature based similarity measure is used as prior. While in real surveillance applications, images are often disturbed by noise so that it is not enough to only use the prior from pixel domain to achieve satisfied high-resolution. In this paper, a two-phase based face hallucination approach via shape prior (SP) is proposed. Firstly, the Active Appearance Model (AAM) is used as shape prior to reconstruct the global face. Then the high frequency information as detailed inartificial facial features of HR face image is compensated to the global face. Experiments show that the proposed algorithm improves the subjective and objective quality of the input LR facial images and outperform many states-of-the- art superresolution methods.

A two-step face hallucination approach for video surveillance applications

In this paper we propose a novel face hallucination algorithm to synthesize a high-resolution face image from several low-resolution input face images. Face hallucination normally uses two models: a global parametric model which synthesizes the global face shapes from eigenfaces, and a local parametric model which enhances the local high frequency details. We follow a similar process to develop a robust face hallucination algorithm. First, we obtain eigenfaces from a number of low resolution face images segmented from a video sequence using a face tracking algorithm. Then we compute the difference between an interpolated low-resolution face and a mean face, and use this difference as the query to retrieve an approximate sparse eigenface representation. The eigenfaces are combined using the coefficients obtained from the sparse representation and added into the interpolated low-resolution face. In this way, the global shape of the interpolated low resolution face can be successfully enhanced. Second, we improve the example-based super-resolution method for local high frequency information enhancement. Our proposed algorithm uses the Approximate Nearest Neighbors (ANN) search method to find a number of nearest neighbors for a stack of queries, instead of finding the exact match for each low frequency patch. Median filtering is used to remove the noise from the nearest neighbors in order to enhance the signal. Our proposed algorithm uses a sparse representation and the ANN method to enhance both global face shape and local high frequency information while greatly improving the processing speed, as confirmed empirically.

An innovative face image enhancement based on principle component analysis

In this paper, we propose an innovative face hallucination approach based on principle component analysis (PCA) and residue technique. First, the relationship of projection coefficients between high-resolution and low-resolution images using PCA is investigated. Then based on this analysis, a high resolution global face image is constructed from a low resolution one. Next a high-resolution residue is derived based on the similarity between the projections on high and low resolution residue training sets. Finally by combining the global face and residue in high resolution, a high resolution face image is generated. Also the recursive and two-stage methods are proposed, which improve the results of face image enhancement. Extensive experiments validate the proposed approaches.

Generalized Color Face Hallucination with Linear Regression Model in MPCA

This paper proposes a novel hallucination technique for color face image reconstruction in the RGB, YCbCr, HSV and CIELAB color systems. Our hallucination method depends on multilinear principal component analysis (MPCA) with a linear regression model. In the hallucination framework, many color face images are expressed in color spaces. These images can be naturally described as tensors or multilinear arrays. This novel hallucination technique can perform feature extraction by determining a multilinear projection that captures most of the original tensorial input variation. In our experiments, we used facial images from the FERET database to test our hallucination approach which is demonstrated by extensive experiments with high-quality hallucinated color face images. The experimental results show that a correlation between the color channel and the proposed hallucination method can reduce the complexity in the color face hallucination process.

Hallucinating faces: LPH super-resolution and neighbor reconstruction for residue compensation

A two-phase face hallucination approach is proposed in this paper to infer high-resolution face image from the low-resolution observation based on a set of training image pairs. The proposed locality preserving hallucination (LPH) algorithm combines locality preserving projection (LPP) and radial basis function (RBF) regression together to hallucinate the global high-resolution face. Furthermore, in order to compensate the inferred global face with detailed inartificial facial features, the neighbor reconstruction based face residue hallucination is used. Compared with existing approaches, the proposed LPH algorithm can generate global face more similar to the ground truth face efficiently, moreover, the patch structure and search strategy carefully designed for the neighbor reconstruction algorithm greatly reduce the computational complexity without diminishing the quality of high-resolution face detail. The details of synthetic high-resolution face are further improved by a global linear smoother. Experiments indicate that our approach can synthesize distinct high-resolution faces with various facial appearances such as facial expressions, eyeglasses efficiently.