High-resolution Face Image Research Articles

SRLSP: A Face Image Super-Resolution Algorithm Using Smooth Regression With Local Structure Prior

The performance of traditional face recognition systems is sharply reduced when encountered with a low-resolution (LR) probe face image. To obtain much more detailed facial features, some face super-resolution (SR) methods have been proposed in the past decade. The basic idea of a face image SR is to generate a high-resolution (HR) face image from an LR one with the help of a set of training examples. It aims at transcending the limitations of optical imaging systems. In this paper, we regard face image SR as an image interpolation problem for domain-specific images. A missing intensity interpolation method based on smooth regression with a local structure prior (LSP), named SRLSP for short, is presented. In order to interpolate the missing intensities in a target HR image, we assume that face image patches at the same position share similar local structures, and use smooth regression to learn the relationship between LR pixels and missing HR pixels of one position patch. Performance comparison with the state-of-the-art SR algorithms on two public face databases and some real-world images shows the effectiveness of the proposed method for a face image SR in general. In addition, we conduct a face recognition experiment on the extended Yale-B face database based on the super-resolved HR faces. Experimental results clearly validate the advantages of our proposed SR method over the state-of-the-art SR methods in face recognition application.

A new approach for face hallucination based on a two-dimensional direct combined model

This study develops an example-based face hallucination system based on a novel two-dimensional direct combined model (2DDCM) approach. The 2DDCM model combines the low-resolution and high-resolution pairwise images in the training set in a combined (or concatenated) matrix form in order to better preserve the correlation between the two images during the system learning process. Notably, the images processed by the 2DDCM model have the form of two-dimensional (2D) matrices rather than one-dimensional (1D) vectors, and hence the facial geometry features in the vertical and horizontal directions can be more reliably extracted. The proposed hallucination system comprises two 2DDCM-based modules, namely a global module for global facial structure reconstruction and a local module for facial texture detail compensation. In implementing the local module, a 2DDCM-based bi-directional transformation method is adopted to identify the detailed facial textures which are lost in the global synthesis process. The experimental results show that the synthesized results obtained using the proposed 2DDCM framework are in good quantitative agreement with the ground-truth images. Moreover, the proposed framework demonstrates the ability to synthesize high-resolution facial images given only a small number of training pairs, even when the facial features, alignment and appearance of the testing image differ from those of the original training set. Finally, the 2DDCM representation ensures that the synthesized results better preserve the subject-specific characteristics of the input facial image, and therefore improves the performance of downstream applications such as automatic face recognition.

이질적 얼굴인식을 위한 심층 정준상관분석을 이용한 지역적 얼굴 특징 학습 방법

Face recognition has received a great deal of attention for the wide range of applications in real-world scenario. In this scenario, mismatches (so called heterogeneity) in terms of resolution and illumination between gallery and test face images are inevitable due to the different capturing conditions. In order to deal with the mismatch problem, we propose a local feature learning method using deep canonical correlation analysis (DCCA) for heterogeneous face recognition. By the DCCA, we can effectively reduce the mismatch between the gallery and the test face images. Furthermore, the proposed local feature learned by the DCCA is able to enhance the discriminative power by using facial local structure information. Through the experiments on two different scenarios (i.e., matching near-infrared to visible face images and matching low-resolution to high-resolution face images), we could validate the effectiveness of the proposed method in terms of recognition accuracy using publicly available databases.

Face super resolution based on parent patch prior for VLQ scenarios

Face Super Resolution (FSR) is to infer High Resolution (HR) facial images from given Low Resolution (LR) ones with the assistance of LR and HR training pairs. Among existing methods, Neighbor Embedding(NE) FSR methods are superior in visual and objective quality than holistic based methods. These NE methods are based on the consistency assumption that the neighbors in HR/LR space form similar local geometry. But when LR images are in Very Low Quality (VLQ), the LR patches are seriously contaminated that even two distinct patches form similar appearance, which means that the consistency assumption is not well held anymore. To solve this problem, in this paper we use the target patch as well as the surrounding pixels, which we call parent patch, to represent the target patch. By incorporating the peripheral information, the parent patch is much more robust to noise in the LR and HR consistency learning. The effectiveness of proposed method is verified both quantitatively and qualitatively. In this paper, we also discuss the boundary and the paradox of the multi-scaled parent patch prior in NE based FSR framework.

Face super-resolution reconstruction and recognition using non-local similarity dictionary learning based algorithm

One of the challenges of face recognition in surveillance is the low resolution of face region. Therefore many superresolution (SR) face reconstruction methods are proposed to produce a high-resolution face image from one or a set of low-resolution face images. However, existing dictionary learning based algorithms are sensitive to noise and very time-consuming. In this paper, we define and prove the multi-scale linear combination consistency. In order to improve the performance of SR, we propose a novel SR face reconstruction method based on nonlocal similarity and multi-scale linear combination consistency (NLS-MLC). We further proposed a new recognition approach for very low resolution face images based on resolution scale invariant feature (RSIF). A series of experiments are conducted on two public face image databases to test feasibility of our proposed methods. Experimental results show that the proposed SR method is more robust and computationally effective in face hallucination, and the recognition accuracy of RSIF is higher than some state-of-art algorithms.

Face recognition using support vector model classifier for user authentication

Most existing user authentication approaches for detecting fraud in e-commerce applications have focused on Secure Sockets Layer (SSL)-based authentication to inspect a username and a password from a server, rather than the inspection of personal biometric information. Because of the lack of support for mutual authentication or two-way authentication between a consumer and a mercantile agent, one-way SSL authentication cannot prevent man-in-the-middle attacks. In practice, in user authentication systems, machine learning and the generalisation capability of support vector models (SVMs) are used to guarantee a small classification error. This study developed an online face-recognition system by training an SVM classifier based on user facial features associated with wavelet transforms and a spatially enhanced local binary pattern. A cross-validation scheme and SVMs associated with the Olivetti Research Laboratory database of user facial features were used for solving classification precision problems. Experimental results showed that the classification error decreased with an increase in the size of the training samples. By using the aggregation of both the low-resolution and the high-resolution face image samples, the global precision of face recognition was over 97% with tenfold cross-validation scheme for an image data size of 168 and 341, respectively. Overall, the proposed scheme provided a higher precision of face recognition compared with the average precision for low-resolution face image (approximately 89%) of the existing schemes.

HRM graph constrained dictionary learning for face image super-resolution

Sparse coding based face image Super-Resolution (SR) approaches have received increasing amount of interest recently. However, most of the existing sparse coding based approaches fail to consider the geometrical structure of the face space, as a result, artificial effects on reconstructed High Resolution (HR) face images come into being. In this paper, a novel sparse coding based face image SR method is proposed to reconstruct a HR face image from a Low Resolution (LR) observation. In training stage, it aims to get a more expressive HR-LR dictionary pair for certain input LR patch. The intrinsic geometric structure of training samples is incorporated into the sparse coding procedure for dictionary learning. Unlike the existing SR methods which use the graph constructed in LR Manifold (LRM) as regularization term, the proposed method uses graph constructed in HR Manifold (HRM) as regularization term. In reconstruction stage, K selection mean constrains is used in l1 convex optimization, aiming at finding an optimal weight for HR face image patch reconstruction. Experimental results on both simulation and real world images suggest that our proposed one achieves better quality when compared with other state-of-the-art methods.

A novel face super resolution approach for noisy images using contour feature and standard deviation prior

Face Super Resolution (FSR) is to infer high resolution (HR) face images from given low resolution (LR) face images with the help of HR/LR training examples. The most representative FSR is NE methods, which are based on the consistency assumption that the HR/LR patch pairs form similar local geometric structures. But NE methods have difficulty in dealing with noisy facial images. The reason lies in the wrong neighborhood relationship caused by low quality scenarios that even two distinct patches have similar relation in local geometry. Therefore, the consistency assumption is not well held anymore. This paper presents a novel FSR approach suitable for noisy facial images. Our work are twofold. Firstly, different from the existing methods which directly enhance the noisy input image in intensity feature space, the proposed method introduces a contour feature which is robust to noise. By applying the contour feature as constraint, the noise effects can be effectively suppressed. Secondly, different from the existing methods which directly constrain the noisy input image with low quality contour feature, a standard deviation prior is proposed to enhance the low quality contour feature. Through enhancing the contour feature into high quality, the FSR reconstruction can be better constrained. Both simulation and the real-world scenario experiments demonstrate that the proposed approach outperforms most classic methods both quantitatively and qualitatively.

Low-quality facial biometric verification via dictionary-based random pooling

In the past decade, visual surveillance has emerged as an effective tool in public security applications. Due to the technical limitations of both surveillance cameras and transmission speed, videos collected from surveillance sites are usually of low resolution. Especially, facial images at a distance in surveillance videos are usually at very low quality, making it difficult to carry out automated facial biometric verification. To handle with this challenge, in this work, we introduce dictionary based techniques to cope with low quality facial images, and propose a random pooling scheme to enhance the accuracy of facial biometric verification. In the proposed scheme, a dictionary is first learned from paired low-resolution and high-resolution facial images, and the input low-resolution query face can then be modelled by a set of high-resolution visual words via a dictionary lookup. A random pooling strategy is then applied to select subsets of visual words, and kernel Fisher׳s linear discriminant analysis (k-LDA) is introduced to find the discriminant metrics. The final decision is based on the average over different pooling results. The experiment on three publically available face datasets validated that our proposed scheme can robustly cope with the challenges from low quality facial images, and attained an improved accuracy over all datasets, making our method a promising candidate for facial biometric based security applications.

Simultaneous Hallucination and Recognition of Low-Resolution Faces Based on Singular Value Decomposition

In video surveillance, the captured face images are usually of low resolution (LR). Thus, a framework based on singular value decomposition (SVD) for performing both face hallucination and recognition simultaneously is proposed in this paper. Conventionally, LR face recognition is carried out by super-resolving the LR input face first, and then performing face recognition to identify the input face. By considering face hallucination and recognition simultaneously, the accuracy of both the hallucination and the recognition can be improved. In this paper, singular values are first proved to be effective for representing face images, and the singular values of a face image at different resolutions have approximately a linear relation. In our algorithm, each face image is represented using SVD. For each LR input face, the corresponding LR and high-resolution (HR) face-image pairs can then be selected from the face gallery. Based on these selected LR–HR pairs, the mapping functions for interpolating the two matrices in the SVD representation for the reconstruction of HR face images can be learned more accurately. Therefore, the final estimation of the high-frequency details of the HR face images will become more reliable and effective. The experimental results demonstrate that our proposed framework can achieve promising results for both face hallucination and recognition.

Face hallucination based on two-dimensional joint learning

In this paper, a face hallucination method based on two-dimensional joint learning is presented. Unlike the existing works on face super-resolution algorithms that first reshape the image or image patch into 1D vector, in our study the spatial construction of the high resolution (HR) and the low resolution (LR) face image are efficiently maintained in the reconstruction procedure. Enlightened by the 1D joint learning approach for image super-resolution, we propose a 2D joint learning algorithm to map the original 2D LR and HR image patch spaces onto a unified feature subspace. Subsequently, the neighbor-embedding (NE) based super-resolution algorithm can be conducted on the unified feature subspace to estimate the reconstruction weights. With these weights, the initial HR facial image can be generated. To refine further the initial HR estimate, the global reconstruction constraint is exploited to improve the quality of reconstruction result. Experiments on the face databases and real-world face images demonstrate the effectiveness of the proposed algorithm.

CDMMA: Coupled discriminant multi-manifold analysis for matching low-resolution face images

Face images captured by surveillance cameras usually have low-resolution (LR) in addition to uncontrolled poses and illumination conditions, all of which adversely affect the performance of face matching algorithms. In this paper, we develop a novel method to address the problem of matching a LR or poor quality face image to a gallery of high-resolution (HR) face images. In recent years, extensive efforts have been made on LR face recognition (FR) research. Previous research has focused on introducing a learning based super-resolution (LBSR) method before matching or transforming LR and HR faces into a unified feature space (UFS) for matching. To identify LR faces, we present a method called coupled discriminant multi-manifold analysis (CDMMA). In CDMMA, we first explore the neighborhood information as well as local geometric structure of the multi-manifold space spanned by the samples. And then, we explicitly learn two mappings to project LR and HR faces to a unified discriminative feature space (UDFS) through a supervised manner, where the discriminative information is maximized for classification. After that, the conventional classification method is applied in the CDMMA for final identification. Experimental results conducted on two standard face recognition databases demonstrate the superiority of the proposed CDMMA.

Kernel-Based Face Hallucination via Dual Regularization Priors

Recently, patch-based face hallucination methods have shown the ability for achieving high-quality face images. The high-resolution (HR) patches can be reconstructed by a linear combination of training patches, while the combination coefficients are learned according to the corresponding low-resolution (LR) patches. In order to reflect the local features, face images are usually divided into very small patches, e.g. $3 \times 3$ for LR case. Though we assume the linear relationship between training patches, it may fail to obtain suitable combination coefficients due to the low dimension of LR patches. In this letter, the kernel function is utilized for mapping the LR patches into a high-dimensional feature space. By taking into account the nonlinear structures, it is more effective to estimate the combination coefficients in the kernel feature space. Furthermore, a pixel-based model is also employed according to the characteristics of face images, which is useful to compensate the local textures. The final HR face images are obtained from a global optimization function by an iterative process. Experimental results show the advantage of the proposed approach in both reconstruction error and visual quality.

Robust Framework of Single-Frame Face Superresolution Across Head Pose, Facial Expression, and Illumination Variations

This paper presents a robust framework to solve the face hallucination problem across multiple factors, i.e., different expressions, head poses, and illuminations. It proposes a redundant transformation with diagonal loading for modeling the mappings among different new face factors, and a local reconstruction with geometry and position constraints for incorporating image details in the new factor spaces. Our proposed redundant and sparse strategies are discussed, and the experiments indicate that it is not necessary to adopt sparse representation in the proposed framework. The experimental results demonstrate that the proposed framework offers robustness when dealing with the inputs that have different expressions, head poses, and illuminations compared with the state-of-the-art methods, can generate high-resolution face images with better image qualities than the hierarchical tensor-based method, and improves the state of the art from single one output to multiple outputs with new factors.

High-Resolution Face Verification Using Pore-Scale Facial Features

Face recognition methods, which usually represent face images using holistic or local facial features, rely heavily on alignment. Their performances also suffer a severe degradation under variations in expressions or poses, especially when there is one gallery per subject only. With the easy access to high-resolution (HR) face images nowadays, some HR face databases have recently been developed. However, few studies have tackled the use of HR information for face recognition or verification. In this paper, we propose a pose-invariant face-verification method, which is robust to alignment errors, using the HR information based on pore-scale facial features. A new keypoint descriptor, namely, pore-Principal Component Analysis (PCA)-Scale Invariant Feature Transform (PPCASIFT)-adapted from PCA-SIFT-is devised for the extraction of a compact set of distinctive pore-scale facial features. Having matched the pore-scale features of two-face regions, an effective robust-fitting scheme is proposed for the face-verification task. Experiments show that, with one frontal-view gallery only per subject, our proposed method outperforms a number of standard verification methods, and can achieve excellent accuracy even the faces are under large variations in expression and pose.

Low-resolution degradation face recognition over long distance based on CCA

Canonical correlation analysis (CCA) is a kind of classical multivariate analysis method. Less canonical correlation variables are used to describe the relationship between two variables completely but easily. To get high face recognition rate under low-resolution degradation over a long distance solidly, in this work, CCA is used to extract the correlation between high-resolution face images and low-resolution ones and to find the transform pair between them. Therefore, face images of the same individual with variable resolutions can be matched accurately. This is the first method that uses CCA to do low-resolution degradation face recognition over long distances. We conduct the experiments on the Extended Yale B and ORL database, and the experimental results validate the efficacy of the proposed method.

Single image super-resolution method via refined local learning

In this paper, we propose a refined local learning scheme to reconstruct a high resolution (HR) face image from a low resolution (LR) observation. The contribution of this work is twofold. Firstly, multi-direction gradient features are extracted to search the nearest neighbors for each image patch, then the non-negative matrix factorization (NMF) is used to reduce the complexity in weight calculation, and the initial HR embedding is estimated from the training pairs by preserving local geometry. Secondly, a global reconstruction constraint and post-processing by non-local filtering is incorporated into super-resolution (SR) reconstruction process to reduce the image artifacts and further improve the image visual quality. Experimental results show that the proposed algorithm improves the SR performance both in subjective and objective assessments compared with several existing methods.

Position constraint based face image super-resolution by learning multiple local linear projections

In recent years, many super-resolution methods reveal that the mapping between low- and high-resolution images can be approximated by multiple local linear ones. Moreover, since face image patches located at the same position resemble each other, it is reasonable to assume they favor the same local linear mapping. Inspired by these phenomena, we propose a position constraint based face image super-resolution method which offline trains multiple local linear projections. Two goals are incorporated: First, a low-resolution patch can be linearly mapped to a high-resolution patch using the corresponding local linear projection. Second, the intrinsic sparse structure between low-resolution patches should be preserved by the reconstructed high-resolution ones. The final high-resolution face image is formed by integrating the reconstructed patches. Experimental results demonstrate that the proposed method can achieve face images of satisfactory quality and the online reconstruction stage is computationally fast. Besides, to some extent, the proposed method is insensitive to overlap size and the number of training images.

A CORRELATIVE TWO-STEP APPROACH TO HALLUCINATING FACES

Face hallucination is to synthesize high-resolution face image from the input low-resolution one. Although many two-step learning-based face hallucination approaches have been developed, they suffer from the expensive computational cost due to the separate calculation of the global and local models. To overcome this problem, we propose a correlative two-step learning-based face hallucination approach which bridges the gap between the global model and the local model. In the global phase, we build a global face hallucination framework by combining the steerable pyramid decomposition and the reconstruction. In the residue compensation phase, based on the combination weights and constituent samples obtained in the global phase, a residue face image is synthesized by the neighbor reconstruction algorithm to compensate the hallucinated global face image with subtle facial features. The ultimate hallucinated result is synthesized by adding the residue face image to the global face image. Compared with existing methods, in the global phase, our global face image is more similar to the original high-resolution face image. Furthermore, in the residue compensation phase, we use the combination weights and constituent samples obtained in the global phase to compute the residue face image, by which the computational efficiency can be greatly improved without compromising the quality of facial details. The experimental results and comparisons demonstrate that our approach can not only generate convincible high-resolution face images efficiently, but also has high computational efficiency. Furthermore, our proposed approach can be used to restore the damaged face images in image inpainting. The efficacy of our approach is validated by recovering the damaged face images with visually good results.

Improving cross-resolution face matching using ensemble-based co-transfer learning.

Face recognition algorithms are generally trained for matching high-resolution images and they perform well for similar resolution test data. However, the performance of such systems degrades when a low-resolution face image captured in unconstrained settings, such as videos from cameras in a surveillance scenario, are matched with high-resolution gallery images. The primary challenge, here, is to extract discriminating features from limited biometric content in low-resolution images and match it to information rich high-resolution face images. The problem of cross-resolution face matching is further alleviated when there is limited labeled positive data for training face recognition algorithms. In this paper, the problem of cross-resolution face matching is addressed where low-resolution images are matched with high-resolution gallery. A co-transfer learning framework is proposed, which is a cross-pollination of transfer learning and co-training paradigms and is applied for cross-resolution face matching. The transfer learning component transfers the knowledge that is learnt while matching high-resolution face images during training to match low-resolution probe images with high-resolution gallery during testing. On the other hand, co-training component facilitates this transfer of knowledge by assigning pseudolabels to unlabeled probe instances in the target domain. Amalgamation of these two paradigms in the proposed ensemble framework enhances the performance of cross-resolution face recognition. Experiments on multiple face databases show the efficacy of the proposed algorithm and compare with some existing algorithms and a commercial system. In addition, several high profile real-world cases have been used to demonstrate the usefulness of the proposed approach in addressing the tough challenges.