Non-frontal Face Images Research Articles

Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.

An illumination direction estimation algorithm of face images

Illumination is one of the important factors affecting the accuracy of face recognition. This paper proposes an illumination direction estimation algorithm of face images and it can be applied to the non-frontal face image. In order to eliminate the influence of face’s albedo, the algorithm use SfSNet to get the shading image. The algorithm divides the face region into eight regions by using five landmarks in face and estimate the illumination direction by analyzing the average brightness of each regions. Experiment on YaleB dataset shows that the algorithm can estimate the illumination direction correctly in most cases.

Face Hallucination With Finishing Touches.

Obtaining a high-quality frontal face image from a low-resolution (LR) non-frontal face image is primarily important for many facial analysis applications. However, mainstreams either focus on super-resolving near-frontal LR faces or frontalizing non-frontal high-resolution (HR) faces. It is desirable to perform both tasks seamlessly for daily-life unconstrained face images. In this paper, we present a novel Vivid Face Hallucination Generative Adversarial Network (VividGAN) for simultaneously super-resolving and frontalizing tiny non-frontal face images. VividGAN consists of coarse-level and fine-level Face Hallucination Networks (FHnet) and two discriminators, i.e., Coarse-D and Fine-D. The coarse-level FHnet generates a frontal coarse HR face and then the fine-level FHnet makes use of the facial component appearance prior, i.e., fine-grained facial components, to attain a frontal HR face image with authentic details. In the fine-level FHnet, we also design a facial component-aware module that adopts the facial geometry guidance as clues to accurately align and merge the frontal coarse HR face and prior information. Meanwhile, two-level discriminators are designed to capture both the global outline of a face image as well as detailed facial characteristics. The Coarse-D enforces the coarsely hallucinated faces to be upright and complete while the Fine-D focuses on the fine hallucinated ones for sharper details. Extensive experiments demonstrate that our VividGAN achieves photo-realistic frontal HR faces, reaching superior performance in downstream tasks, i.e., face recognition and expression classification, compared with other state-of-the-art methods.

Application of Ear Detection in Non-frontal Face Recognition

Face gesture recognition is a research hotspot in the field of machine vision and pattern recognition.In this paper, we use the Haar-Like feature-based Gentle Adaboost algorithm to detect the ear of non-frontal face. An ear sample library for ear detection of non-frontal face images was constructed. Through zoom detection window and determining the ROI area, it can successfully detect ear and work better than the basic Adaboost algorithm. The detection rate of the algorithm in the experiment reached 82%. It is suitable for most of the non-frontal faces we photographed.It still has a good detection effect for images with a small amount of hair or eyeglass frame interference.Based on the contour information of the ear, an approximate method for determining the lower end of the ear based on edge detection and a head and neck boundary determination algorithm based on the lower end of the human ear are proposed, which is more complete than the previous algorithm based on the determination of the concave points on both sides of the neck. The face area is segmented to make the face contour finally applied on the fitted ellipse more accurate.

Generating Images of Face Poses for Pose Varying Face Recognition

Deep learning has attracted several researchers in the field of computer vision due to its ability to perform face and object recognition tasks with high accuracy than the traditional shallow learning systems. The convolutional layers present in the deep learning systems help to successfully capture the distinctive features of the face. For biometric authentication, face recognition (FR) has been preferred due to its passive nature. Processing face images are accompanied by a series of complexities, like variation of pose, light, face expression, and make up. Although all aspects are important, the one that impacts the most face-related computer vision applications is pose. In face recognition, it has been long desired to have a method capable of bringing faces to the same pose, usually a frontal view, in order to ease recognition. Synthesizing different views of a face is still a great challenge, mostly because in non-frontal face images there are loss of information when one side of the face occludes the other. Most solutions for FR fail to perform well in cases involving extreme pose variations as in such scenarios, the convolutional layers of the deep models are unable to find discriminative parts of the face for extracting information. Most of the architectures proposed earlier deal with the scenarios where the face images used for training as well as testing the deep learning models are frontal and nearfrontal. On the contrary, here a limited number of face images at different poses is used to train the model, where a number of separate generator models learn to map a single face image at any arbitrary pose to specific poses and the discriminator performs the task of face recognition along with discriminating a synthetic face from a realworld sample. To this end, this paper proposes a representation learning by rotating the face. Here an encoder-decoder structure of the generator enables to learn a representation that is both generative and discriminative, which can be used for face image synthesis and pose-invariant face recognition. This representation is explicitly disentangled from other face variations such as pose, through the pose code provided to the decoder and pose estimation in the discriminator.

Local Comparative Decimal Pattern for Face Recognition

Rapid growth of social networks has provided an extraordinary medium to share a large volume of photographs online. This calls for designing efficient face recognition techniques that are applicable to images with low resolutions and arbitrary poses. This paper proposes a new pose invariant face recognition method for low resolution images using only a single training sample. A 3D model, reconstructed using Generic Elastic Model (3D GEM) from a frontal view training sample, is used to generate a set of nonfrontal gallery face images. The face region of the nonfrontal query sample is then extracted using the same landmark detection technique as in the 3D GEM algorithm. Afterwards, a novel texture representation technique called Local Comparative Decimal Pattern (LCDP) is proposed to extract features from each of the training and query samples. A set of experimental results on the ORL, Georgia Tech (GT), and LFW face databases demonstrates the efficiency of the proposed method compared to other state-of-the-art approaches.

Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces.

In popular TV programs (such as CSI), a very low-resolution face image of a person, who is not even looking at the camera in many cases, is digitally super-resolved to a degree that suddenly the person's identity is made visible and recognizable. Of course, we suspect that this is merely a cinematographic special effect and such a magical transformation of a single image is not technically possible. Or, is it? In this paper, we push the boundaries of super-resolving (hallucinating to be more accurate) a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very-low resolution (i.e., 16 × 16 pixels) out-of-plane rotated face images (including profile views) and aggressively super-resolve them (8×), regardless of their original poses and without using any 3D information. TANN is composed of two components: a transformative upsampling network which embodies encoding, spatial transformation and deconvolutional layers, and a discriminative network that enforces the generated high-resolution frontal faces to lie on the same manifold as real frontal face images. We evaluate our method on a large set of synthesized non-frontal face images to assess its reconstruction performance. Extensive experiments demonstrate that TANN generates both qualitatively and quantitatively superior results achieving over 4 dB improvement over the state-of-the-art.

Face Set Recognition

In this paper we present a Cluster Aggregation Network (CAN) for face set recognition. This network takes a set of face images, which could be either face videos or clusters with a different number of face images as its input, and then it is able to produce a compact and fixed-dimensional feature representation for the face set for the purpose of recognition. The whole network is made up of two modules, among which the first one is a face feature embedding module and the second one is the face feature aggregation module. The first module is a deep Convolutional Neural Network (CNN) which maps each of the face images to a fixed-dimensional vector. The second module is also a CNN which is trained to be able to automatically assess the quality of input face images and thus assign various weights to the images’ corresponding feature vectors. Then the one aggregated feature vector representing the input set is formed inside the convex hull formed by the input single face image features. Due to the mechanism that quality assessment is invariant to the order of one image in a set and the number of images in the set, the aggregation is invariant to these factors. Our CAN is trained with standard classification loss without any other supervision information and we found that our network is automatically attracted to high quality face images, while repelling low quality images, such as blurred, blocked, and non-frontal face images. We trained our networks with CASIA and YouTube Face datasets and the experiments on IJB-C video face recognition benchmark show that our method outperforms the current state-of-the-art feature aggregation methods and our challenging baseline aggregation method.

Representation Learning by Rotating Your Faces.

The large pose discrepancy between two face images is one of the fundamental challenges in automatic face recognition. Conventional approaches to pose-invariant face recognition either perform face frontalization on, or learn a pose-invariant representation from, a non-frontal face image. We argue that it is more desirable to perform both tasks jointly to allow them to leverage each other. To this end, this paper proposes a Disentangled Representation learning-Generative Adversarial Network (DR-GAN) with three distinct novelties. First, the encoder-decoder structure of the generator enables DR-GAN to learn a representation that is both generative and discriminative, which can be used for face image synthesis and pose-invariant face recognition. Second, this representation is explicitly disentangled from other face variations such as pose, through the pose code provided to the decoder and pose estimation in the discriminator. Third, DR-GAN can take one or multiple images as the input, and generate one unified identity representation along with an arbitrary number of synthetic face images. Extensive quantitative and qualitative evaluation on a number of controlled and in-the-wild databases demonstrate the superiority of DR-GAN over the state of the art in both learning representations and rotating large-pose face images.

Pose-robust feature learning for facial expression recognition

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods require the use of sophisticated normalization and initialization procedures. Thus head-pose invariant facial expression recognition continues to be an issue to traditional methods. In this paper, we propose a novel approach for pose-invariant FER based on pose-robust features which are learned by deep learning methods -- principal component analysis network (PCANet) and convolutional neural networks (CNN) (PRP-CNN). In the first stage, unlabeled frontal face images are used to learn features by PCANet. The features, in the second stage, are used as the target of CNN to learn a feature mapping between frontal faces and non-frontal faces. We then describe the non-frontal face images using the novel descriptions generated by the maps, and get unified descriptors for arbitrary face images. Finally, the pose-robust features are used to train a single classifier for FER instead of training multiple models for each specific pose. Our method, on the whole, does not require pose/ landmark annotation and can recognize facial expression in a wide range of orientations. Extensive experiments on two public databases show that our framework yields dramatic improvements in facial expression analysis.

Automatic Frontal Face Reconstruction Approach for Pose Invariant Face Recognition

Handling pose variations for face recognition system is a challenging task. The recognition rate is drastically decreasing with the images captured in uncontrolled environment having pose variations in yaw, pitch and roll angles. When the face image with frontal pose it is proved that the recognition system performs well. In this research an attempt is made to reconstruct frontal pose face images from non-frontal face images to improve the face recognition accuracy. By estimating the change in pose with respect to yaw, pitch and roll angles based on the landmark points best viewed side of the pose is identified. Using tilting, stretching and mirroring operation to the best viewed side, frontal pose is obtained. This approach is database independent, training free and no need to generate 3D model and not using any fitting approach, which is a complex task and handle any combination of roll, yaw, pitch angle up to ± 22.5 degrees only from the 2D landmark points. Experiments were conducted on FERET, HP, LFW, PUB-FIG data bases and the experimental result proves that our approach can handle the uncontrolled faces with arbitrary poses Experimental results on various controlled and uncontrolled poses proved the effectiveness of the proposed method.

Multi-view facial landmark detection by using a 3D shape model

An algorithm for accurate localization of facial landmarks coupled with a head pose estimation from a single monocular image is proposed. The algorithm is formulated as an optimization problem where the sum of individual landmark scoring functions is maximized with respect to the camera pose by fitting a parametric 3D shape model. The landmark scoring functions are trained by a structured output SVM classifier that takes a distance to the true landmark position into account when learning. The optimization criterion is non-convex and we propose a robust initialization scheme which employs a global method to detect a raw but reliable initial landmark position. Self-occlusions causing landmarks invisibility are handled explicitly by excluding the corresponding contributions from the data term. This allows the algorithm to operate correctly for a large range of viewing angles. Experiments on standard “in-the-wild” datasets demonstrate that the proposed algorithm outperforms several state-of-the-art landmark detectors especially for non-frontal face images. The algorithm achieves the average relative landmark localization error below 10% of the interocular distance in 98.3% of the 300W dataset test images.

Face recognition in the SWIR band when using single sensor multi-wavelength imaging systems

In this paper, we study the problem of Face Recognition (FR) when using Single Sensor Multi-Wavelength (SSMW) imaging systems that operate in the Short-Wave Infrared (SWIR) band. The contributions of our work are four fold: First, a SWIR database is collected when using our developed SSMW system under the following scenarios, i.e. Multi-Wavelength (MW) multi-pose images were captured when the camera was focused at either 1150, 1350 or 1550nm. Second, an automated quality-based score level fusion scheme is proposed for the classification of input MW images. Third, a weighted quality-based score level fusion scheme is proposed for the automated classification of full frontal (FF) vs. nonfrontal (NFF) face images. Fourth, a set of experiments is performed indicating that our proposed algorithms, for the classification of (i) multiwavelength images and (ii) FF vs. NFF face images, are beneficial when designing different steps of multi-spectral face recognition (FR) systems, including face detection, eye detection and face recognition. We also determined that when our SWIR-based system is focused at 1350nm, the identification performance increases compared to focusing the camera at any of the other SWIR wavelengths available. This outcome is particularly important for unconstrained FR scenarios, where imaging at 1550nm, at long distances and when operating at night time environments, is preferable over different SWIR wavelengths.

Identification of Eye Movements from Non-frontal Face Images for Eye-controlled Systems

The novel eye-based human-computer interaction (HCI) system aims to provide people, especially, disabled persons, a new way of communication with surroundings. It adopts a series of continual eye movements as input to perform simple control activities. Identification of eye movements is the crucial technology in these eye-based HCI systems. At present, researches on eye movement identification mainly focus on frontal face images. In fact, acquisition of non-frontal face images is more reasonable in real applications. In this paper, we discuss the identification process of eye movements from non-frontal face images. Firstly, the original head-shoulder images of 0°---±60° azimuths are sampled without any auxiliary light source. Secondly, the non-frontal face region is detected by using the Adaboost cascade classifiers. After that, we roughly extract eye windows by the integral projection function. Then, we propose a new method to calculate the x ? y coordinates of the pupil center point by searching the minimal intensity value in the eye windows. According to the trajectory of the pupil center points, different eye movements (eye moving left, right, up or down) are successfully identified. A set of experiments is presented.

Pose-Invariant Face Recognition Using Markov Random Fields

One of the key challenges for current face recognition techniques is how to handle pose variations between the probe and gallery face images. In this paper, we present a method for reconstructing the virtual frontal view from a given nonfrontal face image using Markov random fields (MRFs) and an efficient variant of the belief propagation algorithm. In the proposed approach, the input face image is divided into a grid of overlapping patches, and a globally optimal set of local warps is estimated to synthesize the patches at the frontal view. A set of possible warps for each patch is obtained by aligning it with images from a training database of frontal faces. The alignments are performed efficiently in the Fourier domain using an extension of the Lucas-Kanade algorithm that can handle illumination variations. The problem of finding the optimal warps is then formulated as a discrete labeling problem using an MRF. The reconstructed frontal face image can then be used with any face recognition technique. The two main advantages of our method are that it does not require manually selected facial landmarks or head pose estimation. In order to improve the performance of our pose normalization method in face recognition, we also present an algorithm for classifying whether a given face image is at a frontal or nonfrontal pose. Experimental results on different datasets are presented to demonstrate the effectiveness of the proposed approach.

Energy Normalization for Pose-Invariant Face Recognition Based on MRF Model Image Matching

A pose-invariant face recognition system based on an image matching method formulated on MRFs is presented. The method uses the energy of the established match between a pair of images as a measure of goodness-of-match. The method can tolerate moderate global spatial transformations between the gallery and the test images and alleviate the need for geometric preprocessing of facial images by encapsulating a registration step as part of the system. It requires no training on non-frontal face images. A number of innovations, such as a dynamic block size and block shape adaptation, as well as label pruning and error pre-whitening measures have been introduced to increase the effectiveness of the approach. The experimental evaluation of the method is performed on two publicly available databases. First, the method is tested on the rotation shots of the XM2VTS data set in a verification scenario. Next, the evaluation is conducted in an identification scenario on the CMU-PIE database. The method compares favorably with the existing 2D or 3D generative model-based methods on both databases in both identification and verification scenarios.

Pose invariant face recognition based on hybrid-global linear regression

The paper presents a simple but efficient novel H-eigenface (Hybrid-eigenface) method for pose invariant face recognition ranging from frontal to profile view. H-eigenfaces are entirely new basis for face image representation under different poses and are used for virtual frontal view synthesis. The proposed method is based on the fact that face samples of same person under different poses are similar in terms of the combination pattern of facial features. H-eigenfaces exploit this fact and thus two H-eigenfaces under different poses capture same features of the face. Thereby providing a compact view-based subspace, which can be further used to generate virtual frontal view from inputted non-frontal face image using least square projection technique. The use of proposed methodology on FERET and ORL face database shows an impressive improvement in recognition accuracy and a distinct reduction in online computation when compared to global linear regression method.

ROBUST ADAPTED PRINCIPAL COMPONENT ANALYSIS FOR FACE RECOGNITION

Recognizing faces with uncontrolled pose, illumination, and expression is a challenging task due to the fact that features insensitive to one variation may be highly sensitive to the other variations. Existing techniques dealing with just one of these variations are very often unable to cope with the other variations. The problem is even more difficult in applications where only one gallery image per person is available. In this paper, we describe a recognition method, Adapted Principal Component Analysis (APCA), that can simultaneously deal with large variations in both illumination and facial expression using only a single gallery image per person. We have now extended this method to handle head pose variations in two steps. The first step is to apply an Active Appearance Model (AAM) to the non-frontal face image to construct a synthesized frontal face image. The second is to use APCA for classification robust to lighting and pose. The proposed technique is evaluated on three public face databases — Asian Face, Yale Face, and FERET Database — with images under different lighting conditions, facial expressions, and head poses. Experimental results show that our method performs much better than other recognition methods including PCA, FLD, PRM and LTP. More specifically, we show that by using AAM for frontal face synthesis from high pose angle faces, the recognition rate of our APCA method increases by up to a factor of 4.

Locally Linear Regression for Pose-Invariant Face Recognition

The variation of facial appearance due to the viewpoint (/pose) degrades face recognition systems considerably, which is one of the bottlenecks in face recognition. One of the possible solutions is generating virtual frontal view from any given nonfrontal view to obtain a virtual gallery/probe face. Following this idea, this paper proposes a simple, but efficient, novel locally linear regression (LLR) method, which generates the virtual frontal view from a given nonfrontal face image. We first justify the basic assumption of the paper that there exists an approximate linear mapping between a nonfrontal face image and its frontal counterpart. Then, by formulating the estimation of the linear mapping as a prediction problem, we present the regression-based solution, i.e., globally linear regression. To improve the prediction accuracy in the case of coarse alignment, LLR is further proposed. In LLR, we first perform dense sampling in the nonfrontal face image to obtain many overlapped local patches. Then, the linear regression technique is applied to each small patch for the prediction of its virtual frontal patch. Through the combination of all these patches, the virtual frontal view is generated. The experimental results on the CMU PIE database show distinct advantage of the proposed method over Eigen light-field method.