Non-frontal Face Research Articles

Surveillance video face recognition with single sample per person based on 3D modeling and blurring

Video surveillance has attracted more and more interests in the last decade, video-based Face Recognition (FR) therefore became an important task. However, the surveillance videos include many vague non-frontal faces especially the view of faces looking down and up. As a result, most FR algorithms would perform worse when they were applied in surveillance videos. On the other hand, it was common at video monitoring field that only Single training Sample Per Person (SSPP) is available from their identification card. In order to effectively improve FR for both the SSPP problem and the low-quality problem, this paper proposed an approach to synthesis face images-based on 3D face modeling and blurring. In the proposed algorithm, firstly a 2D frontal face with high-resolution was used to build a 3D face model, then several virtual faces with different poses were synthesized from the 3D model, and finally some degraded face images were constructed from the original and the virtual faces through blurring process. At last multiple face images could be chosen from frontal, virtual and degraded faces to build a training set. Both SCface and LFW databases were employed to evaluate the proposed algorithm by using PCA, FLDA, scale invariant feature transform, compressive sensing and deep learning. The results on both datasets showed that the performance of these methods could be improved when virtual faces were generated to train the classifiers. Furthermore, in SCface database the average recognition rates increased up to 10%, 16.62%, 13.03%, 19.44% and 23.28% respectively for the above-mentioned methods when virtual view and blurred faces were taken to train their classifiers. Experimental results indicated that the proposed method for generating more train samples was effective and could be considered to be applied in intelligent video monitoring system.

Automatic Driver Head State Estimation in Challenging Naturalistic Driving Videos

Driver distraction represents a major safety problem in the United States. Naturalistic driving data, such as SHRP 2 Naturalistic Driving Study (NDS) data, provide a new window into driver behavior that promises a deeper understanding than was previously possible. Unfortunately, the current practice of manual coding is infeasible for large data sets such as SHRP 2 NDS, which contains millions of hours of video. Computer vision algorithms have the potential to automatically code SHRP 2 NDS videos. However, existing algorithms are brittle in the presence of challenges such as low video quality, underexposure and overexposure, driver occlusion, nonfrontal faces, and unpredictable and significant illumination changes, which are all substantially present in SHRP 2 NDS videos. This paper presents and evaluates algorithms developed to quantify high-level features pertinent to driver distraction and engagement in challenging videos like those in SHRP 2 NDS. Specifically, a novel three-stage video analysis system is presented for tracking head position and estimating head pose and eye and mouth states. The accuracy of the new head pose estimation module is competitive with the state of the art on publicly available data sets and produces good qualitative results on SHRP 2 NDS videos.

Pose-robust feature learning for facial expression recognition

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods require the use of sophisticated normalization and initialization procedures. Thus head-pose invariant facial expression recognition continues to be an issue to traditional methods. In this paper, we propose a novel approach for pose-invariant FER based on pose-robust features which are learned by deep learning methods -- principal component analysis network (PCANet) and convolutional neural networks (CNN) (PRP-CNN). In the first stage, unlabeled frontal face images are used to learn features by PCANet. The features, in the second stage, are used as the target of CNN to learn a feature mapping between frontal faces and non-frontal faces. We then describe the non-frontal face images using the novel descriptions generated by the maps, and get unified descriptors for arbitrary face images. Finally, the pose-robust features are used to train a single classifier for FER instead of training multiple models for each specific pose. Our method, on the whole, does not require pose/ landmark annotation and can recognize facial expression in a wide range of orientations. Extensive experiments on two public databases show that our framework yields dramatic improvements in facial expression analysis.

Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos

This paper proposes a facial expression recognition (FER) method in videos. The proposed method automatically selects the peak expression face from a video sequence using closeness of the face to the neutral expression. The severely non-frontal faces and poorly aligned faces are discarded in advance to eliminate their negative effects on the peak expression face selection and FER. To reduce the effect of the facial identity in the feature extraction, we compute difference information between the peak expression face and its intra class variation (ICV) face. An ICV face is generated by combining the training faces of an expression class and looks similar to the peak expression face in identity. Because the difference information is defined as the distances of locally pooled texture features between the two faces, the feature extraction is robust to face rotation and mis-alignment. Results show that the proposed method is practical with videos containing spontaneous facial expressions and pose variations.

Automatic Frontal Face Reconstruction Approach for Pose Invariant Face Recognition

Handling pose variations for face recognition system is a challenging task. The recognition rate is drastically decreasing with the images captured in uncontrolled environment having pose variations in yaw, pitch and roll angles. When the face image with frontal pose it is proved that the recognition system performs well. In this research an attempt is made to reconstruct frontal pose face images from non-frontal face images to improve the face recognition accuracy. By estimating the change in pose with respect to yaw, pitch and roll angles based on the landmark points best viewed side of the pose is identified. Using tilting, stretching and mirroring operation to the best viewed side, frontal pose is obtained. This approach is database independent, training free and no need to generate 3D model and not using any fitting approach, which is a complex task and handle any combination of roll, yaw, pitch angle up to ± 22.5 degrees only from the 2D landmark points. Experiments were conducted on FERET, HP, LFW, PUB-FIG data bases and the experimental result proves that our approach can handle the uncontrolled faces with arbitrary poses Experimental results on various controlled and uncontrolled poses proved the effectiveness of the proposed method.

Feature Extraction for Non-frontal Faces

Feature Extraction for Non-frontal Faces

Multi-view facial landmark detection by using a 3D shape model

An algorithm for accurate localization of facial landmarks coupled with a head pose estimation from a single monocular image is proposed. The algorithm is formulated as an optimization problem where the sum of individual landmark scoring functions is maximized with respect to the camera pose by fitting a parametric 3D shape model. The landmark scoring functions are trained by a structured output SVM classifier that takes a distance to the true landmark position into account when learning. The optimization criterion is non-convex and we propose a robust initialization scheme which employs a global method to detect a raw but reliable initial landmark position. Self-occlusions causing landmarks invisibility are handled explicitly by excluding the corresponding contributions from the data term. This allows the algorithm to operate correctly for a large range of viewing angles. Experiments on standard “in-the-wild” datasets demonstrate that the proposed algorithm outperforms several state-of-the-art landmark detectors especially for non-frontal face images. The algorithm achieves the average relative landmark localization error below 10% of the interocular distance in 98.3% of the 300W dataset test images.

Neural Network-Based Approach for Face Recognition Across Varying Pose

At present, frontal or even near frontal face recognition problem is no longer considered as a challenge. Recently, the shift has been to improve the recognition rate for the nonfrontal face. In this work, a neural network paradigm based on the radial basis function approach is proposed to tackle the challenge of recognizing faces in different poses. Exploiting the symmetrical properties of human face, our work takes the advantage of the existence of even half of the face. The strategy is to maximize the linearity relationship based on the local information of the face rather than on the global information. To establish the relationship, our proposed method employs discrete wavelet transform and multi-color uniform local binary pattern (ULBP) in order to obtain features for the local information. The local information will then be represented by a single vector known as the face feature vector. This face feature vector will be used to estimate the frontal face feature vector which will be used for matching with the actual vector. With such an approach, our proposed method relies on a database that contains only single frontal face images. The results shown in this paper demonstrate the robustness of our proposed method even at low-resolution conditions.

Robust Eye Center Localization through Face Alignment and Invariant Isocentric Patterns.

The localization of eye centers is a very useful cue for numerous applications like face recognition, facial expression recognition, and the early screening of neurological pathologies. Several methods relying on available light for accurate eye-center localization have been exploited. However, despite the considerable improvements that eye-center localization systems have undergone in recent years, only few of these developments deal with the challenges posed by the profile (non-frontal face). In this paper, we first use the explicit shape regression method to obtain the rough location of the eye centers. Because this method extracts global information from the human face, it is robust against any changes in the eye region. We exploit this robustness and utilize it as a constraint. To locate the eye centers accurately, we employ isophote curvature features, the accuracy of which has been demonstrated in a previous study. By applying these features, we obtain a series of eye-center locations which are candidates for the actual position of the eye-center. Among these locations, the estimated locations which minimize the reconstruction error between the two methods mentioned above are taken as the closest approximation for the eye centers locations. Therefore, we combine explicit shape regression and isophote curvature feature analysis to achieve robustness and accuracy, respectively. In practical experiments, we use BioID and FERET datasets to test our approach to obtaining an accurate eye-center location while retaining robustness against changes in scale and pose. In addition, we apply our method to non-frontal faces to test its robustness and accuracy, which are essential in gaze estimation but have seldom been mentioned in previous works. Through extensive experimentation, we show that the proposed method can achieve a significant improvement in accuracy and robustness over state-of-the-art techniques, with our method ranking second in terms of accuracy. According to our implementation on a PC with a Xeon 2.5Ghz CPU, the frame rate of the eye tracking process can achieve 38 Hz.

Cross-pose face recognition based on multiple virtual views and alignment error

Although studied for decades, effective face recognition remains difficult to accomplish on account of occlusions and pose and illumination variations. Pose variance is a particular challenge in face recognition. Effective local descriptors have been proposed for frontal face recognition. When these descriptors are directly applied to cross-pose face recognition, the performance significantly decreases. To improve the descriptor performance for cross-pose face recognition, we propose a face recognition algorithm based on multiple virtual views and alignment error. First, warps between poses are learned using the Lucas–Kanade algorithm. Based on these warps, multiple virtual profile views are generated from a single frontal face, which enables non-frontal faces to be matched using the scale-invariant feature transform (SIFT) algorithm. Furthermore, warps indicate the correspondence between patches of two faces. A two-phase alignment error is proposed to obtain accurate warps, which contain pose alignment and individual alignment. Correlations between patches are considered to calculate the alignment error of two faces. Finally, a hybrid similarity between two faces is calculated; it combines the number of matched keypoints from SIFT and the alignment error. Experimental results show that our proposed method achieves better recognition accuracy than existing algorithms, even when the pose difference angle was greater than 30°.

Face recognition in the SWIR band when using single sensor multi-wavelength imaging systems

In this paper, we study the problem of Face Recognition (FR) when using Single Sensor Multi-Wavelength (SSMW) imaging systems that operate in the Short-Wave Infrared (SWIR) band. The contributions of our work are four fold: First, a SWIR database is collected when using our developed SSMW system under the following scenarios, i.e. Multi-Wavelength (MW) multi-pose images were captured when the camera was focused at either 1150, 1350 or 1550nm. Second, an automated quality-based score level fusion scheme is proposed for the classification of input MW images. Third, a weighted quality-based score level fusion scheme is proposed for the automated classification of full frontal (FF) vs. nonfrontal (NFF) face images. Fourth, a set of experiments is performed indicating that our proposed algorithms, for the classification of (i) multiwavelength images and (ii) FF vs. NFF face images, are beneficial when designing different steps of multi-spectral face recognition (FR) systems, including face detection, eye detection and face recognition. We also determined that when our SWIR-based system is focused at 1350nm, the identification performance increases compared to focusing the camera at any of the other SWIR wavelengths available. This outcome is particularly important for unconstrained FR scenarios, where imaging at 1550nm, at long distances and when operating at night time environments, is preferable over different SWIR wavelengths.

Identification of Eye Movements from Non-frontal Face Images for Eye-controlled Systems

The novel eye-based human-computer interaction (HCI) system aims to provide people, especially, disabled persons, a new way of communication with surroundings. It adopts a series of continual eye movements as input to perform simple control activities. Identification of eye movements is the crucial technology in these eye-based HCI systems. At present, researches on eye movement identification mainly focus on frontal face images. In fact, acquisition of non-frontal face images is more reasonable in real applications. In this paper, we discuss the identification process of eye movements from non-frontal face images. Firstly, the original head-shoulder images of 0°---±60° azimuths are sampled without any auxiliary light source. Secondly, the non-frontal face region is detected by using the Adaboost cascade classifiers. After that, we roughly extract eye windows by the integral projection function. Then, we propose a new method to calculate the x ? y coordinates of the pupil center point by searching the minimal intensity value in the eye windows. According to the trajectory of the pupil center points, different eye movements (eye moving left, right, up or down) are successfully identified. A set of experiments is presented.

Pose robust face recognition based on kernel regression in Bayesian framework

To reduce the effect brought by variation of pose, a face recognition system is proposed, in which the correlation between two faces by probabilistic models is computed and the gallery is extended with profile faces generated by a regression-based method. Each face is divided into 3 × 3 regions, and by measuring the distance between the two corresponding regions the distributions of similarities are learned and Bayesian models are formed for each region. These probabilistic models are used to determine whether two faces belong to the same subject. Besides, the gallery is expanded with profile facial images. With this approach, the landmarks on non-frontal faces can be estimated from their frontal landmarks by the mappings learned offline via kernel rank reduced regression (KRRR). Then the profile images are achieved by piecewise affine warping from the corresponding frontal face. Experiments indicate the proposed method performed well, especially when the head rotation angle is large.

Using Rotated Asymmetric Haar-Like Features for Non-Frontal Face Detection

Using Rotated Asymmetric Haar-Like Features for Non-Frontal Face Detection

3D shape regression for real-time facial animation

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated.

Pose-Invariant Face Recognition Using Markov Random Fields

One of the key challenges for current face recognition techniques is how to handle pose variations between the probe and gallery face images. In this paper, we present a method for reconstructing the virtual frontal view from a given nonfrontal face image using Markov random fields (MRFs) and an efficient variant of the belief propagation algorithm. In the proposed approach, the input face image is divided into a grid of overlapping patches, and a globally optimal set of local warps is estimated to synthesize the patches at the frontal view. A set of possible warps for each patch is obtained by aligning it with images from a training database of frontal faces. The alignments are performed efficiently in the Fourier domain using an extension of the Lucas-Kanade algorithm that can handle illumination variations. The problem of finding the optimal warps is then formulated as a discrete labeling problem using an MRF. The reconstructed frontal face image can then be used with any face recognition technique. The two main advantages of our method are that it does not require manually selected facial landmarks or head pose estimation. In order to improve the performance of our pose normalization method in face recognition, we also present an algorithm for classifying whether a given face image is at a frontal or nonfrontal pose. Experimental results on different datasets are presented to demonstrate the effectiveness of the proposed approach.

VARIANT POSE FACE RECOGNITION USING DISCRETE WAVELET TRANSFORM AND LINEAR REGRESSION

Face recognition in constraint conditions is no longer a further challenge. However, even the best method is not able to cope with real world situations. In this paper, a robust method is proposed such that the performance of the face recognition system is still highly reliable even if the face undergoes large head rotation. Our proposed method considers local regions from half side of face rather than using the holistic face approach since in the former approach the "linearity" of features within the limited region is somewhat preserved regardless of the pose variation. Discrete wavelet transform is then utilized onto these patches in order to form face feature vectors. We train our recognizer using linear regression algorithm to interpret the relationship between a face vector for a specific pose and its corresponding frontal face feature vector. We demonstrate that our proposed method is able to recognize a non-frontal face with high accuracy even under low-resolution image by relying only on single frontal face in the database.

Combining Head Pose and Eye Location Information for Gaze Estimation

Head pose and eye location for gaze estimation have been separately studied in numerous works in the literature. Previous research shows that satisfactory accuracy in head pose and eye location estimation can be achieved in constrained settings. However, in the presence of nonfrontal faces, eye locators are not adequate to accurately locate the center of the eyes. On the other hand, head pose estimation techniques are able to deal with these conditions; hence, they may be suited to enhance the accuracy of eye localization. Therefore, in this paper, a hybrid scheme is proposed to combine head pose and eye location information to obtain enhanced gaze estimation. To this end, the transformation matrix obtained from the head pose is used to normalize the eye regions, and in turn, the transformation matrix generated by the found eye location is used to correct the pose estimation procedure. The scheme is designed to enhance the accuracy of eye location estimations, particularly in low-resolution videos, to extend the operative range of the eye locators, and to improve the accuracy of the head pose tracker. These enhanced estimations are then combined to obtain a novel visual gaze estimation system, which uses both eye location and head information to refine the gaze estimates. From the experimental results, it can be derived that the proposed unified scheme improves the accuracy of eye estimations by 16% to 23%. Furthermore, it considerably extends its operating range by more than 15° by overcoming the problems introduced by extreme head poses. Moreover, the accuracy of the head pose tracker is improved by 12% to 24%. Finally, the experimentation on the proposed combined gaze estimation system shows that it is accurate (with a mean error between 2° and 5°) and that it can be used in cases where classic approaches would fail without imposing restraints on the position of the head.

Energy Normalization for Pose-Invariant Face Recognition Based on MRF Model Image Matching

A pose-invariant face recognition system based on an image matching method formulated on MRFs is presented. The method uses the energy of the established match between a pair of images as a measure of goodness-of-match. The method can tolerate moderate global spatial transformations between the gallery and the test images and alleviate the need for geometric preprocessing of facial images by encapsulating a registration step as part of the system. It requires no training on non-frontal face images. A number of innovations, such as a dynamic block size and block shape adaptation, as well as label pruning and error pre-whitening measures have been introduced to increase the effectiveness of the approach. The experimental evaluation of the method is performed on two publicly available databases. First, the method is tested on the rotation shots of the XM2VTS data set in a verification scenario. Next, the evaluation is conducted in an identification scenario on the CMU-PIE database. The method compares favorably with the existing 2D or 3D generative model-based methods on both databases in both identification and verification scenarios.

Pose invariant face recognition based on hybrid-global linear regression

The paper presents a simple but efficient novel H-eigenface (Hybrid-eigenface) method for pose invariant face recognition ranging from frontal to profile view. H-eigenfaces are entirely new basis for face image representation under different poses and are used for virtual frontal view synthesis. The proposed method is based on the fact that face samples of same person under different poses are similar in terms of the combination pattern of facial features. H-eigenfaces exploit this fact and thus two H-eigenfaces under different poses capture same features of the face. Thereby providing a compact view-based subspace, which can be further used to generate virtual frontal view from inputted non-frontal face image using least square projection technique. The use of proposed methodology on FERET and ORL face database shows an impressive improvement in recognition accuracy and a distinct reduction in online computation when compared to global linear regression method.