Face Pose Research Articles

Dual Gaussian Modeling for Deep Face Embeddings

• framework using dual Gaussian distributions to model face embeddings with uncertainty. • Learn a sample-specific weight adaptively to integrate sub-Gaussian features into a compact representation. • Propose to minimize weighted Euclidean distance and the entropy of adaptive weight to constrain the two distributions. • Perform comprehensive experiments and analyses to illustrate the effect of our method for challenging face recognition. Most currently existing face recognition methods model face images as deterministic points in the latent space. However, it could encounter performance drop inevitably in a fully unconstrained scenario for the intrinsic noise of the images. In order to mitigate the detrimental impact of noisy data on model training, distribution estimation is introduced to face recognition, which models each face image as a Gaussian distribution and improves the robustness against noise effectively. But the uncertainty (variance) learned only relates to one attribute: the quality of the image. We propose dual Gaussian modeling (DGM) for deep face embeddings. For an input image, the network learns two Gaussian distributions simultaneously. The main Gaussian branch focuses on learning easy samples in the training dataset, while the other one mainly deals with faces with large pose. The uncertainty is not only correlated with image quality, but also with facial pose. During training, a sample-specific adaptive weight is learned to integrate the two sub-Gaussian features into a more compact discriminant embedding for classification . Besides, we introduce weighted Euclidean distance and minimize the entropy of the adaptive weight to regulate the relationship between the two distributions. Comprehensive experiments and analysis demonstrate that our method can boost the performance of face recognition under common or more unconstrained benchmarks, such as IJB-C.

Towards a Reliable and Rapid Automated Grading System in Facial Palsy Patients: Facial Palsy Surgery Meets Computer Science.

Background: Reliable, time- and cost-effective, and clinician-friendly diagnostic tools are cornerstones in facial palsy (FP) patient management. Different automated FP grading systems have been developed but revealed persisting downsides such as insufficient accuracy and cost-intensive hardware. We aimed to overcome these barriers and programmed an automated grading system for FP patients utilizing the House and Brackmann scale (HBS). Methods: Image datasets of 86 patients seen at the Department of Plastic, Hand, and Reconstructive Surgery at the University Hospital Regensburg, Germany, between June 2017 and May 2021, were used to train the neural network and evaluate its accuracy. Nine facial poses per patient were analyzed by the algorithm. Results: The algorithm showed an accuracy of 100%. Oversampling did not result in altered outcomes, while the direct form displayed superior accuracy levels when compared to the modular classification form (n = 86; 100% vs. 99%). The Early Fusion technique was linked to improved accuracy outcomes in comparison to the Late Fusion and sequential method (n = 86; 100% vs. 96% vs. 97%). Conclusions: Our automated FP grading system combines high-level accuracy with cost- and time-effectiveness. Our algorithm may accelerate the grading process in FP patients and facilitate the FP surgeon’s workflow.

Learning Unsupervised Face Normalization Through Frontal View Reconstruction

Face normalization from large pose is a challenging problem. Many Generative Adversarial Network (GAN) based models can infer frontal view of profile faces, while they require paired faces and pose label. Instead, we focus on frontal face synthesis with unpaired and unlabeled training data. We present a Frontal View Reconstruction based GAN (FVR-GAN) for large pose face normalization and recognition. The generator of FVR-GAN can be considered as a dual-input auto-encoder, where the identity encoder extracts identity features from an identity image and the template encoder extracts contour features from a frontal image. The decoder combines those two kinds of features and synthesizes a corresponding frontal face. To learn face normalization effectively, we incorporate the Frontal View Reconstruction (FVR) operation into training stage. The FVR operation includes self-reconstruction and frontalization mapping. A group of sub-discriminators which receive different facial parts are employed for discrimination. Considering that different face parts have different contributions to discrimination, we introduce a dynamic weighting mechanism to balance the output of sub-discriminators. FVR-GAN can recover high-quality frontal images under arbitrary poses. Experimental results on datasets of Multi-PIE, IJB-A, LFW and CFP demonstrate the efficacy of our model in terms of quality of synthesized images and recognition accuracy.

A semi-automatic toolbox for markerless effective semantic feature extraction

VisionTool is an open-source python toolbox for semantic features extraction, capable to provide accurate features detectors for different applications, including motion analysis, markerless pose estimation, face recognition and biological cell tracking. VisionTool leverages transfer-learning with a large variety of deep neural networks allowing high-accuracy features detection with few training data. The toolbox offers a friendly graphical user interface, efficiently guiding the user through the entire process of features extraction. To facilitate broad usage and scientific community contribution, the code and a user guide are available at https://github.com/Malga-Vision/VisionTool.git.

Natural Feature Recognition of Multi Pose Face Images based on Augmented Reality

Traditional face feature recognition methods have a few problems, such as low accuracy of image feature recognition and high time consumption of image feature recognition. This paper introduces augmented reality technology to realize natural feature recognition of multi pose face images. The multi pose features of face image are obtained by augmented reality technology, the color normalization of face image is realized by gamma algorithm, and the natural features of multi pose face image are extracted by three-dimensional registration technology. The feature error reconstruction of multi pose face image is realized according to sparse representation, the face image samples are projected into the optimal discriminant vector space by using two-dimensional linear discriminant analysis algorithm, and the natural feature recognition of multi pose face image is completed according to augmented reality technology. The experimental results show that the comprehensive accuracy of facial image expression recognition is as high as 95%, and the average error rate of natural feature recognition is only 3%; The time taken for feature recognition is only 3.5s, which shows that the method used in this study has higher efficiency of natural feature recognition of face image.

Needle in a Haystack: Spotting and recognising micro-expressions “in the wild”

Computational research on facial micro-expressions has long focused on videos captured under constrained laboratory conditions due to the challenging elicitation process and limited samples that are publicly available. Moreover, processing micro-expressions is extremely challenging under unconstrained scenarios. This paper introduces, for the first time, a completely automatic micro-expression “spot-and-recognize” framework that is performed on in-the-wild videos, such as in poker games and political interviews. The proposed method first spots the apex frame from a video by handling head movements and unconscious actions which are typically larger in motion intensity, with alignment employed to enforce a canonical face pose. Optical flow guided features play a central role in our method: they can robustly identify the location of the apex frame, and are used to learn a shallow neural network model for emotion classification. Experimental results demonstrate the feasibility of the proposed methodology, establishing good baselines for both spotting and recognition tasks – ASR of 0.33 and F1-score of 0.6758 respectively on the MEVIEW micro-expression database. In addition, we present comprehensive qualitative and quantitative analyses to further show the effectiveness of the proposed framework, with new suggestion for an appropriate evaluation protocol. In a nutshell, this paper provides a new benchmark for apex spotting and emotion recognition in an in-the-wild setting.

Real-Time Face Recognition and Detection Using Python

Abstract: While humans can recognize faces without much effort, facial recognition is a challenging pattern recognition problem in computing. Facial recognition systems attempt to identify a human face, which is three-dimensional and changes in appearance with lighting and facial expression, based on its two-dimensional image. To accomplish this computational task, facial recognition systems perform four steps. First face detection is used to segment the face from the image background. In the second step the segmented face image is aligned to account for face pose, image size and photographic properties, such as illumination and grayscale. The purpose of the alignment process is to enable the accurate localization of facial features in the third step, the facial nature extraction. Features such as eyes, nose and mouth are pinpointed and measured in the image to represent the face. The so established feature vector of the face is then, in the fourth step, matched against a database of faces. Keywords: face, detection, recognition, system, OpenCV

Driver Fatigue Detection Method Based on Human Pose Information Entropy

Driver fatigue detection (DFD) is an effective method to prevent traffic accidents. The existing research on DFD using facial features is an effective and noninvasive fatigue detection method. However, this approach is affected by facial occlusions (glasses, sunglasses, masks, etc.) and the large facial pose deformations in the extraction of effective fatigue features. In this paper, we introduce a novel DFD method using human pose information entropy. The method first estimates human pose from video sequences and then uses them as clues to extract multiple fatigue-related features which can reduce the influence of facial occlusion and head pose deformation. Information entropy and sliding window algorithm are applied to analyse and calculate sufficient consecutive video frames to obtain more robust and accurate fatigue-related values than by using a single frame. These information entropy values are combined resorting to the support vector machine (SVM) to recognize the driver fatigue state. Experimental results show that the method can achieve much higher accuracy and robustness, and the detection speed meets the requirements of real time.

COMSATS Face: A Dataset of Face Images with Pose Variations, Its Design, and Aspects

Due to the three-dimensional formation and flexibility, a human face may appear different in numerous events. Researchers are developing robust and efficient algorithms for face detection, face recognition, and face expression analysis, causing several difficulties due to face poses, illumination, face expression, head orientation, occlusion, hairstyle, etc. To determine the effectiveness of the algorithms, it needs to be tested using a specific benchmark of face images/databases. Face pose is an important factor that severely reduces the recognition ability. In this paper, two contributions are made: (i) a dataset of face images with multiple poses is introduced. The dataset includes 850 images of 50 individuals under 17 different poses (0°, 5°, 10°, 15°, 20°, 25°, 30°, 35°, 55°, -5°, -10°, -15°, -20°, -25°, -30°, -35°, -55°). These images were captured closed to real-world conditions in the time span of five months in COMSATS University, Abbottabad Campus. Face images included in this dataset can reveal the efficiency and robustness of future face detection and face recognition algorithms. (ii) A comparative analysis of three face recognition algorithms such as PAL, PCA, and LDA is presented based on the proposed face database.

Face Anonymity Based on Facial Pose Consistency

With the development of artificial intelligence, there are more and more applications related to face images. The recording of face information causes potential cyber security risks and personal privacy disclosure risks to the public. To solve this problem, we hope to protect face privacy through face anonymity. This paper designs a conditional autoencoder that uses the data preprocessing method of image inpainting. Based on the realistic generation ability of StyleGAN, our autoencoder model introduces facial pose information as conditional information. The input image only contains pre-processed face-removed images. Our method can generate high-resolution images and maintain the posture of the original face. It can be used for identity-independent computer vision tasks. Experiments further proves the effectiveness of our anonymization framework.

Driver’s Face Pose Estimation Using Fine-Grained Wi-Fi Signals for Next-Generation Internet of Vehicles

Driver’s behavior and gesture recognition are most significant in the emerging next-generation vehicular technology. Driver’s face may provide important cues about his/her attention and fatigue behavior. Therefore, driver’s face pose is one of the key indicators to be considered for automatic driver monitoring system in next-generation Internet of Vehicles (IoV) technology. Driver behavior monitoring is most significant in order to reduce road accidents. This paper aims to address the problem of driver’s attentiveness monitoring using face pose estimation in a nonintrusive manner. The proposed system is based on wireless sensing, leveraging channel state information (CSI) of WiFi signals. In this paper, we present a novel classification algorithm that is based on the combination of support vector machine (SVM) and K nearest neighbor (KNN) to enhance the classification accuracy. Experimental results demonstrate that the proposed device-free wireless implementation can localize a driver’s face very accurately with an average recognition rate of 91.8 % .

Face Recognition System for Control Access to Restrictive Domain

Recent progress in computer vision applied to facial analysis has led to state-of-the-art face detection and facial feature extraction models. A cautious implementation of these models into face recognition pipelines can enable achieving superior performances and popularized daily applications of face recognition in a variety of domains. However, modern face recognition system is a multi-steps process including face detection, feature extraction and classification model. Developing a high-performance face recognition application generalizing on local data set remains challenging. In this paper, we present Deep learning based face recognition system employing MTCNN for face detection and FaceNet for feature extraction. We compare KNN and SVM classification models trained on the facial features extracted from prepared labeled faces. Both models demonstrated almost 100% accuracy on static test faces. Moreover, as face pose get more pronounced, far above 30°, both SVM and KNN models demonstrate efficient recognition rate of 95.95% and 96.67% respectively. Real-time evaluation shows less than 1% deviation from the static performances with both classifiers on less 30° tilted images.

PCA-AE: Principal Component Analysis Autoencoder for Organising the Latent Space of Generative Networks

Autoencoders and generative models produce some of the most spectacular deep learning results to date. However, understanding and controlling the latent space of these models presents a considerable challenge. Drawing inspiration from principal component analysis and autoencoders, we propose the principal component analysis autoencoder (PCA-AE). This is a novel autoencoder whose latent space verifies two properties. Firstly, the dimensions are organised in decreasing importance with respect to the data at hand. Secondly, the components of the latent space are statistically independent. We achieve this by progressively increasing the latent space during training, and with a covariance loss applied to the latent codes. The resulting autoencoder produces a latent space which separates the intrinsic attributes of the data into different components of the latent space, in a completely unsupervised manner. We also describe an extension of our approach to the case of powerful, pre-trained GANs. We show results on both synthetic examples of shapes and on a state-of-the-art GAN. For example, we are able to separate the colour shade scale of hair, pose of faces and gender, without accessing any labels. We compare the PCA-AE with other state-of-the-art approaches, in particular with respect to the ability to disentangle attributes in the latent space. We hope that this approach will contribute to better understanding of the intrinsic latent spaces of powerful deep generative models.

Application of Optimized Local Binary Pattern Algorithm in Small Pose Face Recognition under Machine Vision

Application of Optimized Local Binary Pattern Algorithm in Small Pose Face Recognition under Machine Vision

Thermal stability measurement of terminal face relative pose for complex structure based on coordinate measuring machine

Abstract The coordinate measuring machine has high measurement accuracy, but it is only suitable for the precise measurement of the geometric size of the component in the normal temperature environment, and some complex structures need to assess the stability of the terminal face relative pose in the high and low temperature environment. In order to apply the high-precision measurement of the coordinate measuring machine to the thermal stability evaluation of the relative pose of the terminal face of complex structures, an extended temperature control method based on the coordinate measuring machine is proposed. The method realizes the loading control of the temperature environment of the specimen by the mobile temperature control module and the temperature control medium circulation module. The flexible connection mode is used between the temperature control module and the temperature control medium circulation module to reduce the influence of mechanical vibration on the measurement accuracy. The mixing fan and the temperature control medium dryer are installed in the pipeline to avoid frosting and dewing in the temperature control box and ensure the uniform distribution of the temperature field, so as to realize the accurate measurement of the relative pose of the terminal face of the complex structure under the high and low temperature environment by the coordinate measuring machine. The method is applied to the thermal stability test of the relative pose of the end face of a joint component, and the test results are good, which provides a new idea for the extended application of coordinate measuring machines.

AP-GAN: Improving Attribute Preservation in Video Face Swapping

Face swapping is a popular subject in face manipulation, which aims to replace the identity of the target face with that of the source face. Existing methods cannot well preserve facial attributes (e.g., pose, expression, skin color, illumination, make-up, occlusion, etc.) of the target face, causing noticeable temporal discontinuity and instability artifacts for video face swapping. In this paper, we propose a lightweight Generative Adversarial Networks based framework named AP-GAN, which can precisely control the attribute of the generated face to be consistent with that of the target face, achieving efficient and high-fidelity video face swapping. Specifically, we derive a U-Net based generator with ID blocks to translate identity and PE blocks to correct pose and expression. Besides, a PE-aware discriminator is designed to help supervise pose and expression of the synthetic face. Furthermore, we propose a discriminator based perceptual loss leveraging multi-scale features of the discriminator to preserve facial attributes like skin color, illumination, make-up and occlusion. AP-GAN is trained on Flickr-Faces-HQ, CelebA-HQ and VGGFace2 and evaluated on FaceForensics++. Extensive experiments and comparisons to the existing state-of-the-art face swapping methods demonstrate the efficacy of our framework. Comprehensive ablation studies are also carried out to isolate the validity of each proposed component and to contrast with other face manipulation approaches.

Unsupervised Facial Action Representation Learning by Temporal Prediction.

Due to the cumbersome and expensive data collection process, facial action unit (AU) datasets are generally much smaller in scale than those in other computer vision fields, resulting in overfitting AU detection models trained on insufficient AU images. Despite the recent progress in AU detection, deployment of these models has been impeded due to their limited generalization to unseen subjects and facial poses. In this paper, we propose to learn the discriminative facial AU representation in a self-supervised manner. Considering that facial AUs show temporal consistency and evolution in consecutive facial frames, we develop a self-supervised pseudo signal based on temporally predictive coding (TPC) to capture the temporal characteristics. To further learn the per-frame discriminativeness between the sibling facial frames, we incorporate the frame-wisely temporal contrastive learning into the self-supervised paradigm naturally. The proposed TPC can be trained without AU annotations, which facilitates us using a large number of unlabeled facial videos to learn the AU representations that are robust to undesired nuisances such as facial identities, poses. Contrary to previous AU detection works, our method does not require manually selecting key facial regions or explicitly modeling the AU relations manually. Experimental results show that TPC improves the AU detection precision on several popular AU benchmark datasets compared with other self-supervised AU detection methods.

Joint Face Image Restoration and Frontalization for Recognition

In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise. To address these challenges, previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. However, most of these methods are stage-wise, which is sub-optimal and deviates from the reality. In this paper, we address all these challenges jointly for unconstrained face recognition. We propose an Multi-Degradation Face Restoration (MDFR) model to restore frontalized high-quality faces from the given low-quality ones under arbitrary facial poses, with three distinct novelties. First, MDFR is a well-designed encoder-decoder architecture which extracts feature representation from an input face image with arbitrary low-quality factors and restores it to a high-quality counterpart. Second, MDFR introduces a pose residual learning strategy along with a 3D-based Pose Normalization Module (PNM), which can perceive the pose gap between the input initial pose and its real-frontal pose to guide the face frontalization. Finally, MDFR can generate frontalized high-quality face images by a single unified network, showing a strong capability of preserving face identity. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks demonstrate the superiority of MDFR over state-of-the-art methods on both face frontalization and face restoration.

2D facial landmark localization method for multi-view face synthesis image using a two-pathway generative adversarial network approach.

One of the key challenges in facial recognition is multi-view face synthesis from a single face image. The existing generative adversarial network (GAN) deep learning methods have been proven to be effective in performing facial recognition with a set of pre-processing, post-processing and feature representation techniques to bring a frontal view into the same position in-order to achieve high accuracy face identification. However, these methods still perform relatively weak in generating high quality frontal-face image samples under extreme face pose scenarios. The novel framework architecture of the two-pathway generative adversarial network (TP-GAN), has made commendable progress in the face synthesis model, making it possible to perceive global structure and local details in an unsupervised manner. More importantly, the TP-GAN solves the problems of photorealistic frontal view synthesis by relying on texture details of the landmark detection and synthesis functions, which limits its ability to achieve the desired performance in generating high-quality frontal face image samples under extreme pose. We propose, in this paper, a landmark feature-based method (LFM) for robust pose-invariant facial recognition, which aims to improve image resolution quality of the generated frontal faces under a variety of facial poses. We therefore augment the existing TP-GAN generative global pathway with a well-constructed 2D face landmark localization to cooperate with the local pathway structure in a landmark sharing manner to incorporate empirical face pose into the learning process, and improve the encoder-decoder global pathway structure for better representation of facial image features by establishing robust feature extractors that select meaningful features that ease the operational workflow toward achieving a balanced learning strategy, thus significantly improving the photorealistic face image resolution. We verify the effectiveness of our proposed method on both Multi-PIE and FEI datasets. The quantitative and qualitative experimental results show that our proposed method not only generates high quality perceptual images under extreme poses but also significantly improves upon the TP-GAN results.

Pose-invariant face recognition with multitask cascade networks

In this work, a face recognition method is proposed for face under pose variations using a multitask convolutional neural network (CNN). Furthermore, a pose estimation method followed by a face identification module is combined in a cascaded structure and used separately. In the presence of various facial expressions as well as low illuminations, datasets that include separated face poses can enhance the robustness of face recognition. The proposed method relies on a pose estimation module using a convolutional neural network model and trained on three categories of face image capture such as the left side, frontal, and right side. Second, three CNN models are used for face identification according to the estimated pose. The Left-CNN model, Front-CNN model, and Right-CNN model are used to identify the face for the left, frontal, and right pose of the face, respectively. Because face images may contain some useless information (e.g., background content), we propose a skin-based face segmentation method using structure–texture decomposition and the color-invariant descriptor. Experimental evaluation has been conducted using the proposed cascade-based face recognition system that consists of the aforementioned steps (i.e., pose estimation, face segmentation, and face identification) and is assessed on four different datasets and its superiority has been shown over related state-of-the-art techniques. Results reveal the contribution of the separate representation, skin segmentation, and pose estimation in the recognition robustness.