Frontal Pose Research Articles

Frontal person image generation based on arbitrary‐view human images

AbstractFrontal person images contain the richest detailed features of humans, which can effectively assist in behavioral recognition, virtual dress fitting and other applications. While many remarkable networks are devoted to the person image generation task, most of them need accurate target poses as the network inputs. However, the target pose annotation is difficult and time‐consuming. In this work, we proposed a first frontal person image generation network based on the proposed anchor pose set and the generative adversarial network. Specifically, our method first classify a rough frontal pose to the input human image based on the proposed anchor pose set, and regress all key points of the rough frontal pose to estimate an accurate frontal pose. Then, we consider the estimated frontal pose as the target pose, and construct a two‐stream generator based on the generative adversarial network to update the person's shape and appearance feature in a crossing way and generate a realistic frontal person image. Experiments on the challenging CMU Panoptic dataset show that our method can generate realistic frontal images from arbitrary‐view human images.

Toward Robust and Unconstrained Full Range of Rotation Head Pose Estimation.

Estimating the head pose of a person is a crucial problem for numerous applications that is yet mainly addressed as a subtask of frontal pose prediction. We present a novel method for unconstrained end-to-end head pose estimation to tackle the challenging task of full range of orientation head pose prediction. We address the issue of ambiguous rotation labels by introducing the rotation matrix formalism for our ground truth data and propose a continuous 6D rotation matrix representation for efficient and robust direct regression. This allows to efficiently learn full rotation appearance and to overcome the limitations of the current state-of-the-art. Together with new accumulated training data that provides full head pose rotation data and a geodesic loss approach for stable learning, we design an advanced model that is able to predict an extended range of head orientations. An extensive evaluation on public datasets demonstrates that our method significantly outperforms other state-of-the-art methods in an efficient and robust manner, while its advanced prediction range allows the expansion of the application area. We open-source our training and testing code along with our trained models: https://github.com/thohemp/6DRepNet360.

Academic Emotion Classification Using FER: A Systematic Review

Facial emotion expressions are among the most potent, natural, and powerful means of human communication. Due to the COVID-19 pandemic, educational institutions worldwide are forced to switch rapidly to remote and online learning. Students are currently in an emergency state and must adapt to various and readily accessible learning methods, such as mobile learning applications or an e-learning system. A systematic literature review (SLR) is conducted to extract and synthesize information such as the emotion classifier used in the facial expression recognition (FER) system, the dataset used, the preprocessing technique applied, the feature extraction approach used, and the strength and limitation of the previous studies. Based on the search criteria, 701 publications were initially retrieved from five different digital databases, of which 48 studies have been chosen as primary studies for further analysis. Based on the findings of this study, the deep learning approach is the most frequently adopted approach in classifying student emotions during online learning. FER-2013 is the most commonly used FER dataset in FER studies, while DAiSEE is the most used academic emotion dataset. Moreover, support vector machine (SVM) is the conventional learning emotion classifier that is widely used in the FER systems, while convolutional neural network (CNN) is the most frequently used deep learning classifier. Next, it was found that the number of real-time FER systems is less than that of non-real-time FER systems. Finally, the top-1 accuracy of 94.6% was achieved by the long-term recurrent convolutional network on the academic emotion dataset, and the limitation is that it has low illumination and a lack of frontal pose.

FP-Net: Frontal Pose Estimation Based on Single Human Body Image withArbitrary View

Accurate extraction of the frontal pose of the human body can assist behavior recognition, image generation, and virtually try-on work. However, side images and back images have problems, such as self-occlusion of the human body and invisible key points, which make the extraction of the frontal pose very difficult. Therefore, a frontal pose estimation network FP-Net (frontal pose network) based on single human body image at arbitrary view is designed and implemented. First, a multi-view human body image data set is produced to provide data support for model design. Second, in order to improve the accuracy of model prediction results, a regression module based on Anchor pose and a feature fusion module based on 3D pose are designed. Finally, FP-Net realizes the frontal pose extraction of human body images from arbitrary view. The PCK evaluation index is used for ablation experiments on the BJUT Taichi and CMU Panoptic datasets. The results show that proposed method effectively improves the accuracy of the frontal pose estimation of the human body.

Quantitative Analysis of Preoperative and Postoperative Photographs Posted on Social Media by Oculoplastic Surgeons.

To investigate and quantitatively analyze preoperative and postoperative photographs posted on social media by oculoplastic surgeons. Pre- and postoperative blepharoplasty photographs posted by oculoplastic surgeons on the social media platform Instagram were identified over a 4-month period. Photographs demonstrating a frontal pose of the periocular region were selected, and image analysis was performed to compare the preoperative and postoperative photographs. Quantitative image analysis was performed with a semiautomated script using ImageJ (v1.52k) software. Image analysis compared magnification, patient position, luminosity, relative color profiles, and edge detection/sharpness. Three oculoplastic surgeons graded these imaging metrics for the same photoset and the consensus grade was assigned to each category. Finally, the quantitative calculations were then compared with the consensus grades to develop receiver operating characteristic curves for further analysis. The study included 104 posts from 35 members of ASOPRS. Regarding patient positioning, 9 postings were 5%-10% offset from the patients' center, and 6 postings were more than 10% from the patients' center. The majority of photosets demonstrated less than or equal to 2.5% difference in magnification (69/104), with 7 postings demonstrating a greater than 5% difference, and 3 postings exhibiting greater than 10% difference in magnification. In the periocular region, 17.3% (18/104) of postings were found to have a greater than 10% difference in luminosity (>25.5 pixel-value). A more than 5% difference (>12.75 pixel-value) in pre- and postoperative edge-detection measurements was identified in 12 of 104 postings. In 22 of 104 photosets, the mean absolute deviation in color channel ratios was more than 0.01 and 15 of 104 photosets were found to have a mean absolute deviation more than 0.05. Numerous quantifiable photographic inconsistencies were identified in surgical photographs posted on social media by members of ASOPRS. Adoption of an automated image analysis tool that compares before-and-after surgical photographs could promote standardized oculoplastic surgery photography on social media.

YinYang-Net: Complementing Face and Body Information for Wild Gender Recognition

Soft biometrics inference in surveillance scenarios is a topic of interest for various applications, particularly in security-related areas. However, soft biometric analysis is not extensively reported in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">wild conditions</i> . In particular, previous works on gender recognition report their results in face datasets, with relatively good image quality and frontal poses. Given the uncertainty of the availability of the facial region in wild conditions, we consider that these methods are not adequate for surveillance settings. To overcome these limitations, we: 1) present frontal and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">wild</i> face versions of three well-known surveillance datasets; and 2) propose YinYang-Net (YY-Net), a model that effectively and dynamically complements facial and body information, which makes it suitable for gender recognition in wild conditions. The frontal and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">wild</i> face datasets derive from widely used Pedestrian Attribute Recognition (PAR) sets (PETA, PA-100K, and RAP), using a pose-based approach to filter the frontal samples and facial regions. This approach retrieves the facial region of images with varying image/subject conditions, where the state-of-the-art face detectors often fail. YY-Net combines facial and body information through a learnable fusion matrix and a channel-attention sub-network, focusing on the most influential body parts according to the specific image/subject features. We compare it with five PAR methods, consistently obtaining state-of-the-art results on gender recognition, and reducing the prediction errors by up to 24% in frontal samples. The announced PAR datasets versions and YY-Net serve as the basis for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">wild</i> soft biometrics classification and are available in here.

An Investigation on a Coptic Embroidered Panel from the 13th Century “Crucifixion with the Twelve Apostles” (Benaki Museum, Athens)

The “Crucifixion with the twelve Apostles”, a unique Coptic embroidered panel, was on display at the Benaki Museum (Athens, Greece). The representation of the “Crucifixion” with Christ in the center and six Apostles on either side, standing next to each other in frontal poses, is quite a rare one. This rare iconographic image of the twelve Apostles could be linked to the Ascension or the Pentecost. This unique representation of the Crucifixion with the twelve Apostles, which also involves the Ascension, is a one-of-a-kind compositional formula representing Christ’s Death as a triumph over Death, emphasizing, along with the other factors, its non-Chalcedonic origin. Moreover, the interpretation of an inscription, written in at least three languages embroidered in black silk thread, is a matter which confuses the issue even more. In the present study, we will attempt a comprehensive investigation, a detailed description, and interpretation of this rare iconography, based on written and iconographic evidence traced in the history of art heritage objects.

Is Gender “In-the-Wild” Inference Really a Solved Problem?

Soft biometrics analysis is seen as an important research topic, given its relevance to various applications. However, even though it is frequently seen as a solved task, it can still be very hard to perform in wild conditions, under varying image conditions, uncooperative poses, and occlusions. Considering the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gender</i> trait as our topic of study, we report an extensive analysis of the feasibility of its inference regarding image (resolution, luminosity, and blurriness) and subject-based features (face and body keypoints confidence). Using three state-of-the-art datasets (PETA, PA-100K, RAP) and five Person Attribute Recognition models, we correlate feature analysis with gender inference accuracy using the Shapley value, enabling us to perceive the importance of each image/subject-based feature. Furthermore, we analyze face-based gender inference and assess the pose effect on it. Our results suggest that: 1) image-based features are more influential for low-quality data; 2) an increase in image quality translates into higher subject-based feature importance; 3) face-based gender inference accuracy correlates with image quality increase; and 4) subjects’ frontal pose promotes an implicit attention towards the face. The reported results are seen as a basis for subsequent developments of inference approaches in uncontrolled outdoor environments, which typically correspond to visual surveillance conditions.

Real-Time Facial Expression Recognition System for Video Big Sensor Data Security Application

Facial video big sensor data (BSD) is the core data of wireless sensor network industry application and technology research. It plays an important role in many industries, such as urban safety management, unmanned driving, senseless attendance, and venue management. The construction of video big sensor data security application and intelligent algorithm model has become a hot and difficult topic in related fields based on facial expression recognition. This paper focused on the experimental analysis of Cohn–Kanade dataset plus (CK+) dataset with frontal pose and great clarity. Firstly, face alignment and the selection of peak image were utilized to preprocess the expression sequence. Then, the output vector from convolution network 1 and β-VAE were connected proportionally and input to support vector machine (SVM) classifier to complete facial expression recognition. The testing accuracy of the proposed model in CK + dataset can reach 99.615%. The number of expression sequences involved in training was 2417, and the number of expression sequences in testing was 519.

Detailed Feature Guided Generative Adversarial Pose Reconstruction Network

Face frontalization is a critical and difficult task on face pose reconstruction. Previous researches use simple posture information as guidance, such as pose coding and facial landmarks. To explore the guidance effect of profile faces, we propose detailed features that provide much detailed information. In this paper, a D etailed Feature G uided Generative Adversarial P ose R econstruction Network (DGPR) is proposed. Firstly, frontal pose coding and profile detailed features are fed into DGPR to generate detailed features of front face. Then, the second generator combines frontal detailed features and profile face to reconstruct front face. Besides, we propose a conditional enhancement loss to strengthen the guiding role of detailed features, and a smoothing loss to reduce edge sharpness in generated faces. Experimental results show that our method generates photorealistic front faces and outperforms state-of-the-art methods on M2FPA and CAS-PEAL. Specifically, DGPR improves the face recognition accuracy under pose angles of ±60°, ±75°, ±90° by 2%, 1%, and 6% respectively over the state-of-the-art methods on M2FPA, achieves the average rank-1 recognition rate to 99.95% and improves it by 0.05% on CAS-PEAL. These results demonstrate the effects of detailed features and corresponding modules.

Dual-View Normalization for Face Recognition

Face normalization refers to a family of approaches that rotate a non-frontal face to the frontal pose for better handling of face recognition. While a great majority of face normalization methods focus on frontal pose only, we proposed a framework for dual-view normalization that generates a frontal pose and an additional yaw-45° pose to an input face of an arbitrary pose. The proposed Dual-View Normalization (DVN) framework is designed to learn the transformation from a source set to two normal sets. The source set contains faces collected in the wild and covers a wide scope of variables. One normal set contains face images taken under controlled conditions and all faces are in frontal pose and balanced in illumination. The other normal set contains faces also taken under controlled conditions and balanced in illumination, but in 45° pose. The DVN framework is composed of one face encoder, two layers of generators, and two sets of discriminators. The encoder is made of a state-of-the-art face recognition network, which is not updated during training, and it acts as a facial feature extractor. The Layer-1 generators are trained on both the source and normal sets, aiming at learning the transformation from the source set to both normal sets. The trained generators can transform an arbitrary face into a pair of normalized faces, one in frontal pose and the other in 45° pose. The Layer-2 generators are trained to enhance the identity preservation of the faces made by the Layer-1 generators by minimizing the cross-pose identity loss. The discriminators are trained to ensure the photo-realistic quality of the dual-view normalized face images generated by the generators. The loss functions employed in the generators and the discriminators are designed to achieve satisfactory dual-view normalization outcomes and identity preservation. We verify the DVN framework on benchmark databases and compare with other state-of-the-art approaches for tackling face recognition.

Geometry Structure Preserving Based GAN for Multi-Pose Face Frontalization and Recognition

Face frontalization is the process of converting a face image under arbitrary pose to an image with frontal pose. Benefited from significant improvement of generative adversarial networks (GAN), generative models can use face frontalization to overcome the problem of model degradation owing to the variation of head pose in face recognition. Existing GAN based models can generate a synthesis face image with the same identity as the input, while those models are hard to capture the geometry structure or facial patterns via pixel-wise constraint, e.g. face contour. In this paper, we propose a Geometry Structure Preserving based GAN, i.e. GSP-GAN, for multi-pose face frontalization and recognition. The generator of our model takes the form of a typical auto-encoder, where the encoder extracts identity feature and the decoder synthesizes the corresponding frontal face image. In this process, the perception loss constrains the generator to synthesize a face image with the same identity as the input image. Meanwhile, we adopt real frontal face images as extra input data during training process, where a L1 norm loss is utilized to construct a pixel-wise mapping from arbitrary pose image to frontal image. More importantly, for discriminator of our model, we use the self-attention block to preserve the geometry structure of a face. The discriminator consists of a series of parallel sub-discriminators that carry the global and local attention information. Compared with the state-of-the-art models on datasets of Multi-PIE, LFW and CFP, the proposed GSP-GAN can generate high-quality frontal images under arbitrary pose, and get satisfactory recognition performance.

Adversarial Cross-Spectral Face Completion for NIR-VIS Face Recognition

Near infrared-visible (NIR-VIS) heterogeneous face recognition refers to the process of matching NIR to VIS face images. Current heterogeneous methods try to extend VIS face recognition methods to the NIR spectrum by synthesizing VIS images from NIR images. However, due to the self-occlusion and sensing gap, NIR face images lose some visible lighting contents so that they are always incomplete compared to VIS face images. This paper models high-resolution heterogeneous face synthesis as a complementary combination of two components: a texture inpainting component and a pose correction component. The inpainting component synthesizes and inpaints VIS image textures from NIR image textures. The correction component maps any pose in NIR images to a frontal pose in VIS images, resulting in paired NIR and VIS textures. A warping procedure is developed to integrate the two components into an end-to-end deep network. A fine-grained discriminator and a wavelet-based discriminator are designed to improve visual quality. A novel 3D-based pose correction loss, two adversarial losses, and a pixel loss are imposed to ensure synthesis results. We demonstrate that by attaching the correction component, we can simplify heterogeneous face synthesis from one-to-many unpaired image translation to one-to-one paired image translation, and minimize the spectral and pose discrepancy during heterogeneous recognition. Extensive experimental results show that our network not only generates high-resolution VIS face images but also facilitates the accuracy improvement of heterogeneous face recognition.

Automatic pose normalization for open-set single-sample face recognition in video surveillance

Face images acquired by video surveillance cameras usually involve large pose variations which significantly degrade the performance of face recognition systems. Existing techniques address the pose variation problem by normalizing the arbitrary poses to the desired pose prior to recognition. However, these methods may require 2D or 3D model fitting and manual facial landmarks annotation. In this work, we present an automatic pose normalization technique that is free from model fitting and manual intervention. Our method utilizes an automatic facial landmark detection algorithm and thin-plate splines warping method to normalize pose-varied face images to a canonical frontal pose. Detecting facial landmarks automatically in face images provides 2D surface points that are used by thin-plate splines warping to geometrically transform face images to the desired pose. Experimental results carried out on the FERET database have shown that the proposed method achieved a comparable or higher performance compared to the state-of-the-art pose normalization approaches under constrained conditions. Moreover, extended experiments on the ChokePoint database have shown that our method substantially improved the performance of our open-set single-sample face recognition approach in the surveillance environment.

Recognizing Profile Faces by Imagining Frontal View

Extreme pose variation is one of the key obstacles to accurate face recognition in practice. Compared with current techniques for pose-invariant face recognition, which either expect pose invariance from hand-crafted features or data-driven deep learning solutions, or first normalize profile face images to frontal pose before feature extraction, we argue that it is more desirable to perform both tasks jointly to allow them to benefit from each other. To this end, we propose a Pose-Invariant Model (PIM) for face recognition in the wild, with three distinct novelties. First, PIM is a novel and unified deep architecture, containing a Face Frontalization sub-Net (FFN) and a Discriminative Learning sub-Net (DLN), which are jointly learned from end to end. Second, FFN is a well-designed dual-path Generative Adversarial Network which simultaneously perceives global structures and local details, incorporating an unsupervised cross-domain adversarial training and a meta-learning (“learning to learn”) strategy using siamese discriminator with dynamic convolution for high-fidelity and identity-preserving frontal view synthesis. Third, DLN is a generic Convolutional Neural Network (CNN) for face recognition with our enforced cross-entropy optimization strategy for learning discriminative yet generalized feature representations with large intra-class affinity and inter-class separability. Qualitative and quantitative experiments on both controlled and in-the-wild benchmark datasets demonstrate the superiority of the proposed model over the state-of-the-arts.

Feedback cascade regression model for face alignment

Face alignment has made great progress in recent years and the cascade regression framework is one of the main contributors. However, the performance of this framework is unsatisfactory on heavily occluded faces or those far from the frontal pose. This is because regression is sensitive to hidden landmarks and unified initialisation can often lead to the method falling into local minima. The authors propose a new pipeline of salient‐to‐inner‐to‐all to progressively compute the locations of landmarks. Additionally, a feedback process is utilised to improve the robustness of regression. They bring out a pose‐invariant shape retrieval method to generate the discriminative initialisation. Experiments are performed on two benchmarks, and the experimental results demonstrate that the proposed method has a considerable improvement on the cascade regression model, and achieves favourable results compared with the state‐of‐the‐art deep learning‐based methods.

Efficient Iris and Eyelids Detection from Facial Sketch Images

In this paper, we propose a simple yet effective technique for an automatic iris and eyelids detection method for facial sketch images. Our system uses Circular Hough Transformation (CHT) algorithm for iris localization process and a low level grayscale analysis for eyelids contour segmentation procedure. We limit the input face for the system to facial sketch photos with frontal pose, illumination invariant, neutral expression and without occlusions. CUHK and IIIT-D sketch databases are used to acquire the experimental results. As to validate the proposed algorithm, experiments on ground truth for iris and eyelids segmentation, which are prepared at our lab, is conducted. The iris segmentation from the proposed method gives the best accuracy of 92.93 and 86.71 based on F-measure evaluation for IIIT-D and CUHK, respectively. For eyelids segmentation, on the other hand, the proposed algorithm achieves an average of 4 standard deviation which indicates the closeness of proposed method to ground truth.

Two-Dimensional Canonical Correlation Analysis of the Logically Concatenated Cross Binary Pattern for Cross Pose Face Recognition

Face pose recognition is one of the challenging areas in computer vision. Cross-pose change causes the change in the information of face appearance. The maximization of intrasubject correlation helps to widen the intersubject differences which helps further in achieving pose invariance. In this paper, for cross pose recognition, the authors propose to maximize the cross pose correlation by using the logically concatenated cross binary pattern (LC-CBP) descriptor and two dimensional canonical correlation analysis (2DCCA). The LC-CBP descriptor extracts the local texture details of face images with low computation complexity and the 2DCCA explicitly searches for the maximization of the correlated features to retain most informative content. Joint feature consideration via 2DCCA helps in setting up a better correspondence between a discrete set of nonfrontal pose and the frontal pose of the same subject. Experimental results demonstrate the two dimensional canonical correlation LC-CBP descriptor along with intensity values improve the correlation.

Neuro-fuzzy model for estimating race and gender from geometric distances of human face across pose

Classifying human face based on race and gender is a vital process in face recognition. It contributes to an index database and eases 3D synthesis of the human face. Identifying race and gender based on intrinsic factor is problematic, which is more fitting to utilizing nonlinear model for estimating process. In this paper, we aim to estimate race and gender in varied head pose. For this purpose, we collect dataset from PICS and CAS-PEAL databases, detect the landmarks and rotate them to the frontal pose. After geometric distances are calculated, all of distance values will be normalized. Implementation is carried out by using Neural Network Model and Fuzzy Logic Model. These models are combined by using Adaptive Neuro-Fuzzy Model. The experimental results showed that the optimization of address fuzzy membership. Model gives a better assessment rate and found that estimating race contributing to a more accurate gender assessment.

Learning Pose-Aware Models for Pose-Invariant Face Recognition in the Wild.

We propose a method designed to push the frontiers of unconstrained face recognition in the wild with an emphasis on extreme out-of-plane pose variations. Existing methods either expect a single model to learn pose invariance by training on massive amounts of data or else normalize images by aligning faces to a single frontal pose. Contrary to these, our method is designed to explicitly tackle pose variations. Our proposed Pose-Aware Models (PAM) process a face image using several pose-specific, deep convolutional neural networks (CNN). 3D rendering is used to synthesize multiple face poses from input images to both train these models and to provide additional robustness to pose variations at test time. Our paper presents an extensive analysis of the IARPA Janus Benchmark A (IJB-A), evaluating the effects that landmark detection accuracy, CNN layer selection, and pose model selection all have on the performance of the recognition pipeline. It further provides comparative evaluations on IJB-A and the PIPA dataset. These tests show that our approach outperforms existing methods, even surprisingly matching the accuracy of methods that were specifically fine-tuned to the target dataset. Parts of this work previously appeared in[1] and[2].