Head Pose Research Articles

Collaborative 3D face alignment and head pose estimation with frontal face constraint based on RGB and sparse depth

A collaborative multi-task learning framework is proposed for integrating 3D face alignment and head pose estimation using RGB and sparse depth with a frontal face constraint. Most previous works involve using dense depth images to exploit spatial information, but the acquisition of high-resolution dense depth images is not always available due to the limitation of the image capture device. Besides, the existing multi-task learning methods have the problem of insufficient learning since they have not bridged the semantic gap between both tasks. In this letter, integrating 3D face alignment and head pose estimation into a collaborative multi-task learning framework is proposed, which is supervised by a frontal face constraint. In addition, sparse depth is incorporated into the network to provide additional facial geometrical information. A UV position map is generated to conduct face alignment, and a head pose vector is output to predict the head orientation. The performance on AFLW2000-3D dataset is evaluated, and the minimum quantization errors are achieved for both tasks when compared with state-of-the-art methods, demonstrating the superiority of the proposed method.

Automatic Brain Midline Surface Delineation on 3D CT Images With Intracranial Hemorrhage.

Brain midline delineation plays an important role in guiding intracranial hemorrhage surgery, which still remains a challenging task since hemorrhage shifts the normal brain configuration. Most previous studies detected brain midline on 2D plane and did not handle hemorrhage cases well. We propose a novel and efficient hemisphere-segmentation framework (HSF) for 3D brain midline surface delineation. Specifically, we formulate the brain midline delineation as a 3D hemisphere segmentation task, and employ an edge detector and a smooth regularization loss to generate the midline surface. We also introduce a distance-weighted map to keep the attention on the midline. Furthermore, we adopt rectification learning to handle various head poses. Finally, considering the complex situation of ventricle break-in for hemorrhages in bilateral intraventricular (B-IVH) cases, we identify those cases via a classification model and design a midline correction strategy to locally adjust the midline. To our best knowledge, it is the first study focusing on delineating the brain midline surface on 3D CT images of hemorrhage patients and handling the situation of ventricle break-in. Extensive validation on our large in-house datasets (519 patients) and the public CQ500 dataset (491 patients), demonstrates that our method outperforms state-of-the-art methods on brain midline delineation.

Head pose healthiness prediction using a novel image quality based stacked autoencoder

This paper introduces an approach aiming to determine head pose healthiness of computer users. The main contributions of this paper are: 1) Image Quality Assessment (IQA) based Stacked Autoencoder (referred to as IQASAE) which adjusts the value of learning rate based on the quality of images; 2) Head Pose Healthiness Prediction (HPHP) framework which leverages the proposed IQASAE algorithm in combination with image processing operations; 3) A set of features suitable for face analysis applications; 4) Ontology-driven semantic framework which enables further exploiting pose estimation results within applications in synergy with healthcare expert domain knowledge about pose healthiness. Our framework was evaluated on both offline (BIWI and AFLW) and online (our own, collected using Arduino) datasets. Furthermore, it was compared to several state-of-art methods, including Multi-Layer Perceptron (MLP), CART, Random Forest, Convolutional Neural Networks (CNN), Temporal Deep Learning Model (TDLM), hybrid CNN with Support Vector Machine (SVM), Quatnet and Trinet. According to the achieved experimental results, it reaches accuracy up to 79.63% outperforming all of them, except Quatnet and Trinet. However, the main advantages of IQASAE compared to state-of-art methods are: 1) it does not require selection of features, so the processing time is reduced, 2) utilizing angle between chin and mouth reduces training time for SAE, 3) leveraging vector-based feature set to create training data resulted in a significant improvement, especially in offline facial images.

FacialSCDnet: A deep learning approach for the estimation of subject-to-camera distance in facial photographs

Facial biometrics play an essential role in the fields of law enforcement and forensic sciences. When comparing facial traits for human identification in photographs or videos, the analysis must account for several factors that impair the application of common identification techniques, such as illumination, pose, or expression. In particular, facial attributes can drastically change depending on the distance between the subject and the camera at the time of the picture. This effect is known as perspective distortion, which can severely affect the outcome of the comparative analysis. Hence, knowing the subject-to-camera distance of the original scene where the photograph was taken can help determine the degree of distortion, improve the accuracy of computer-aided recognition tools, and increase the reliability of human identification and further analyses. In this paper, we propose a deep learning approach to estimate the subject-to-camera distance of facial photographs: FacialSCDnet. Furthermore, we introduce a novel evaluation metric designed to guide the learning process, based on changes in facial distortion at different distances. To validate our proposal, we collected a novel dataset of facial photographs taken at several distances using both synthetic and real data. Our approach is fully automatic and can provide a numerical distance estimation for up to six meters, beyond which changes in facial distortion are not significant. The proposed method achieves an accurate estimation, with an average error below 6 cm of subject-to-camera distance for facial photographs in any frontal or lateral head pose, robust to facial hair, glasses, and partial occlusion.

The Multimodal Driver Monitoring Database: A Naturalistic Corpus to Study Driver Attention

A smart vehicle should be able to monitor the actions and behaviors of the human driver to provide critical warnings or intervene when necessary. Recent advancements in deep learning and computer vision have shown great promise in monitoring human behaviors and activities. While these algorithms work well in a controlled environment, naturalistic driving conditions add new challenges such as illumination variations, occlusions and extreme head poses. A vast amount of in-domain data is required to train models that provide high performance in predicting driving related tasks to effectively monitor driver actions and behaviors. Toward building the required infrastructure, this paper presents the multimodal driver monitoring (MDM) dataset, which was collected with 59 subjects that were recorded performing various tasks. We use the Fi- Cap device that continuously tracks the head movement of the driver using fiducial markers, providing frame-based annotations to train head pose algorithms in naturalistic driving conditions. We ask the driver to look at predetermined gaze locations to obtain accurate correlation between the driver's facial image and visual attention. We also collect data when the driver performs common secondary activities such as navigation using a smart phone and operating the in-car infotainment system. All of the driver's activities are recorded with high definition RGB cameras and time-of-flight depth camera. We also record the controller area network-bus (CAN-Bus), extracting important information. These high quality recordings serve as the ideal resource to train various efficient algorithms for monitoring the driver, providing further advancements in the field of in-vehicle safety systems.

A deep transfer learning model for head pose estimation in rhesus macaques during cognitive tasks: Towards a nonrestraint noninvasive 3Rs approach

Head orientation is a measure of attention used in behavioral psychological research with non-human primates. It is used across a broad range of disciplines and settings, from the field to the laboratory. Field methods are time consuming with risk of coding bias and visibility issues with free-ranging animals. Laboratory methods may require restraint and use of invasive procedures. Automated systems to measure head orientation in unrestrained animals, that are robust to partial occlusion of the head, would improve coding efficiency and accuracy and provide 3Rs animal welfare benefits. We present a free-to-use deep transfer learning model for non-invasive head pose estimation in unrestrained Macaca mulatta taking part in cognitive experiments. Monkeys housed in social groups were filmed viewing two conspecific face stimuli presented on either side of a video camera. Video frames were manually annotated for three head positions relative to the video camera: ‘left’, ‘center’ and ‘right’. The dataset (total = 8135 images from 26 monkeys) was partitioned into training and testing datasets using a leave-k-out strategy, so that 70% of the images were used in training and 30% were used in testing. We used the VGG16, VGG19, InceptionV3 and Resnet50 as base models to train the proposed head pose classifier. We achieved model accuracy up to 93 %. The head pose estimation model presented here will be of use across contexts ranging from field-based playback experiments to assessment of welfare in zoo and clinical veterinary settings and refinement of neuroscience research practices. Model code with instructions is provided.

An Improved Tiered Head Pose Estimation Network with Self-Adjust Loss Function.

As an important task in computer vision, head pose estimation has been widely applied in both academia and industry. However, there remains two challenges in the field of head pose estimation: (1) even given the same task (e.g., tiredness detection), the existing algorithms usually consider the estimation of the three angles (i.e., roll, yaw, and pitch) as separate facets, which disregard their interplay as well as differences and thus share the same parameters for all layers; and (2) the discontinuity in angle estimation definitely reduces the accuracy. To solve these two problems, a THESL-Net (tiered head pose estimation with self-adjust loss network) model is proposed in this study. Specifically, first, an idea of stepped estimation using distinct network layers is proposed, gaining a greater freedom during angle estimation. Furthermore, the reasons for the discontinuity in angle estimation are revealed, including not only labeling the dataset with quaternions or Euler angles, but also the loss function that simply adds the classification and regression losses. Subsequently, a self-adjustment constraint on the loss function is applied, making the angle estimation more consistent. Finally, to examine the influence of different angle ranges on the proposed model, experiments are conducted on three popular public benchmark datasets, BIWI, AFLW2000, and UPNA, demonstrating that the proposed model outperforms the state-of-the-art approaches.

VGG16-random fourier hybrid model for masked face recognition.

With the recent COVID-19 pandemic, wearing masks has become a necessity in our daily lives. People are encouraged to wear masks to protect themselves from the outside world and thus from infection with COVID-19. The presence of masks raised serious concerns about the accuracy of existing facial recognition systems since most of the facial features are obscured by the mask. To address these challenges, a new method for masked face recognition is proposed that combines a cropping-based approach (upper half of the face) with an improved VGG-16 architecture. The finest features from the un-occluded facial region are extracted using a transfer learned VGG-16 model (Forehead and eyes). The optimal cropping ratio is investigated to give an enhanced feature representation for recognition. To avoid the overhead of bias, the obtained feature vector is mapped into a lower-dimensional feature representation using a Random Fourier Feature extraction module. Comprehensive experiments on the Georgia Tech Face Dataset, Head Pose Image Dataset, and Face Dataset by Robotics Lab show that the proposed approach outperforms other state-of-the-art approaches for masked face recognition.

Auritus: An Open-Source Optimization Toolkit for Training and Development of Human Movement Models and Filters Using Earables.

Smart ear-worn devices (called earables) are being equipped with various onboard sensors and algorithms, transforming earphones from simple audio transducers to multi-modal interfaces making rich inferences about human motion and vital signals. However, developing sensory applications using earables is currently quite cumbersome with several barriers in the way. First, time-series data from earable sensors incorporate information about physical phenomena in complex settings, requiring machine-learning (ML) models learned from large-scale labeled data. This is challenging in the context of earables because large-scale open-source datasets are missing. Secondly, the small size and compute constraints of earable devices make on-device integration of many existing algorithms for tasks such as human activity and head-pose estimation difficult. To address these challenges, we introduce Auritus an extendable and open-source optimization toolkit designed to enhance and replicate earable applications. Auritus serves two primary functions. Firstly, Auritus handles data collection, pre-processing, and labeling tasks for creating customized earable datasets using graphical tools. The system includes an open-source dataset with 2.43 million inertial samples related to head and full-body movements, consisting of 34 head poses and 9 activities from 45 volunteers. Secondly, Auritus provides a tightly-integrated hardware-in-the-loop (HIL) optimizer and TinyML interface to develop lightweight and real-time machine-learning (ML) models for activity detection and filters for head-pose tracking. To validate the utlity of Auritus, we showcase three sample applications, namely fall detection, spatial audio rendering, and augmented reality (AR) interfacing. Auritus recognizes activities with 91% leave 1-out test accuracy (98% test accuracy) using real-time models as small as 6-13 kB. Our models are 98-740× smaller and 3-6% more accurate over the state-of-the-art. We also estimate head pose with absolute errors as low as 5 degrees using 20kB filters, achieving up to 1.6× precision improvement over existing techniques. We make the entire system open-source so that researchers and developers can contribute to any layer of the system or rapidly prototype their applications using our dataset and algorithms.

Appearance-Based Gaze Estimation for ASD Diagnosis.

Biomarkers, such as magnetic resonance imaging (MRI) and electroencephalogram have been used to help diagnose autism spectrum disorder (ASD). However, the diagnosis needs the assist of specialized medical equipment in the hospital or laboratory. To diagnose ASD in a more effective and convenient way, in this article, we propose an appearance-based gaze estimation algorithm-AttentionGazeNet, to accurately estimate the subject's 3-D gaze from a raw video. The experimental results show its competitive performance on the MPIIGaze dataset and the improvement of 14.7% for static head pose and 46.7% for moving head pose on the EYEDIAP dataset compared with the state-of-the-art gaze estimation algorithms. After projecting the obtained gaze vector onto the screen coordinate, we apply accumulated histogram to taking into account both spatial and temporal information of estimated gaze-point and head-pose sequences. Finally, classification is conducted on our self-collected autistic children video dataset (ACVD), which contains 405 videos from 135 different ASD children, 135 typically developing (TD) children in a primary school, and 135 TD children in a kindergarten. The classification results on ACVD shows the effectiveness and efficiency of our proposed method, with the accuracy 94.8%, the sensitivity 91.1% and the specificity 96.7% for ASD.

RFAU: A Database for Facial Action Unit Analysis in Real Classrooms

Emotion analysis of students plays an important role in teaching effect evaluation. To develop robust algorithms for emotion analysis of students, a database from real classrooms is required. However, most existing databases were collected from adults and constructed in laboratory settings. In this article, we present a manually-annotated facial action unit database from juveniles in real classrooms. Our database has three main characteristics: (1) it provides numerous education-related action units data from primary and high schools, complementing the vacancy of the publicly available educational action unit databases; (2) it contains 256,220 manually-annotated facial images of 1,796 juveniles, frame-by-frame annotated with 12 action units and 6-level intensities for each action unit; (3) it covers many challenges in the wild, including various head poses, low facial resolution, illuminations, and occlusions, supplementing action unit databases in the wild for research. The baselines for action unit detection and action unit intensity estimation are provided for future references. Especially, we apply the weighted balance loss to solve imbalances <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">within</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">between</i> labels. Our database will be available to the research community: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://www.dlc.sjtu.edu.cn/rfau</uri> .

Deep Facial Expression Recognition: A Survey

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose, and identity bias. In this survey, we provide a comprehensive review of deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. We then describe the standard pipeline of a deep FER system with the related background knowledge and suggestions for applicable implementations for each stage. For the state-of-the-art in deep FER, we introduce existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences and discuss their advantages and limitations. Competitive performances and experimental comparisons on widely used benchmarks are also summarized. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

Flow-Based Unconstrained Lip to Speech Generation

Unconstrained lip-to-speech aims to generate corresponding speeches based on silent facial videos with no restriction to head pose or vocabulary. It is desirable to generate intelligible and natural speech with a fast speed in unconstrained settings. Currently, to handle the more complicated scenarios, most existing methods adopt the autoregressive architecture, which is optimized with the MSE loss. Although these methods have achieved promising performance, they are prone to bring issues including high inference latency and mel-spectrogram over-smoothness. To tackle these problems, we propose a novel flow-based non-autoregressive lip-to-speech model (GlowLTS) to break autoregressive constraints and achieve faster inference. Concretely, we adopt a flow-based decoder which is optimized by maximizing the likelihood of the training data and is capable of more natural and fast speech generation. Moreover, we devise a condition module to improve the intelligibility of generated speech. We demonstrate the superiority of our proposed method through objective and subjective evaluation on Lip2Wav-Chemistry-Lectures and Lip2Wav-Chess-Analysis datasets. Our demo video can be found at https://glowlts.github.io/.

High-Accuracy 3D Gaze Estimation with Efficient Recalibration for Head-Mounted Gaze Tracking Systems.

The problem of 3D gaze estimation can be viewed as inferring the visual axes from eye images. It remains a challenge especially for the head-mounted gaze tracker (HMGT) with a simple camera setup due to the complexity of the human visual system. Although the mainstream regression-based methods could establish the mapping relationship between eye image features and the gaze point to calculate the visual axes, it may lead to inadequate fitting performance and appreciable extrapolation errors. Moreover, regression-based methods suffer from a degraded user experience because of the increased burden in recalibration procedures when slippage occurs between HMGT and head. To address these issues, a high-accuracy 3D gaze estimation method along with an efficient recalibration approach is proposed with head pose tracking in this paper. The two key parameters, eyeball center and camera optical center, are estimated in head frame with geometry-based method, so that a mapping relationship between two direction features is proposed to calculate the direction of the visual axis. As the direction features are formulated with the accurately estimated parameters, the complexity of mapping relationship could be reduced and a better fitting performance can be achieved. To prevent the noticeable extrapolation errors, direction features with uniform angular intervals for fitting the mapping are retrieved over human’s field of view. Additionally, an efficient single-point recalibration method is proposed with an updated eyeball coordinate system, which reduces the burden of calibration procedures significantly. Our experiment results show that the calibration and recalibration methods could improve the gaze estimation accuracy by 35 percent (from a mean error of 2.00 degrees to 1.31 degrees) and 30 percent (from a mean error of 2.00 degrees to 1.41 degrees), respectively, compared with the state-of-the-art methods.

Wide Range Head Pose Estimation Using a Single RGB Camera for Intelligent Surveillance

Head pose estimation is one of the sensing systems needed for some intelligent surveillance, such as human behavior analysis, intelligent driver assistance, visual attention, and monitoring. These systems require accurate alignment and head movement direction prediction. The previous methods are greatly dependent on the facial landmarks and depth information. Usually, the head pose is measured by estimating several keypoints that require a correct head pose mapping to get accurate results. Moreover, facial landmarks have a detrimental effect on performance when the face is occluded or not adequately visualized. This paper proposes a method for head pose estimation of various facial conditions, such as occlusion and challenging viewpoints. We present a combination of coarse and fine feature maps classification to train a multi-loss deep Convolutional Neural Networks (CNN) to get precise Euler angles (yaw, pitch, roll) of the head position without keypoints and landmarks. Our proposed method uses more quantization units for angle classification to learn coarse and fine structure mapping for better spatial clustering features on an RGB image of a single camera. The experiments are performed on benchmark datasets and some head poses in real cases. The mean average error of prediction is 5.06°, 4.06°, and 2.96°, for the AFLW2000, AFLW, and BIWI datasets, which significantly improves the head pose estimation performance compared to the previous methods. Additionally, the proposed method outperforms previous approaches in computation time of 11 frames per second that is beneficial for real-life applications.

Gaze Preserving CycleGANs for Eyeglass Removal and Persistent Gaze Estimation

A driver’s gaze is critical for determining theirattention, state, situational awareness, and readiness to take over control from partially automated vehicles. Estimating the gaze direction is the most obvious way to gauge a driver’s state under ideal conditions when limited to using non-intrusive imaging sensors. Unfortunately, the vehicular environment introduces a variety of challenges that are usually unaccounted for - harsh illumination, nighttime conditions, and reflective eyeglasses. Relying on head pose alone under such conditions can prove to be unreliable and erroneous. In this study, we offer solutions to address these problems encountered in the real world. To solve issues with lighting, we demonstrate that using an infrared camera with suitable equalization and normalization suffices. To handle eyeglasses and their corresponding artifacts, we adopt image-to-image translation using generative adversarial networks to pre-process images prior to gaze estimation. Our proposed Gaze Preserving CycleGAN (GPCycleGAN) is trained to preserve the driver’s gaze while removing potential eyeglasses from face images. GPCycleGAN is based on the well-known CycleGAN approach - with the addition of a gaze classifier and a gaze consistency loss for additional supervision. Our approach exhibits improved performance, interpretability, robustness and superior qualitative results on challenging real-world datasets.

Attention-Mechanism-Based Real-Time Gaze Tracking in Natural Scenes With Residual Blocks

Gaze tracking is widely used in fatigue driving detection, eye disease diagnosis, mental illness diagnosis, website or advertising design, virtual reality, gaze-control devices, and human–computer interaction. However, the influence of light, specular reflection, and occlusion, and the change of head pose, especially the ever-changing human pose in natural scenes, have brought great challenges to the accurate gaze tracking. In this article, gaze tracking in natural scenes is studied, and a method based on the convolutional neural network with residual blocks is proposed, in which the attention mechanism is integrated into the network to improve the accuracy of gaze tracking. Furthermore, it is tested on the GazeFollow database, which contains six kinds of databases. The results show that the performance of the proposed method outperforms that of the other state-of-the-art methods in natural scenes. Moreover, the proposed method has better real-time performance and is more suitable for practical applications.

Vision-Based Gaze Estimation: A Review

Eye gaze is an important natural behavior in social interaction as it delivers complex exchanges between observer and observed, by building up the geometric constraints and relation of the exchanges. These interperson exchanges can be modeled based on gaze direction estimated using computer vision. Despite significant progresses in vision-based gaze estimation in last 10 years, it is still nontrivial since the accuracy of gaze estimation is significantly affected by such intrinsic factors as head pose variance, individual bias between optical axis and visual axis, eye blink, occlusion and image blur, degrade gaze features, lead to inaccurate gaze-involved human social interaction analysis. This article aims to review and discuss existing methods addressing above-mentioned problems, gaze involved applications and data sets against the state of the arts in vision-based gaze estimation. It also points out future research directions and challenges of gaze estimation in terms of metalearning, causal inference, disentangled representation, and social gaze behavior for unconstrained gaze estimation.

Age Estimation of Faces in Videos Using Head Pose Estimation and Convolutional Neural Networks.

Age estimation from human faces is an important yet challenging task in computer vision because of the large differences between physical age and apparent age. Due to the differences including races, genders, and other factors, the performance of a learning method for this task strongly depends on the training data. Although many inspiring works have focused on the age estimation of a single human face through deep learning, the existing methods still have lower performance when dealing with faces in videos because of the differences in head pose between frames, which can lead to greatly different results. In this paper, a combined system of age estimation and head pose estimation is proposed to improve the performance of age estimation from faces in videos. We use deep regression forests (DRFs) to estimate the age of facial images, while a multiloss convolutional neural network is also utilized to estimate the head pose. Accordingly, we estimate the age of faces only for head poses within a set degree threshold to enable value refinement. First, we divided the images in the Cross-Age Celebrity Dataset (CACD) and the Asian Face Age Dataset (AFAD) according to the estimated head pose degrees and generated separate age estimates for images with different poses. The experimental results showed that the accuracy of age estimation from frontal facial images was better than that for faces at different angles, thus demonstrating the effect of head pose on age estimation. Further experiments were conducted on several videos to estimate the age of the same person with his or her face at different angles, and the results show that our proposed combined system can provide more precise and reliable age estimates than a system without head pose estimation.

Real-Time Customer Satisfaction Analysis using Facial Expressions and Head Pose Estimation

Background/Purpose: Quantification of consumer interest is an interesting, innovative, and promising trend in marketing research. For example, an approach for a salesperson is to observe consumer behaviour during the shopping phase and then recall his interest. However, the salesperson needs unique skills because every person may interpret their behaviour in a different manner. The purpose of this research is to track client interest based on head pose positioning and facial expression recognition. Objective: We are going to develop a quantifiable system for measuring customer interest. This system recognizes the important facial expression and then processes current client photos and does not save them for later processing. Design/Methodology/Approach: The work describes a deep learning-based system for observing customer actions, focusing on interest identification. The suggested approach determines client attention by estimating head posture. The system monitors facial expressions and reports customer interest. The Viola and Jones algorithms are utilized to trim the facial image. Findings/Results: The proposed method identifies frontal face postures, then segments facial mechanisms that are critical for facial expression identification and creating an iconized face image. Finally, the obtained values of the resulting image are merged with the original one to analyze facial emotions. Conclusion: This method combines local part-based features with holistic facial information. The obtained results demonstrate the potential to use the proposed architecture as it is efficient and works in real-time. Paper Type: Conceptual Research.