Facial Action Units Research Articles

Can a building read your mind? Results from a small trial in facial action unit detection

In the last few decades, the energy consumption of individual buildings has been steadily improving. As a result, research efforts are shifting towards acquiring a deeper understanding of occupant comfort, health, and well-being in the built environment. However, existing techniques used to measure and predict the comfort of occupants have seen little change since Fanger. New research attempts are hence focusing on methods to gather more data, more frequently, and less intrusively. A little explored source of data is the one gathered from real-time videos of occupants, the so-called facial action units (FAU), which is the focus of this paper. These are the facial movements and positions that constitute the basic elements of emotions. Using software developed in the realm of affective computing, seven building occupants were monitored for a period of 2 weeks, whilst also completing surveys that gathered information about the office environment, and their work and personal life. Results found that participants that were happy with their office space showed significantly higher average values of the Cheek Raiser (AU06) and Lid Tightener (AU07) facial action units. These findings show the potential of using FAUs to assist in the control and design of buildings in a human-centric manner.

D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection.

Facial action units (AUs) relate to specific local facial regions. Recent efforts in automated AU detection have focused on learning the facial patch representations to detect specific AUs. These efforts have encountered three hurdles. First, they implicitly assume that facial patches are robust to head rotation; yet non-frontal rotation is common. Second, mappings between AUs and patches are defined a priori, which ignores co-occurrences among AUs. And third, the dynamics of AUs are either ignored or modeled sequentially rather than simultaneously as in human perception. Inspired by recent advances in human perception, we propose a dynamic patch-attentive deep network, called D-PAttNet, for AU detection that (i) controls for 3D head and face rotation, (ii) learns mappings of patches to AUs, and (iii) models spatiotemporal dynamics. D-PAttNet approach significantly improves upon existing state of the art.

Listen to Your Face: Inferring Facial Action Units from Audio Channel

Extensive efforts have been devoted to recognizing facial action units (AUs). However, it is still challenging to recognize AUs from spontaneous facial displays especially when they are accompanied by speech. Different from all prior work that utilized visual observations for facial AU recognition, this paper presents a novel approach that recognizes speech-related AUs exclusively from audio signals based on the fact that facial activities are highly correlated with voice during speech. Specifically, dynamic and physiological relationships between AUs and phonemes are modeled through a continuous time Bayesian network (CTBN); then AU recognition is performed by probabilistic inference via the CTBN model. A pilot audiovisual AU-coded database has been constructed to evaluate the proposed audio-based AU recognition framework. The database consists of a “clean” subset with frontal and neutral faces and a challenging subset collected with large head movements and occlusions. Experimental results on this database show that the proposed CTBN model achieves promising recognition performance for 7 speech-related AUs and outperforms both the state-of-the-art visual-based and audio-based methods especially for those AUs that are activated at low intensities or “hardly visible” in the visual channel. The improvement is more impressive on the challenging subset, where the visual-based approaches suffer significantly.

3D Facial Action Units Recognition for Emotional Expression

The muscular activities caused the activation of facial action units (AUs) when a facial expression is shown by a human face. This paper presents the methods to recognize AU using a distance feature between facial points which activates the muscles. The seven AU involved are AU1, AU4, AU6, AU12, AU15, AU17 and AU25 that characterizes a happy and sad expression. The recognition is performed on each AU according to the rules defined based on the distance of each facial point. The facial distances chosen are computed from twelve salient facial points. Then the facial distances are trained using Support Vector Machine (SVM) and Neural Network (NN). Classification result using SVM is presented with several different SVM kernels while result using NN is presented for each training, validation and testing phase. By using any SVM kernels, it is consistent that AUs that are corresponded to sad expression has a high recognition compared to happy expression. The highest average kernel performance across AUs is 93%, scored by quadratic kernel. Best results for NN across AUs is for AU25 (Lips parted) with lowest CE (0.38%) and 0% incorrect classification.

Facial expression of pain in Nellore and crossbred beef cattle

Although the science of facial expression of pain in humans is advanced, it has not been extensively explored in nonhuman animals. The study of facial expression as an indicator of pain might represent a substantial advance in pain recognition and management in other species, not yet studied. The main objective of this work was to investigate whether specific facial action units (AUs), previously related to painful facial expressions in human and some nonhuman animals, are also activated in beef cattle during acute painful stimulation. The study also investigated whether specific facial AUs of beef cattle in acute pain differed by sex and two genotypes. The activation of AUs was examined through pictures of a total of 35 beef cattle before and during branding with a hot iron, characterizing moments of no-pain (N) and pain (P). There were 17 female and 18 male beef cattle of two different genotypes: Nellore and crossbred (1/2 Nellore, 1/4 Bonsmara, 1/8 Red Angus, and 1/8 Aberdeen Angus). Results showed no differences in activation of AUs between males and females, and a higher frequency of mouth opening in the crossbred animals. The activation of the AUs backward ears, dilated nostril, open mouth, raised inner brow, and raised outer brow were significantly associated with the presence of the painful stimulus and should be considered for the development of further pain assessment methods using facial expressions for this species. The higher proportion of open mouth reactions in crossbred compared with Nellore cattle in response to an acutely painful stimulus indicates the importance of considering genotype when assessing pain by facial expression in cattle. As more detailed information becomes available, the on-farm use of facial expression to recognize pain in cattle may override some of the difficulties in animal pain diagnosis and, consequently, allow for rapid and consistent intervention to ameliorate suffering.

Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition

Facial action unit (AU) recognition is a crucial task for facial expressions analysis and has attracted extensive attention in the field of artificial intelligence and computer vision. Existing works have either focused on designing or learning complex regional feature representations, or delved into various types of AU relationship modeling. Albeit with varying degrees of progress, it is still arduous for existing methods to handle complex situations. In this paper, we investigate how to integrate the semantic relationship propagation between AUs in a deep neural network framework to enhance the feature representation of facial regions, and propose an AU semantic relationship embedded representation learning (SRERL) framework. Specifically, by analyzing the symbiosis and mutual exclusion of AUs in various facial expressions, we organize the facial AUs in the form of structured knowledge-graph and integrate a Gated Graph Neural Network (GGNN) in a multi-scale CNN framework to propagate node information through the graph for generating enhanced AU representation. As the learned feature involves both the appearance characteristics and the AU relationship reasoning, the proposed model is more robust and can cope with more challenging cases, e.g., illumination change and partial occlusion. Extensive experiments on the two public benchmarks demonstrate that our method outperforms the previous work and achieves state of the art performance.

Dual Semi-Supervised Learning for Facial Action Unit Recognition

Current works on facial action unit (AU) recognition typically require fully AU-labeled training samples. To reduce the reliance on time-consuming manual AU annotations, we propose a novel semi-supervised AU recognition method leveraging two kinds of readily available auxiliary information. The method leverages the dependencies between AUs and expressions as well as the dependencies among AUs, which are caused by facial anatomy and therefore embedded in all facial images, independent on their AU annotation status. The other auxiliary information is facial image synthesis given AUs, the dual task of AU recognition from facial images, and therefore has intrinsic probabilistic connections with AU recognition, regardless of AU annotations. Specifically, we propose a dual semi-supervised generative adversarial network for AU recognition from partially AU-labeled and fully expressionlabeled facial images. The proposed network consists of an AU classifier C, an image generator G, and a discriminator D. In addition to minimize the supervised losses of the AU classifier and the face generator for labeled training data, we explore the probabilistic duality between the tasks using adversary learning to force the convergence of the face-AU-expression tuples generated from the AU classifier and the face generator, and the ground-truth distribution in labeled data for all training data. This joint distribution also includes the inherent AU dependencies. Furthermore, we reconstruct the facial image using the output of the AU classifier as the input of the face generator, and create AU labels by feeding the output of the face generator to the AU classifier. We minimize reconstruction losses for all training data, thus exploiting the informative feedback provided by the dual tasks. Within-database and cross-database experiments on three benchmark databases demonstrate the superiority of our method in both AU recognition and face synthesis compared to state-of-the-art works.

Geometric morphometrics for the study of facial expressions in non-human animals, using the domestic cat as an exemplar

Facial expression is a common channel for the communication of emotion. However, in the case of non-human animals, the analytical methods used to quantify facial expressions can be subjective, relying heavily on extrapolation from human-based systems. Here, we demonstrate how geometric morphometrics can be applied in order to overcome these problems. We used this approach to identify and quantify changes in facial shape associated with pain in a non-human animal species. Our method accommodates individual variability, species-specific facial anatomy, and postural effects. Facial images were captured at four different time points during ovariohysterectomy of domestic short haired cats (n = 29), with time points corresponding to varying intensities of pain. Images were annotated using landmarks specifically chosen for their relationship with underlying musculature, and relevance to cat-specific facial action units. Landmark data were subjected to normalisation before Principal Components (PCs) were extracted to identify key sources of facial shape variation, relative to pain intensity. A significant relationship between PC scores and a well-validated composite measure of post-operative pain in cats (UNESP-Botucatu MCPS tool) was evident, demonstrating good convergent validity between our geometric face model, and other metrics of pain detection. This study lays the foundation for the automatic, objective detection of emotional expressions in a range of non-human animal species.

Capturing Feature and Label Relations Simultaneously for Multiple Facial Action Unit Recognition

Although both feature dependencies and label dependencies are crucial for facial action unit (AU) recognition, little work addresses them simultaneously till now. In this paper, we propose a 4-layer Restricted Boltzmann Machine (RBM) to simultaneously capture feature level and label level dependencies to recognize multiple AUs. The middle hidden layer of the 4-layer RBM model captures dependencies among image features for multiple AUs, while the top latent units capture the high-order semantic dependencies among AU labels. Furthermore, we extend the proposed 4-layer RBM for facial expression-augmented AU recognition, since AU relations are influenced by expressions. By introducing facial expression nodes in the middle visible layer, facial expressions, which are only required during training, facilitate the estimation of both feature dependencies and label dependencies among AUs. Efficient learning and inference algorithms for the extended model are also developed. Experimental results on three benchmark databases, i.e., the CK+ database, the DISFA database and the SEMAINE database, demonstrate that the proposed approaches can successfully capture complex AU relationships from features and labels jointly, and the expression labels available only during training are benefit for AU recognition during testing for both posed and spontaneous facial expressions.

Copula Ordinal Regression Framework for Joint Estimation of Facial Action Unit Intensity

Joint modeling of the intensity of multiple facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for $ordinal$ordinal structure in output variables and their $non$non-$linear$linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.

A comprehensive survey on automatic facial action unit analysis

Facial Action Coding System is the most influential sign judgment method for facial behavior, and it is a comprehensive and anatomical system which could encode various facial movements by the combination of basic AUs (Action Units). AUs define certain facial configurations caused by contraction of one or more facial muscles, and they are independent of interpretation of emotions. However, automatic facial action unit recognition remains challenging due to several open questions such as rigid and non-rigid facial motions, multiple AUs detection, intensity estimation and naturalistic context application. This paper introduces recent advances in automatic facial action unit recognition, focusing on the fundamental components of face detection and registration, facial action representation, feature selection and classification. The comprehensive analysis of facial representations is presented according to the facial data properties (image and video, 2D and 3D) and characteristics of facial features (predesign and learning, appearance and geometry, hybrid and fusion). Facial action unit recognition involves AUs occurrence detection, AUs temporal segment detection and AUs intensity estimation. We discussed the role of each component, main techniques with their characteristics, challenges and potential directions of facial action unit analysis.

The PARK Framework for Automated Analysis of Parkinson's Disease Characteristics

There are about 900,000 people with Parkinson's disease (PD) in the United States. Even though there are benefits of early treatment, unfortunately, over 40% of individuals with PD over 65 years old do not see a neurologist. It is often very difficult for these individuals to get to a physician's office for diagnosis and subsequent monitoring. To address this problem, we present PARK, Parkinson's Analysis with Remote Kinetic-tasks. PARK instructs and guides users through six motor tasks and one audio task selected from the standardized MDS-UPDRS rating scale and records their performance via webcam. An initial experiment was conducted with 127 participants with PD and 127 age-matched controls, in which a total of 1,778 video recordings were collected. 90.6% of the PD participants agreed that PARK was easy to use, and 93.7% mentioned that they would use the system in the future. We explored objective differences between those with and without PD. A novel motion feature based on the Fast Fourier Transform (FFT) of optical flow in a region of interest was designed to quantify these differences in the collected video recordings. Additionally, we found that facial action unit AU4 (brow lowerer) was expressed significantly more often, while AU12 (lip corner puller) was expressed less often in various tasks for participants with PD.

4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations

One of the main challenges in dynamic facial expression recognition is how to capture temporal information in the system. In this study, a novel approach based on time series analysis is adapted for this problem. The proposed dynamic facial expression recognition system comprises four phases: head pose correction and normalization, feature extraction, feature selection and classification. Head pose detection and correction is the first phase to realign locations of the facial landmarks. A comprehensive set of geometric deformations including point, distance and angle deformations are extracted from the key points. The concept of facial action unit analysis is interlocked with this phase to identify related key points from the landmarks. A set of multimodal time series are then constructed from the extracted deformations by applying a sliding window to characterize the dynamics of mean deformations in a window. In the third phase, a feature selection method based on neighborhood component analysis is applied on the peak value of deformation features to select useful features and discard irrelevant ones. Finally, adaptive cost dynamic time warping is utilized to recognize six prototypic expressions from multimodal time series of selected features. Experimental results on BU-4DFE data set confirm that proposed algorithm is efficient in dynamic facial expression recognition compared with state of the art.

A pilot study to identify autism related traits in spontaneous facial actions using computer vision

BackgroundIndividuals with autism spectrum disorders (ASD) may be differentiated from typically developing controls (TDC) based on phenotypic features in spontaneous facial expressions. Computer vision technology can automatically track subtle facial actions to gain quantitative insights into ASD related behavioral abnormalities. MethodThis study proposes a novel psychovisual human-study to elicit spontaneous facial expressions in response to a variety of social and emotional contexts. We introduce a markerless facial motion capture and computer vision methods to track spontaneous and subtle activations of facial muscles. The facial muscle activations are encoded into ten representative facial action units (FAU) to gain quantitative, granular, and contextual insights into the psychophysical development of the participating individuals. Statistical tests are performed to identify differential traits in individuals with ASD after comparing those in a cohort of age-matched TDC individuals. ResultsThe proposed framework has revealed significant difference (p < 0.001) in the activation of ten FAU and contrasting activations of FAU between the group with ASD and the TDC group. Unlike the TDC group, the group with ASD has shown unusual prevalence of mouth frown (FAU 15) and low correlations in temporal activations of several FAU pairs: 6–12, 10–12, and 10–20. The interpretation of different FAU activations suggests quantitative evidence of expression bluntness, lack of expression mimicry, incongruent reaction to negative emotions in the group with ASD. ConclusionOur generalized framework may be used to quantify psychophysical traits in individuals with ASD and replicate in similar studies that require quantitative measurements of behavioral responses.

Action Units recognition based on Deep Spatial-Convolutional and Multi-label Residual network

Facial Action Unit (AU) recognition is an essential step in the facial analysis. A facial image has one or more AU(s). Given an AU, the number of images without the AU is far greater than that of images with the AU. So, AU recognition is not only a sample imbalance problem but also a multi-label learning problem. For the two problems, we proposed a novel Multi-label Slope Rate (MSR) loss function and an Advanced-MSR (Ad-MSR) loss function in deep network architecture to recognize AU. For other characters of AU recognition, a local convolution and residual units are used in the architecture. The experimental results on two expression databases labeled AU show that the proposed loss functions not only address overfitting of the network on the training set and enhancing the generalization ability on the test set. The proposed architecture also gets well performance in the databases.

Intelligent ICU for Autonomous Patient Monitoring Using Pervasive Sensing and Deep Learning

Currently, many critical care indices are not captured automatically at a granular level, rather are repetitively assessed by overburdened nurses. In this pilot study, we examined the feasibility of using pervasive sensing technology and artificial intelligence for autonomous and granular monitoring in the Intensive Care Unit (ICU). As an exemplary prevalent condition, we characterized delirious patients and their environment. We used wearable sensors, light and sound sensors, and a camera to collect data on patients and their environment. We analyzed collected data to detect and recognize patient’s face, their postures, facial action units and expressions, head pose variation, extremity movements, sound pressure levels, light intensity level, and visitation frequency. We found that facial expressions, functional status entailing extremity movement and postures, and environmental factors including the visitation frequency, light and sound pressure levels at night were significantly different between the delirious and non-delirious patients. Our results showed that granular and autonomous monitoring of critically ill patients and their environment is feasible using a noninvasive system, and we demonstrated its potential for characterizing critical care patients and environmental factors.

Conditional adversarial synthesis of 3D facial action units

Employing deep learning-based approaches for fine-grained facial expression analysis, such as those involving the estimation of Action Unit (AU) intensities, is difficult due to the lack of a large-scale dataset of real faces with sufficiently diverse AU labels for training. In this paper, we consider how AU-level facial image synthesis can be used to substantially augment such a dataset. We propose an AU synthesis framework that combines the well-known 3D Morphable Model (3DMM), which intrinsically disentangles expression parameters from other face attributes, with models that adversarially generate 3DMM expression parameters conditioned on given target AU labels, in contrast to the more conventional approach of generating facial images directly. In this way, we are able to synthesize new combinations of expression parameters and facial images from desired AU labels. Extensive quantitative and qualitative results on the benchmark DISFA dataset and BP4D dataset demonstrate the effectiveness of our method on 3DMM facial expression parameter synthesis and data augmentation for deep learning-based AU intensity estimation.

Recognition of Tactile Facial Action Units by Individuals Who Are Blind and Sighted: A Comparative Study

Given that most cues exchanged during a social interaction are nonverbal (e.g., facial expressions, hand gestures, body language), individuals who are blind are at a social disadvantage compared to their sighted peers. Very little work has explored sensory augmentation in the context of social assistive aids for individuals who are blind. The purpose of this study is to explore the following questions related to visual-to-vibrotactile mapping of facial action units (the building blocks of facial expressions): (1) How well can individuals who are blind recognize tactile facial action units compared to those who are sighted? (2) How well can individuals who are blind recognize emotions from tactile facial action units compared to those who are sighted? These questions are explored in a preliminary pilot test using absolute identification tasks in which participants learn and recognize vibrotactile stimulations presented through the Haptic Chair, a custom vibrotactile display embedded on the back of a chair. Study results show that individuals who are blind are able to recognize tactile facial action units as well as those who are sighted. These results hint at the potential for tactile facial action units to augment and expand access to social interactions for individuals who are blind.

Cross-domain AU Detection: Domains, Learning Approaches, and Measures.

Facial action unit (AU) detectors have performed well when trained and tested within the same domain. Do AU detectors transfer to new domains in which they have not been trained? To answer this question, we review literature on cross-domain transfer and conduct experiments to address limitations of prior research. We evaluate both deep and shallow approaches to AU detection (CNN and SVM, respectively) in two large, well-annotated, publicly available databases, Expanded BP4D+ and GFT. The databases differ in observational scenarios, participant characteristics, range of head pose, video resolution, and AU base rates. For both approaches and databases, performance decreased with change in domain, often to below the threshold needed for behavioral research. Decreases were not uniform, however. They were more pronounced for GFT than for Expanded BP4D+ and for shallow relative to deep learning. These findings suggest that more varied domains and deep learning approaches may be better suited for promoting generalizability. Until further improvement is realized, caution is warranted when applying AU classifiers from one domain to another.

A Facial-Action Imposter: How Head Tilt Influences Perceptions of Dominance From a Neutral Face

Research on face perception tends to focus on facial morphology and the activation of facial muscles while ignoring any impact of head position. We raise questions about this approach by demonstrating that head movements can dramatically shift the appearance of the face to shape social judgments without engaging facial musculature. In five studies (total N = 1,517), we found that when eye gaze was directed forward, tilting one’s head downward (compared with a neutral angle) increased perceptions of dominance, and this effect was due to the illusory appearance of lowered and V-shaped eyebrows caused by a downward head tilt. Tilting one’s head downward therefore functions as an action-unit imposter, creating the artificial appearance of a facial action unit that has a strong effect on social perception. Social judgments about faces are therefore driven not only by facial shape and musculature but also by movements in the face’s physical foundation: the head.