Action Unit Intensity Estimation Research Articles

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Recognising continuous emotions and action unit (AU) intensities from face videos, requires a spatial and temporal understanding of expression dynamics. Existing works primarily rely on 2D face appearance features to extract such dynamics. This work focuses on a promising alternative based on parametric 3D face alignment models, which disentangle different factors of variation, including expression-induced shape variations. We aim to understand how expressive 3D face shapes are in estimating valence-arousal and AU intensities compared to the state-of-the-art 2D appearance-based models. We benchmark five recent 3D face models: ExpNet, 3DDFA-V2, RingNet, DECA, and EMOCA. In valence-arousal estimation, expression features of 3D face models consistently surpassed previous works and yielded an average concordance correlation of. 745 and. 574 on SEWA and AVEC 2019 CES corpora, respectively. We also study how 3D face shapes performed on AU intensity estimation on BP4D and DISFA datasets, and report that 3D face features were on par with 2D appearance features in recognising AUs 4, 6, 10, 12, and 25, but not the entire set of AUs. To understand this discrepancy, we conduct a correspondence analysis between valence-arousal and AUs, which points out that accurate prediction of valence-arousal may require the knowledge of only a few AUs.

Region Attentive Action Unit Intensity Estimation With Uncertainty Weighted Multi-Task Learning

Region Attentive Action Unit Intensity Estimation With Uncertainty Weighted Multi-Task Learning

Facial Action Unit Detection Using Attention and Relation Learning

Attention mechanism has recently attracted increasing attentions in the field of facial action unit (AU) detection. By finding the region of interest of each AU with the attention mechanism, AU-related local features can be captured. Most of the existing attention based AU detection works use prior knowledge to predefine fixed attentions or refine the predefined attentions within a small range, which limits their capacity to model various AUs. In this paper, we propose an end-to-end deep learning based attention and relation learning framework for AU detection with only AU labels, which has not been explored before. In particular, multi-scale features shared by each AU are learned first, and then both channel-wise and spatial attentions are adaptively learned to select and extract AU-related local features. Moreover, pixel-level relations for AUs are further captured to refine spatial attentions so as to extract more relevant local features. Without changing the network architecture, our framework can be easily extended for AU intensity estimation. Extensive experiments show that our framework (i) soundly outperforms the state-of-the-art methods for both AU detection and AU intensity estimation on the challenging BP4D, DISFA, FERA 2015, and BP4D+ benchmarks, (ii) can adaptively capture the correlated regions of each AU, and (iii) also works well under severe occlusions and large poses.

RFAU: A Database for Facial Action Unit Analysis in Real Classrooms

Emotion analysis of students plays an important role in teaching effect evaluation. To develop robust algorithms for emotion analysis of students, a database from real classrooms is required. However, most existing databases were collected from adults and constructed in laboratory settings. In this article, we present a manually-annotated facial action unit database from juveniles in real classrooms. Our database has three main characteristics: (1) it provides numerous education-related action units data from primary and high schools, complementing the vacancy of the publicly available educational action unit databases; (2) it contains 256,220 manually-annotated facial images of 1,796 juveniles, frame-by-frame annotated with 12 action units and 6-level intensities for each action unit; (3) it covers many challenges in the wild, including various head poses, low facial resolution, illuminations, and occlusions, supplementing action unit databases in the wild for research. The baselines for action unit detection and action unit intensity estimation are provided for future references. Especially, we apply the weighted balance loss to solve imbalances <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">within</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">between</i> labels. Our database will be available to the research community: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://www.dlc.sjtu.edu.cn/rfau</uri> .

Facial Expression Manipulation for Personalized Facial Action Estimation

Limited sizes of annotated video databases of spontaneous facial expression, imbalanced action unit labels, and domain shift are three main obstacles in training models to detect facial actions and estimate their intensity. To address these problems, we propose an approach that incorporates facial expression generation for facial action unit intensity estimation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and trains a GAN-based network to synthesize novel images with facial action units of interest. We leverage the synthetic images to achieve two goals: 1) generating AU-balanced databases, and 2) tackling domain shift with personalized networks. To generate a balanced database, we synthesize expressions with varying AU intensities and perform semantic resampling. Our experimental results on FERA17 show that networks trained on synthesized facial expressions outperform those trained on actual facial expressions and surpass current state-of-the-art approaches. To tackle domain shift, we propose personalizing pretrained networks. We generate synthetic expressions of each target subject with varying AU intensity labels and use the person-specific synthetic images to fine-tune pretrained networks. To evaluate performance of the personalized networks, we use DISFA and PAIN databases. Personalized networks, which require only a single image from each target subject to generate synthetic images, achieved significant improvement in generalizing to unseen domains.

Deep Facial Action Unit Recognition and Intensity Estimation from Partially Labelled Data

Research on facial action unit (AU) analysis typically require facial images that are labelled with those action units. While unlabelled facial images abound, labelling those images with action units or intensity is costly and time-consuming. Our approach makes it possible to analyze facial AUs when only some of the images have been labelled. We use many facial images to learn a deep framework that is able to take advantage of the facial representations. A restricted Boltzmann machine uses the available AU annotations to learn the AU label or intensity distribution. We train a support vector machine for AU recognition and a support vector regression for AU intensity estimation by maximizing the log likelihood of the AU mapping functions, taking into account the learned multiple AU distribution for all training data, while simultaneously diminishing errors between the predicted action units and ground-truth action unit occurrence or intensities for all labelled data. We perform experiments on two databases. The results demonstrate the superiority of a deep neural network for learning facial features, as well as the benefit of action unit label or intensity constraints for action unit occurrence recognition or intensity estimation in fully or semi-supervised scenarios.

Automatic Estimation of Action Unit Intensities and Inference of Emotional Appraisals

The development of a two-stage approach for appraisal inference from automatically detected Action Unit (AUs) intensities in recordings of human faces is described. AU intensity estimation is based on a hybrid approach fusing information from individually fitted mesh model of the faces and texture information. Evaluation results for two datasets and a comparison against a state of the art system are provided. In the second stage, the emotional appraisals novelty, valence and control are predicted from estimated AU intensities by linear regressions. Prediction performance is evaluated based on face recordings from a market research study, which were rated by human observers in terms of perceived appraisals. Predictions of valence and control from automatically estimated AU intensities closely match those obtained from manually coded AUs in terms of agreement with human observers, while novelty predictions lag somewhat behind. Overall, results highlight the flexibility and interpretability of a two-stage approach to emotion inference.

Landmark-Based Facial Feature Construction and Action Unit Intensity Prediction

Human face recognition has been widely used in many fields, including biorobots, driver fatigue monitoring, and polygraph tests. However, the end-to-end models fit by most of the existing algorithms perform poorly in interpretation because complex classifiers are constructed using facial images directly. In addition, in some of the models, dynamic characteristics of subjects as individuals are not fully considered, so dynamic information is not extracted. In order to solve these problems, this paper proposes an action unit intensity prediction model. The three-dimensional coordinates of 68 landmarks of human faces are obtained based on the convolutional experts constrained local model (CE-CLM), which enables the construction of dynamic facial features. Based on the error analysis of the CE-CLM algorithm, dimension reduction of the constructed features is performed by the principal components analysis (PCA). The radial basis function (RBF) neural network is also constructed to train the action unit prediction models. The proposed method is verified by the experiments, and the overall mean square error (MSE) of the proposed method is 0.01826. Lastly, the network construction process is optimized, so that for the same training samples, the models are fitted using fewer iterations. The number of iterations is decreased by 27 on average. In summary, this paper provides a method to rapidly construct action unit (AU) intensity prediction models and constructs automatic AU intensity estimation models for facial images.

A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation

Action Units (AUs) are geometrically-based atomic facial muscle movements known to produce appearance changes at specific facial locations. Motivated by this observation we propose a novel AU modelling problem that consists of jointly estimating their localisation and intensity. To this end, we propose a simple yet efficient approach based on Heatmap Regression that merges both problems into a single task. A Heatmap models whether an AU occurs or not at a given spatial location. To accommodate the joint modelling of AUs intensity, we propose variable size heatmaps, with their amplitude and size varying according to the labelled intensity. Using Heatmap Regression, we can inherit from the progress recently witnessed in facial landmark localisation. Building upon the similarities between both problems, we devise a transfer learning approach where we exploit the knowledge of a network trained on large-scale facial landmark datasets. In particular, we explore different alternatives for transfer learning through a) fine-tuning, b) adaptation layers, c) attention maps, and d) reparametrisation. Our approach effectively inherits the rich facial features produced by a strong face alignment network, with minimal extra computational cost. We empirically validate that our system sets a new state-of-the-art on three popular datasets, namely BP4D, DISFA, and FERA2017.

A comprehensive survey on automatic facial action unit analysis

Facial Action Coding System is the most influential sign judgment method for facial behavior, and it is a comprehensive and anatomical system which could encode various facial movements by the combination of basic AUs (Action Units). AUs define certain facial configurations caused by contraction of one or more facial muscles, and they are independent of interpretation of emotions. However, automatic facial action unit recognition remains challenging due to several open questions such as rigid and non-rigid facial motions, multiple AUs detection, intensity estimation and naturalistic context application. This paper introduces recent advances in automatic facial action unit recognition, focusing on the fundamental components of face detection and registration, facial action representation, feature selection and classification. The comprehensive analysis of facial representations is presented according to the facial data properties (image and video, 2D and 3D) and characteristics of facial features (predesign and learning, appearance and geometry, hybrid and fusion). Facial action unit recognition involves AUs occurrence detection, AUs temporal segment detection and AUs intensity estimation. We discussed the role of each component, main techniques with their characteristics, challenges and potential directions of facial action unit analysis.

Conditional adversarial synthesis of 3D facial action units

Employing deep learning-based approaches for fine-grained facial expression analysis, such as those involving the estimation of Action Unit (AU) intensities, is difficult due to the lack of a large-scale dataset of real faces with sufficiently diverse AU labels for training. In this paper, we consider how AU-level facial image synthesis can be used to substantially augment such a dataset. We propose an AU synthesis framework that combines the well-known 3D Morphable Model (3DMM), which intrinsically disentangles expression parameters from other face attributes, with models that adversarially generate 3DMM expression parameters conditioned on given target AU labels, in contrast to the more conventional approach of generating facial images directly. In this way, we are able to synthesize new combinations of expression parameters and facial images from desired AU labels. Extensive quantitative and qualitative results on the benchmark DISFA dataset and BP4D dataset demonstrate the effectiveness of our method on 3DMM facial expression parameter synthesis and data augmentation for deep learning-based AU intensity estimation.

Semi-Supervised Deep Neural Network for Joint Intensity Estimation of Multiple Facial Action Units

Facial action units (AUs) are defined to depict movements of facial muscles, which are basic elements to encode facial expressions. Automatic AU intensity estimation is an important task in affective computing. Previous works leverage the representation power of deep neural networks (DNNs) to improve the performance of intensity estimation. However, a large number of intensity annotations are required to train DNNs that contain millions of parameters. But it is expensive and difficult to build a large-scale database with AU intensity annotation since AU annotation requires annotators have strong domain expertise. We propose a novel semi-supervised deep convolutional network that leverages extremely limited AU annotations for AU intensity estimation. It requires only intensity annotations of keyframes of training sequences. Domain knowledge on AUs is leveraged to provide weak supervisory information, including relative appearance similarity, temporal intensity ordering, facial symmetry, and contrastive appearance difference. We also propose a strategy to train a model for joint intensity estimation of multiple AUs under the setting of semi-supervised learning, which greatly improves the efficiency during inference. We perform empirical experiments on two public benchmark expression databases and make comparisons with state-of-the-art methods to demonstrate the effectiveness of the proposed method.

Facial Action Unit Recognition and Intensity Estimation Enhanced through Label Dependencies.

The inherent dependencies among facial action units (AU) caused by the underlying anatomic mechanism are essential for the proper recognition of AUs and estimation of intensity levels, but they have not been exploited to their full potential. We are proposing novel methods to recognize AUs and estimate intensity via hybrid Bayesian networks. The upper two layers are latent regression Bayesian networks (LRBNs), and the lower layers are Bayesian networks (BNs). The visible nodes of the LRBN layers are representations of ground-truth AU occurrences or AU intensities. Through the directed connections from latent layer and visible layer, an LRBN can successfully represent relationships between multiple AUs or AU intensities. The lower layers include Bayesian networks with two nodes for AU recognition, and Bayesian networks with three nodes for AU intensity estimation. The bottom layers incorporate measurements from facial images with AU dependencies for intensity estimation and AU recognition. Efficient learning algorithms of the hybrid Bayesian networks are proposed for AU recognition as well as intensity estimation. Furthermore, the proposed hybrid Bayesian network models are extended for facial expression-assisted AU recognition and intensity estimation, as AU relationships are closely related to facial expressions. We test our methods on three benchmark databases for AU recognition and two benchmark databases for intensity estimation. The results demonstrate that the proposed approaches faithfully model the complex and global inherent AU dependencies, and the expression labels available only during training can boost the estimation of AU dependencies for both AU recognition and intensity estimation.

Feature and label relation modeling for multiple-facial action unit classification and intensity estimation

In this paper, we propose multiple facial Action Unit (AU) recognition and intensity estimation by modeling their relations in both feature and label spaces. First, a multi-task feature learning method is adopted to learn the shared features among the group of facial action units, and recognize or estimate their intensity simultaneously. Second, a Bayesian network is used to model the co-existent and mutual-exclusive semantic relations among action units. Finally, through probabilistic inference, the learned Bayesian network combines the results of the multi-task learning with the AU relations it captures to perform multiple AU recognition and AU intensity estimation. Experiments on the extended Cohn-Kanade database, the MMI database, the McMaster database and the DISFA database demonstrate the effectiveness of our method for both AU classification and AU intensity estimation.

Expression-assisted facial action unit recognition under incomplete AU annotation

Facial action unit (AU) recognition is an important task for facial expression analysis. Traditional AU recognition methods typically include a supervised training, where the AU annotated training images are needed. AU annotation is a time consuming, expensive, and error prone process. While AU is hard to annotate, facial expression is relatively easy to label. To take advantage of this, we introduce a new learning method that trains an AU classifier using images with incomplete AU annotation but with complete expression labels. The goal is to use expression labels as hidden knowledge to complement the missing AU labels. Towards this goal, we propose to construct a Bayesian network (BN) to capture the relationships among facial expressions and AUs. Structural expectation maximization (SEM) is used to learn the structure and parameters of the BN when the AU labels are missing. Given the learned BNs and measurements of AUs and expression, we can then perform AU recognition within the BN through a probabilistic inference. Experimental results on the CK+, ISL and BP4D-Spontaneous databases demonstrate the effectiveness of our method for both AU classification and AU intensity estimation.

Adaptive facial point detection and emotion recognition for a humanoid robot

Automatic perception of facial expressions with scaling differences, pose variations and occlusions would greatly enhance natural human robot interaction. This research proposes unsupervised automatic facial point detection integrated with regression-based intensity estimation for facial action units (AUs) and emotion clustering to deal with such challenges. The proposed facial point detector is able to detect 54 facial points in images of faces with occlusions, pose variations and scaling differences using Gabor filtering, BRISK (Binary Robust Invariant Scalable Keypoints), an Iterative Closest Point (ICP) algorithm and fuzzy c-means (FCM) clustering. Especially, in order to effectively deal with images with occlusions, ICP is first applied to generate neutral landmarks for the occluded facial elements. Then FCM is used to further reason the shape of the occluded facial region by taking the prior knowledge of the non-occluded facial elements into account. Post landmark correlation processing is subsequently applied to derive the best fitting geometry for the occluded facial element to further adjust the neutral landmarks generated by ICP and reconstruct the occluded facial region. We then conduct AU intensity estimation respectively using support vector regression and neural networks for 18 selected AUs. FCM is also subsequently employed to recognize seven basic emotions as well as neutral expressions. It also shows great potential to deal with compound and newly arrived novel emotion class detection. The overall system is integrated with a humanoid robot and enables it to deal with challenging real-life facial emotion recognition tasks.

Adaptive 3D facial action intensity estimation and emotion recognition

Automatic recognition of facial emotion has been widely studied for various computer vision tasks (e.g. health monitoring, driver state surveillance and personalized learning). Most existing facial emotion recognition systems, however, either have not fully considered subject-independent dynamic features or were limited to 2D models, thus are not robust enough for real-life recognition tasks with subject variation, head movement and illumination change. Moreover, there is also lack of systematic research on effective newly arrived novel emotion class detection. To address these challenges, we present a real-time 3D facial Action Unit (AU) intensity estimation and emotion recognition system. It automatically selects 16 motion-based facial feature sets using minimal-redundancy–maximal-relevance criterion based optimization and estimates the intensities of 16 diagnostic AUs using feedforward Neural Networks and Support Vector Regressors. We also propose a set of six novel adaptive ensemble classifiers for robust classification of the six basic emotions and the detection of newly arrived unseen novel emotion classes (emotions that are not included in the training set). A distance-based clustering and uncertainty measures of the base classifiers within each ensemble model are used to inform the novel class detection. Evaluated with the Bosphorus 3D database, the system has achieved the best performance of 0.071 overall Mean Squared Error (MSE) for AU intensity estimation using Support Vector Regressors, and 92.2% average accuracy for the recognition of the six basic emotions using the proposed ensemble classifiers. In comparison with other related work, our research outperforms other state-of-the-art research on 3D facial emotion recognition for the Bosphorus database. Moreover, in on-line real-time evaluation with real human subjects, the proposed system also shows superior real-time performance with 84% recognition accuracy and great flexibility and adaptation for newly arrived novel (e.g. ‘contempt’ which is not included in the six basic emotions) emotion detection.

3D shape estimation in video sequences provides high precision evaluation of facial expressions

Person independent and pose invariant estimations of facial expressions and action unit (AU) intensity estimation are important for situation analysis and for automated video annotation. We evaluated raw 2D shape data of the CK+ database, used Procrustes transformation and the multi-class SVM leave-one-out method for classification. We found close to 100% performance demonstrating the relevance and the strength of details of the shape. Precise 3D shape information was computed by means of constrained local models (CLM) on video sequences. Such sequences offer the opportunity to compute a time-averaged ‘3D personal mean shape’ (PMS) from the estimated CLM shapes, which – upon subtraction – gives rise to person independent emotion estimation. On CK+ data PMS showed significant improvements over AU0 normalization; performance reached and sometimes surpassed state-of-the-art results on emotion classification and on AU intensity estimation. 3D PMS from 3D CLM offers pose invariant emotion estimation that we studied by rendering a 3D emotional database for different poses and different subjects from the BU 4DFE database. Frontal shapes derived from CLM fits of the 3D shape were evaluated. Results demonstrate that shape estimation alone can be used for robust, high quality pose invariant emotion classification and AU intensity estimation.