Action Unit Recognition Research Articles

Hard semantic mask strategy for automatic facial action unit recognition with teacher–student model

Hard semantic mask strategy for automatic facial action unit recognition with teacher–student model

Joint facial action unit recognition and self-supervised optical flow estimation

Facial action unit (AU) recognition and optical flow estimation are two highly correlated tasks, since optical flow can provide motion information of facial muscles to facilitate AU recognition. However, most existing AU recognition methods handle the two tasks independently by offline extracting optical flow as auxiliary information or directly ignoring the use of optical flow. In this paper, we propose a novel end-to-end joint framework of AU recognition and optical flow estimation, in which the two tasks contribute to each other. Moreover, due to the lack of optical flow annotations in AU datasets, we propose to estimate optical flow in a self-supervised manner. To regularize the self-supervised estimation of optical flow, we propose an identical mapping constraint for the optical flow guided image warping process, in which the estimated optical flow between two same images is required to not change the image during warping. Experiments demonstrate that our framework (i) outperforms most of the state-of-the-art AU recognition methods on the challenging BP4D and GFT benchmarks, and (ii) also achieves competitive self-supervised optical flow estimation performance.

Facial Action Unit Recognition Enhanced by Text Descriptions of FACS

Facial Action Unit Recognition Enhanced by Text Descriptions of FACS

Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori Constraints.

Facial action units (AUs) focus on a comprehensive set of atomic facial muscle movements for human expression understanding. Based on supervised learning, discriminative AU representation can be achieved from local patches where the AUs are located. Unfortunately, accurate AU localization and characterization are challenged by the tremendous manual annotations, which limits the performance of AU recognition in realistic scenarios. In this study, we propose an end-to-end self-supervised AU representation learning model (SsupAU) to learn AU representations from unlabeled facial videos. Specifically, the input face is decomposed into six components using auto-encoders: five photo-geometric meaningful components, together with 2D flow field AUs. By constructing the canonical neutral face, posed neutral face, and posed expressional face gradually, these components can be disentangled without supervision, therefore the AU representations can be learned. To construct the canonical neutral face without manually labeled ground truth of emotion state or AU intensity, two priori knowledge based assumptions are proposed: 1) identity consistency, which explores the identical albedos and depths of different frames in a face video, and helps to learn the camera color mode as an extra cue for canonical neutral face recovery. 2) average face, which enables the model to discover a 'neutral facial expression' of the canonical neutral face and decouple the AUs in representation learning. To the best of our knowledge, this is the first attempt to design self-supervised AU representation learning method based on the definition of AUs. Substantial experiments on benchmark datasets have demonstrated the superior performance of the proposed work in comparison to other state-of-the-art approaches, as well as an outstanding capability of decomposing input face into meaningful factors for its reconstruction. The code is made available at https://github.com/Sunner4nwpu/SsupAU.

Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition

As Facial Expression Recognition (FER) systems become integrated into our daily lives, these systems need to prioritise making <i>fair</i> decisions instead of only aiming at higher individual accuracy scores. From surveillance systems, to monitoring the mental and emotional health of individuals, these systems need to balance the <i>accuracy versus fairness</i> trade-off to make decisions that do not unjustly discriminate against specific under-represented demographic groups. Identifying <i>bias</i> as a critical problem in facial analysis systems, different methods have been proposed that aim to mitigate bias both at data and algorithmic levels. In this work, we propose the novel use of Continual Learning (CL), in particular, using Domain-Incremental Learning (Domain-IL) settings, as a potent bias mitigation method to enhance the <i>fairness</i> of Facial Expression Recognition (FER) systems. We compare different non-Continual Learning (CL)-based and CL-based methods for their <i>performance</i> and <i>fairness scores</i> on expression recognition and Action Unit (AU) detection tasks using two popular benchmarks, the RAF-DB and BP4D datasets, respectively. Our experimental results show that CL-based methods, on average, outperform other popular bias mitigation techniques on both <i>accuracy</i> and <i>fairness</i> metrics.

Multi-Order Networks for Action Unit Detection

Action Units (AU) are muscular activations used to describe facial expressions. Therefore accurate AU recognition unlocks unbiaised face representation which can improve face-based affective computing applications. From a learning standpoint AU detection is a multi-task problem with strong inter-task dependencies. To solve such problem, most approaches either rely on weight sharing, or add explicit dependency modelling by decomposing the joint task distribution using Bayes chain rule. If the latter strategy yields comprehensive inter-task relationships modelling, it requires imposing an arbitrary order into an unordered task set. Crucially, this ordering choice has been identified as a source of performance variations. In this paper, we present Multi-Order Network (MONET), a multi-task method with joint task order optimization. MONET uses a differentiable order selection to jointly learn task-wise modules with their optimal chaining order. Furthermore, we introduce warmup and order dropout to enhance order selection by encouraging order exploration. Experimentally, we first demonstrate MONET capacity to retrieve the optimal order in a toy environment. Second, we validate MONET architecture by showing that MONET outperforms existing multi-task baselines on multiple attribute detection problems chosen for their wide range of dependency settings. More importantly, we demonstrate that MONET significantly extends state-of-the-art performance in AU detection.

Micro-expression action unit recognition based on dynamic image and spatial pyramid

Micro-expression action unit recognition based on dynamic image and spatial pyramid

Drop-relationship learning for semi-supervised facial action unit recognition

Facial action units (AUs) recognition becomes important for facial analysis and has been widely applied in psychological research. Existing work on AU relationship learning only considers the relationship between AUs simultaneously, and does not consider the complex co-adaptation between AUs. Co-adaptation is good, but not always good. Learning the AU relationship simultaneously will inhibit the robust features of network learning. And AU relationship learning often requires additional landmark information. To solve these problems, we propose a novel AU Drop-relationship learning inspired by dropout. The AU Drop-relationship learning constructs the AU relationship units through prior knowledge. We further design the relationship regularization module to constraint the relationship between AU pairs. We randomly drop AU relationship units during training to suppress co-adaptation, forcing the network to learn more robust features. In addition, considering that there are massive unlabeled web facial images in reality, manual labeling of AU requires experts and is particularly time-consuming. To address this problem, we propose a method composed of consistency regularization and pseudo-multi-labeling for semi-supervised AU recognition. The proposed method outperforms the state-of-the-art semi-supervised and supervised methods on two widely used AU datasets (BP4D and DISFA).

Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Facial landmark detection and action unit~(AU) recognition are two essential tasks in facial analysis. Previous works rarely consider the relationship between these complementary tasks. In this paper, we introduce a novel multi-task dual learning framework to exploit the relationship between facial landmark detection and AU recognition while simultaneously addressing both tasks. When both tasks share middle-level features, common patterns can be exploited and middle- and high-level features can be used to perform facial landmark detection and AU recognition, respectively. In addition, a dual learning mechanism is designed to convert the predicted landmarks and AUs of the label space to the corresponding facial image of the image space, further exploring the strong correlations between the tasks. By jointly training the proposed method at both the feature and label levels, each task improves the other. Experiments on two benchmark databases demonstrate that the proposed method can leverage dependencies to boost the generalization of both tasks.

Exploring facial expressions and action unit domains for Parkinson detection.

Patients suffering from Parkinson's disease (PD) present a reduction in facial movements called hypomimia. In this work, we propose to use machine learning facial expression analysis from face images based on action unit domains to improve PD detection. We propose different domain adaptation techniques to exploit the latest advances in automatic face analysis and face action unit detection. Three different approaches are explored to model facial expressions of PD patients: (i) face analysis using single frame images and also using sequences of images, (ii) transfer learning from face analysis to action units recognition, and (iii) triplet-loss functions to improve the automatic classification between patients and healthy subjects. Real face images from PD patients show that it is possible to properly model elicited facial expressions using image sequences (neutral, onset-transition, apex, offset-transition, and neutral) with accuracy improvements of up to 5.5% (from 72.9% to 78.4%) with respect to single-image PD detection. We also show that our proposed action unit domain adaptation provides improvements of up to 8.9% (from 78.4% to 87.3%) with respect to face analysis. Finally, we also show that triplet-loss functions provide improvements of up to 3.6% (from 78.8% to 82.4%) with respect to action unit domain adaptation applied upon models created from scratch. The code of the experiments is available at https://github.com/luisf-gomez/Explorer-FE-AU-in-PD. Domain adaptation via transfer learning methods seem to be a promising strategy to model hypomimia in PD patients. Considering the good results and also the fact that only up to five images per participant are considered in each sequence, we believe that this work is a step forward in the development of inexpensive computational systems suitable to model and quantify problems of PD patients in their facial expressions.

A novel dual-channel graph convolutional neural network for facial action unit recognition

Facial Action Unit (AU) recognition is a challenging problem, where the subtle muscle movement brings diverse AU representations. Recently, AU relations are utilized to assist AU recognition and improve the understanding of AUs. Nevertheless, simply adopting the regular Bayesian networks or the relations between AUs and emotions is not enough for modelling complex AU relations. To provide quantitative measurement of AU relations using the knowledge from FACS, we propose an AU relation quantization autoencoder. Moreover, to cope with the diversity of AUs generated from the individual representation differences and other environmental impacts, we propose a dual-channel graph convolutional neural network (DGCN) obtaining both inherent and random AU relations. The first channel is FACS-based relation graph convolution channel (FACS-GCN) embedding prior knowledge of FACS, and it adjusts the network to the inherent AU dependent relations. The second channel is data-learning-based relation graph convolution channel (DLR-GCN) based on metric learning, and it provides robustness for individual differences and environmental changes. Comprehensive experiments have been conducted on three public datasets: CK+, RAF-AU and DISFA. The experimental results demonstrate that our proposed DGCN can extract the hidden relations well, thereby achieving great performance in AU recognition.

Processing Real-Life Recordings of Facial Expressions of Polish Sign Language Using Action Units

Automatic translation between the national language and sign language is a complex process similar to translation between two different foreign languages. A very important aspect is the precision of not only manual gestures but also facial expressions, which are extremely important in the overall context of a sentence. In this article, we present the problem of including facial expressions in the automation of Polish-to-Polish Sign Language (PJM) translation-this is part of an ongoing project related to a comprehensive solution allowing for the animation of manual gestures, body movements and facial expressions. Our approach explores the possibility of using action unit (AU) recognition in the automatic annotation of recordings, which in the subsequent steps will be used to train machine learning models. This paper aims to evaluate entropy in real-life translation recordings and analyze the data associated with the detected action units. Our approach has been subjected to evaluation by experts related to Polish Sign Language, and the results obtained allow for the development of further work related to automatic translation into Polish Sign Language.

Weakly Supervised Regional and Temporal Learning for Facial Action Unit Recognition

Automatic facial action unit (AU) recognition is a challenging task due to the scarcity of manual annotations. To alleviate this problem, a large amount of efforts has been dedicated to exploiting various weakly supervised methods which leverage numerous unlabeled data. However, many aspects with regard to some unique properties of AUs, such as the regional and relational characteristics, are not sufficiently explored in previous works. Motivated by this, we take the AU properties into consideration and propose two auxiliary AU related tasks to bridge the gap between limited annotations and the model performance in a self-supervised manner via the unlabeled data. Specifically, to enhance the discrimination of regional features with AU relation embedding, we design a task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a single image based optical flow estimation task is proposed to leverage the dynamic change of facial muscles and encode the motion information into the global feature representation. Based on these two self-supervised auxiliary tasks, local features, mutual relation and motion cues of AUs are better captured in the backbone network. Furthermore, by incorporating semi-supervised learning, we propose an end-to-end trainable framework named weakly supervised regional and temporal learning (WSRTL) for AU recognition. Extensive experiments on BP4D and DISFA demonstrate the superiority of our method and new state-of-the-art performances are achieved.

An emotion index estimation based on facial action unit prediction

Automated emotion recognition is essential in human-computer interactions. Instant emotion computing based on facial expressions plays a supporting role in understanding interactive intentions. Although continuous emotions can more accurately describe the overall emotional state, most existing research still focuses on discrete facial expression recognition. This paper proposes computing continuous emotions based on action unit (AU) recognition. A continuous emotion representation and computing approach are presented, which is called the emotion index. We first define two polarities as represented by positive and negative emotions. We map compound emotions into continuous space using the relationship between AUs and discrete emotions, which then creates the mapping between AUs and continuous emotion. Second, an end-to-end convolutional neural network (CNN) is used to recognize AU occurrence and intensity. We then build the facial AU matrix. Finally, a function is built to transfer the facial AU occurrence and intensities into the emotion index through the emotion relative matrix. The experiments show that our AU recognition approach can attain a better performance than state-of-the-art approaches. Moreover, our approach has an outstanding average F1 score of 66.9%. The emotion index results are in line with expectations, indicating they can effectively express emotion variations.

Facial Action Unit Recognition by Prior and Adaptive Attention

Facial action unit (AU) recognition remains a challenging task, due to the subtlety and non-rigidity of AUs. A typical solution is to localize the correlated regions of each AU. Current works often predefine the region of interest (ROI) of each AU via prior knowledge, or try to capture the ROI only by the supervision of AU recognition during training. However, the predefinition often neglects important regions, while the supervision is insufficient to precisely localize ROIs. In this paper, we propose a novel AU recognition method by prior and adaptive attention. Specifically, we predefine a mask for each AU, in which the locations farther away from the AU centers specified by prior knowledge have lower weights. A learnable parameter is adopted to control the importance of different locations. Then, we element-wise multiply the mask by a learnable attention map, and use the new attention map to extract the AU-related feature, in which AU recognition can supervise the adaptive learning of a new attention map. Experimental results show that our method (i) outperforms the state-of-the-art AU recognition approaches on challenging benchmark datasets, and (ii) can accurately reason the regional attention distribution of each AU by combining the advantages of both the predefinition and the supervision.

Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition

Subject-invariant facial action unit (AU) recognition remains challenging for the reason that the data distribution varies among subjects. In this paper, we propose a causal inference framework for subject-invariant facial action unit recognition. To illustrate the causal effect existing in AU recognition task, we formulate the causalities among facial images, subjects, latent AU semantic relations, and estimated AU occurrence probabilities via a structural causal model. By constructing such a causal diagram, we clarify the causal-effect among variables and propose a plug-in causal intervention module, CIS, to deconfound the confounder Subject in the causal diagram. Extensive experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of our CIS, and the model with CIS inserted, CISNet, has achieved state-of-the-art performance.

Correction to: Automatic stress analysis from facial videos based on deep facial action units recognition

Correction to: Automatic stress analysis from facial videos based on deep facial action units recognition

Data-aware relation learning-based graph convolution neural network for facial action unit recognition

In the paper, we propose a novel Data-aware relation graph convolutional neural network (DAR-GCN) for AU recognition. With learning and updating the relation dynamically, it facilitates modeling the potential dynamic individual facial expressing manner and accordingly improves the AU recognition under the unconstrained environment. Taking the psychological research knowledge of AUs as a reference, we adopt the consensus widely-used AUs and six basic emotions as vertexes, and their co-occurrence or ex-occurrence relations between AUs and the emotion dependent relation as the edges to construct the graph. Moreover, the Data-aware relation Graph Generator (DAR-GG) module is proposed to learn the relations with data-driven metric learning. This proposed scheme is benefit for calculating and updating AU relations from data, which facilitates to extract specific relations causing by individual expressing characteristics as well as inherent relations due to facial anatomical structure. Comparative experiments are done on three public datasets: CK+, RAF-AU and DISFA. Experimental results demonstrate that our proposed method achieves a higher AU recognition accuracy rate than the baseline based on the graph with fixed AU relations defined from the psychological knowledge. Additionally, our proposed approach outperforms several existing state-of-the-art AU recognition method by utilizing GCN-based dynamic AU relations learning strategies.

Facial Action Units Detection to Identify Interest Emotion: An Application of Deep Learning

For human beings, facial expression is one of the most powerful and natural ways to communicate their emotions and intentions. A human being is able to recognize facial expressions effortlessly, but for a machine, this task is very difficult. Today, facial expression recognition is proving to be one of the most relevant applications in many fields such as human–computer interaction, medicine, security, education, etc. Facial Action Coding System (FACS) is a method that describes face movements. This later became a main description tool used in the studies concerned with facial expression. In this paper, we propose an action units recognition system which reflects interest emotion using deep learning, particularly the Convolutional Neural Network (CNN) architecture “MobileNetV2”. Our choice of this system is motivated by its success in image classification. Our system allows detecting action units which define interest emotion from an input image. In other words, our classifier differentiates interest facial movements from other affective states. This identification is very useful in several areas, particularly in e-learning: For example, knowing whether a distance learner is interested or not in the course certainly influences the quality of learning and reduces the dropout rate. Our proposed approach presented a very satisfactory recognition rate despite the absence of a large database.

Deep Facial Action Unit Recognition and Intensity Estimation from Partially Labelled Data

Research on facial action unit (AU) analysis typically require facial images that are labelled with those action units. While unlabelled facial images abound, labelling those images with action units or intensity is costly and time-consuming. Our approach makes it possible to analyze facial AUs when only some of the images have been labelled. We use many facial images to learn a deep framework that is able to take advantage of the facial representations. A restricted Boltzmann machine uses the available AU annotations to learn the AU label or intensity distribution. We train a support vector machine for AU recognition and a support vector regression for AU intensity estimation by maximizing the log likelihood of the AU mapping functions, taking into account the learned multiple AU distribution for all training data, while simultaneously diminishing errors between the predicted action units and ground-truth action unit occurrence or intensities for all labelled data. We perform experiments on two databases. The results demonstrate the superiority of a deep neural network for learning facial features, as well as the benefit of action unit label or intensity constraints for action unit occurrence recognition or intensity estimation in fully or semi-supervised scenarios.