Facial Action Unit Intensity Research Articles

Depression Classification Based on Facial Action Unit Intensity Features Using CNN-Poolingless Framework

Depression Classification Based on Facial Action Unit Intensity Features Using CNN-Poolingless Framework

Facial Action Unit Detection and Intensity Estimation From Self-Supervised Representation

Facial Action Unit Detection and Intensity Estimation From Self-Supervised Representation

Application of Facial Expression Analysis Technology in Violence Risk Assessment of Individuals with Mental Disorders in Supervised Settings.

To explore the association between violent behaviors and emotions in individuals with mental disorders, to evaluate the application value of facial expression analysis technology in violence risk assessment of individuals with mental disorders in supervised settings, and to provide a reference for violence risk assessment. Thirty-nine male individuals with mental disorders in supervised settings were selected, the participant risk of violence, cognitive function, psychiatric symptoms and severity were assessed using the Modified Overt Aggression Scale (MOAS), the Historical, Clinical, Risk Management-Chinese version(HCR-CV), the Positive and Negative Syndrome Scale (PANSS) and the Brief Psychiatric Rating Scale (BPRS). An emotional arousal was performed on the participants and the intensity of their emotions and facial expression action units was recorded before, during and after the arousal. One-way analysis of variance (ANOVA) was used to compare the differences in the intensity of emotions and facial expression action units before, during and after the arousal. Pearson correlation analysis was used to calculate the correlations between the intensity of the seven basic emotional facial expressions and the scores of the assessment scales. The intensity difference of sadness, surprise and fear in different time periods was statistically significant (P<0.05). The intensity of the left medial eyebrow lift action unit was found significantly different before and after the emotional arousal (P<0.05). The intensity of anger was positively correlated with the Modified Overt Aggression Scale score throughout the experiment (P<0.05). Eye action units such as eyebrow lifting, eyelid tightening and upper eyelid lifting can be used as effective action units to identify sadness, anger and other negative emotions associated with violent behaviors. Facial expression analysis technology can be used as an auxiliary tool to assess the potential risk of violence in individuals with mental disorders in supervised settings.

Assessing Consciousness in Patients With Disorders of Consciousness Using a Facial Action Unit Intensity Estimation Method Based on a Hybrid Similar Feature Network

Assessing Consciousness in Patients With Disorders of Consciousness Using a Facial Action Unit Intensity Estimation Method Based on a Hybrid Similar Feature Network

How Brow Rotation Affects Emotional Expression Utilizing Artificial Intelligence.

It is well known that brow position affects emotional expression. However, there is little literature on how and to what degree this change in emotional expression happens. Previous studies on this topic have utilized manual rating; this method of study remains small and labor intensive. Our objective is to correlate manual brow rotations with emotional outcomes using artificial intelligence to objectively determine how specific brow manipulations affected human expression. We included 53 brow-lift patients in this study. Pre-operative patients' brows were rotated to - 20, - 10, +10, and +20 degrees in respect to the central axis of their existing brow using PIXLR, a cloud-based set of image editing tools and utilities. These images were analyzed using FaceReader, a validated software package that uses computer vision technology for facial expression recognition. The primary facial emotion and intensity of facial action units (0 = no action unit detected to 4 = most intense action unit detected) generated by the software were recorded. 265 total images [5 images (pre-operative, - 20 degree brow rotation, - 10, +10, and +20) per patient] were analyzed using FaceReader. The primary emotion detected in the majority of images was neutral. The percentage of disgust in patients' expressions, as detected by FaceReader, increased with increased positive brow rotation (1.76% disgust detected at - 20 degrees, 2.09% at - 10 degrees, 2.65% at neutral, 2.61% at +10 degrees, and 2.95% at +20 degrees). In contrast, the percentage of sadness in patients' expressions decreased with increased positive brow rotation (29.92% sadness detected at - 20 degrees, 21.5% at - 10 degrees, 11.42% at neutral, 15.75% at +10 degrees, and 12.86% at +20 degrees). Our facial action unit analysis corresponded with primary emotion analysis. The intensity of the inner brow raiser decreased with increased positive brow rotation 8.54% at - 20 degrees, 4.21% at - 10 degrees, 1.48% at neutral, 0.84% at +10 degrees, and 0.76% at +20 degrees). The intensity of the outer brow raiser increased with increased positive brow rotation (0.97% at - 20 degrees, 0.45% at - 10 degrees, 1.12% at neutral, 5.45% at +10 degrees, and 11.19% at +20 degrees). We demonstrated that increasing the degree of brow rotation correlated positively with the percentage of disgust and inversely with the percentage of sadness detected by FaceReader. This study demonstrated how different manipulated brow positions affected emotional outcomes using artificial intelligence. Physicians can use these findings to better understand how brow-lifts can affect the perceived emotion of their patients. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

C3DBed: Facial micro-expression recognition with three-dimensional convolutional neural network embedding in transformer model

Facial micro-expression is often used for emotional recognition of people in a high-risk or pressure scene, which may reflect genuine emotions due to the low intensity of facial action units. Current methods focus on locating regions with emotional changes and cropping these regions for local feature extraction. However, these methods may lead to the problem of information redundancy caused by overlapping cropped regions. This paper proposes a novel three-dimensional convolutional neural network embedding in the transformer model (C3DBed). This model learns the attention weight of each local region of the micro-expression image, thereby perceiving the detail changes of the facial image and extracting robust local detail features. Solve the problem of model complexity and information redundancy caused by low-intensity local area positioning of facial muscle movement. The experiment results demonstrated that the proposed C3DBed model achieved competitive performance with accuracy rates of 78.04%, 77.64%, and 75.73% on SMIC, CASME II, and SAMM datasets, respectively.

Predicting Vasovagal Reactions to Needles from Facial Action Units

Merely the sight of needles can cause extreme emotional and physical (vasovagal) reactions (VVRs). However, needle fear and VVRs are not easy to measure nor prevent as they are automatic and difficult to self-report. This study aims to investigate whether a blood donors' unconscious facial microexpressions in the waiting room, prior to actual blood donation, can be used to predict who will experience a VVR later, during the donation. The presence and intensity of 17 facial action units were extracted from video recordings of 227 blood donors and were used to classify low and high VVR levels using machine-learning algorithms. We included three groups of blood donors as follows: (1) a control group, who had never experienced a VVR in the past (n = 81); (2) a 'sensitive' group, who experienced a VVR at their last donation (n = 51); and (3) new donors, who are at increased risk of experiencing a VVR (n = 95). The model performed very well, with an F1 (=the weighted average of precision and recall) score of 0.82. The most predictive feature was the intensity of facial action units in the eye regions. To our knowledge, this study is the first to demonstrate that it is possible to predict who will experience a vasovagal response during blood donation through facial microexpression analyses prior to donation.

Imbalanced regression for intensity series of pain expression from videos by regularizing spatio-temporal face nets

• Addressing limited data with expression intensity labels. • Pushing the pain assessment performance by a large margin. • Proposing to add a center-loss-based regularizer. • Proposing a more sensible evaluation metric . Obtaining accurate patient-reported pain intensity is essential for effective pain management. An automatic pain recognition system can simplify the pain reporting process and reduce the strain on manual efforts. Limited and imbalanced labeled data are available for the research of estimating the intensity of pain based on facial expressions. However, the ability to train deep networks for automated pain assessment is limited by small datasets with imbalanced labels of patient-reported pain levels. Fortunately, fine-tuning from a data-extensive pre-trained domain, such as face verification or recognition, can alleviate this problem to some extent. In this paper, we propose a network which fine-tunes a face verification or recognition network using a regularized regression loss and additional data with pain-intensity labels. The expression intensity regression task can benefit from the rich feature representations trained on a large number of data for face analysis tasks. In order to explore the temporal information between frames, we combine CNN with LSTM to obtain a better prediction result of each frame in videos. A weighted evaluation metric and re-sampling technique are also proposed to address the imbalance issue of different pain levels. The proposed regularized deep regressor is applied to estimate the pain expression intensity and verified on the widely-used UNBC-McMaster Shoulder-Pain dataset and BioVid Heat Pain dataset, achieving the state-of-the-art performance. As pain detection is a form of micro facial expression recognition, we also apply the transferred deep regressor to estimate the intensity of facial action units, obtaining high quality performance.

Facial Expression Manipulation for Personalized Facial Action Estimation

Limited sizes of annotated video databases of spontaneous facial expression, imbalanced action unit labels, and domain shift are three main obstacles in training models to detect facial actions and estimate their intensity. To address these problems, we propose an approach that incorporates facial expression generation for facial action unit intensity estimation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and trains a GAN-based network to synthesize novel images with facial action units of interest. We leverage the synthetic images to achieve two goals: 1) generating AU-balanced databases, and 2) tackling domain shift with personalized networks. To generate a balanced database, we synthesize expressions with varying AU intensities and perform semantic resampling. Our experimental results on FERA17 show that networks trained on synthesized facial expressions outperform those trained on actual facial expressions and surpass current state-of-the-art approaches. To tackle domain shift, we propose personalizing pretrained networks. We generate synthetic expressions of each target subject with varying AU intensity labels and use the person-specific synthetic images to fine-tune pretrained networks. To evaluate performance of the personalized networks, we use DISFA and PAIN databases. Personalized networks, which require only a single image from each target subject to generate synthetic images, achieved significant improvement in generalizing to unseen domains.

Driver State Modeling Through Latent Variable State Space Framework in the Wild

Analyzing the impact of the environment on drivers’ stress level and workload is of high importance for designing human-centered driver-vehicle interaction systems and to ultimately help build a safer driving experience. However, driver’s state, including stress level and workload, are latent variables that cannot be measured on their own and should be estimated through sensor measurements such as psychophysiological measures. We propose using a latent-variable state-space modeling framework for driver state analysis. By using latent-variable state-space models, we model drivers’ workload and stress levels as latent variables estimated through multimodal human sensing data, under the perturbations of the environment in a state-space format and in a holistic manner. Through using a case study of multimodal driving data collected from 11 participants, we first estimate the latent stress level and workload of drivers from their heart rate, gaze measures, and intensity of facial action units. We then show that external contextual elements such as the number of vehicles as a proxy for traffic density and secondary task demands may be associated with changes in driver’s stress levels and workload. We also show that different drivers may be impacted differently by the aforementioned perturbations. We found out that drivers’ latent states at previous timesteps are highly associated with their current states. Additionally, we discuss the utility of state-space models in analyzing the possible lag between the two latent variables of stress level and workload, which might be indicative of information transmission between the different parts of the driver’s psychophysiology in the wild.

Labiodentalization of bilabial stops in spontaneous english smiled speech

The co-occurrence of facial expressions and speech production presents a conflict of opposing muscular activation. Existing English research indicates a pattern of bilabial stops being resolved as labiodentals during smiled speech. Electromyographic data suggests that suppression is imposed on one muscle force to resolve the smiling-lip closure conflict [Liu et al., in ISSP Proceeding (2020), pp. 130–133]. These results have limited generalizability as data was collected from one speaker performing reading tasks under laboratory context. The present study continues the investigation of bilabial productions by investigating bilabial phonemes in spontaneous smiled speech. Natural speech samples from 16 native English speakers were extracted from YouTube interviews and vlogs. Bilabial phonemes in neutral and smiling conditions were analyzed for labiodentalization using facial imaging software. The intensity of Facial Action Units (FAU) “lip corner puller” and “lip tightener” during labial token productions were examined. Results show that labiodentalized variants of bilabial tokens also occurred during natural smiled speech. FAU intensity measurements indicate lower “lip corner puller” intensity during bilabial closures than labiodental ones when smiling. This suggests that smile suppression was observed when lip closure was prioritized, validating earlier laboratory conclusions that selective muscular suppression is utilized in the resolution of conflict between articulators.

Deep Facial Action Unit Recognition and Intensity Estimation from Partially Labelled Data

Research on facial action unit (AU) analysis typically require facial images that are labelled with those action units. While unlabelled facial images abound, labelling those images with action units or intensity is costly and time-consuming. Our approach makes it possible to analyze facial AUs when only some of the images have been labelled. We use many facial images to learn a deep framework that is able to take advantage of the facial representations. A restricted Boltzmann machine uses the available AU annotations to learn the AU label or intensity distribution. We train a support vector machine for AU recognition and a support vector regression for AU intensity estimation by maximizing the log likelihood of the AU mapping functions, taking into account the learned multiple AU distribution for all training data, while simultaneously diminishing errors between the predicted action units and ground-truth action unit occurrence or intensities for all labelled data. We perform experiments on two databases. The results demonstrate the superiority of a deep neural network for learning facial features, as well as the benefit of action unit label or intensity constraints for action unit occurrence recognition or intensity estimation in fully or semi-supervised scenarios.

Systematic Evaluation of Design Choices for Deep Facial Action Coding Across Pose

The performance of automated facial expression coding is improving steadily. Advances in deep learning techniques have been key to this success. While the advantage of modern deep learning techniques is clear, the contribution of critical design choices remains largely unknown, especially for facial action unit occurrence and intensity across pose. Using the The Facial Expression Recognition and Analysis 2017 (FERA 2017) database, which provides a common protocol to evaluate robustness to pose variation, we systematically evaluated design choices in pre-training, feature alignment, model size selection, and optimizer details. Informed by the findings, we developed an architecture that exceeds state-of-the-art on FERA 2017. The architecture achieved a 3.5% increase in F1 score for occurrence detection and a 5.8% increase in Intraclass Correlation (ICC) for intensity estimation. To evaluate the generalizability of the architecture to unseen poses and new dataset domains, we performed experiments across pose in FERA 2017 and across domains in Denver Intensity of Spontaneous Facial Action (DISFA) and the UNBC Pain Archive.

Landmark-Based Facial Feature Construction and Action Unit Intensity Prediction

Human face recognition has been widely used in many fields, including biorobots, driver fatigue monitoring, and polygraph tests. However, the end-to-end models fit by most of the existing algorithms perform poorly in interpretation because complex classifiers are constructed using facial images directly. In addition, in some of the models, dynamic characteristics of subjects as individuals are not fully considered, so dynamic information is not extracted. In order to solve these problems, this paper proposes an action unit intensity prediction model. The three-dimensional coordinates of 68 landmarks of human faces are obtained based on the convolutional experts constrained local model (CE-CLM), which enables the construction of dynamic facial features. Based on the error analysis of the CE-CLM algorithm, dimension reduction of the constructed features is performed by the principal components analysis (PCA). The radial basis function (RBF) neural network is also constructed to train the action unit prediction models. The proposed method is verified by the experiments, and the overall mean square error (MSE) of the proposed method is 0.01826. Lastly, the network construction process is optimized, so that for the same training samples, the models are fitted using fewer iterations. The number of iterations is decreased by 27 on average. In summary, this paper provides a method to rapidly construct action unit (AU) intensity prediction models and constructs automatic AU intensity estimation models for facial images.

Toward Multimodal Modeling of Emotional Expressiveness.

Emotional expressiveness captures the extent to which a person tends to outwardly display their emotions through behavior. Due to the close relationship between emotional expressiveness and behavioral health, as well as the crucial role that it plays in social interaction, the ability to automatically predict emotional expressiveness stands to spur advances in science, medicine, and industry. In this paper, we explore three related research questions. First, how well can emotional expressiveness be predicted from visual, linguistic, and multimodal behavioral signals? Second, how important is each behavioral modality to the prediction of emotional expressiveness? Third, which behavioral signals are reliably related to emotional expressiveness? To answer these questions, we add highly reliable transcripts and human ratings of perceived emotional expressiveness to an existing video database and use this data to train, validate, and test predictive models. Our best model shows promising predictive performance on this dataset (RMSE = 0.65, R 2 = 0.45, r = 0.74). Multimodal models tend to perform best overall, and models trained on the linguistic modality tend to outperform models trained on the visual modality. Finally, examination of our interpretable models' coefficients reveals a number of visual and linguistic behavioral signals-such as facial action unit intensity, overall word count, and use of words related to social processes-that reliably predict emotional expressiveness.

Copula Ordinal Regression Framework for Joint Estimation of Facial Action Unit Intensity

Joint modeling of the intensity of multiple facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for $ordinal$ordinal structure in output variables and their $non$non-$linear$linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.

An Adaptive Bayesian Source Separation Method for Intensity Estimation of Facial AUs

Automated measurement of the intensity of spontaneous facial Action Units (AU) defined by the Facial Action Coding System (FACS) in video sequences is a challenging problem. This paper proposes a person-adaptive methodology for the intensity estimation of spontaneous AUs. We formulate this problem as a source separation problem where we consider the observed AUs as the source signals to be separated from each other and other information given by a sequence of facial images. We first compute an initial estimation of the sources, called observations, using sparse linear regression functions. We then develop and apply a Bayesian source separation method that recruits the prior information of the sources to iteratively improve the initial estimations/observations in an adaptive fashion. Furthermore, our approach adaptively uses some testing information (but not the ground-truth labels) to improve the performance of the approach (i.e., Person-Adaptive model). Our experimental results on DISFA, UNBC-McMaster and FERA2015 databases show that this approach is very promising for automated measurement of the intensity of spontaneous facial AUs.

Facial Action Unit Intensity Detection by Extracting Complimentary Information using Distance Metric Learning

ABSTRACTFacial action unit intensity detection is the key concern of researchers as it gives a much broader information about facial expressions of the individual. In the proposed work, an attempt is done to detect the intensity of the facial action unit by combining geometric deformations and appearance deformations of facial features. Thin plate spline is adopted for extracting geometric deformations, and Gabor filters are adopted for extracting appearance deformations. To combine both the description mentioned above, a metric learning method is used to combine the descriptors in such a way that complimentary information is extracted from them. Moreover, it also maps the features to higher discriminative space. The features are applied to support vector machine for the facial action unit intensity detection. The proposed approach is evaluated on the popularly accepted database: DISFA database and UNBC shoulder pain database. The results are compared with the state-of-the-art approaches to prove the efficacy of the suggested approach.

Multi-Instance Dynamic Ordinal Random Fields for Weakly-supervised Facial Behavior Analysis.

We propose a Multi-Instance-Learning (MIL) approach for weakly-supervised learning problems, where a training set is formed by bags (sets of feature vectors or instances) and only labels at bag-level are provided. Specifically, we consider the Multi-Instance Dynamic-Ordinal-Regression (MI-DOR) setting, where the instance labels are naturally represented as ordinal variables and bags are structured as temporal sequences. To this end, we propose Multi-Instance Dynamic Ordinal Random Fields (MI-DORF). In this framework, we treat instance-labels as temporally-dependent latent variables in an Undirected Graphical Model. Different MIL assumptions are modelled via newly introduced high-order potentials relating bag and instance-labels within the energy function of the model. We also extend our framework to address the Partially-Observed MI-DOR problems, where a subset of instance labels are available during training.We show on the tasks of weakly-supervised facial behavior analysis, Facial Action Unit (DISFA dataset) and Pain (UNBC dataset) Intensity estimation, that the proposed framework outperforms alternative learning approaches. Furthermore, we show that MIDORF can be employed to reduce the data annotation efforts in this context by large-scale.

An efficient approach for facial action unit intensity detection using distance metric learning based on cosine similarity

Emotions of human beings are largely represented by facial expressions. Facial expressions, simple as well as complex, are well decoded by facial action units. Any facial expression can be detected and analyzed if facial action units are decoded well. In the presented work, an attempt has been made to detect facial action unit intensity by mapping the features based on their cosine similarity. Distance metric learning based on cosine similarity maps the data by learning a metric that measures orientation rather than magnitude. The motivation behind using cosine similarity is that change in facial expressions can be better represented by changes in orientation as compared to the magnitude. The features are applied to support vector machine for classification of various intensities of action units. Experimental results on the popularly accepted database such as DISFA database and UNBC McMaster shoulder pain database confirm the efficacy of the proposed approach.