Natural Facial Expressions Research Articles

Realistic Speech-Driven Facial Animation with GANs

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

Copula Ordinal Regression Framework for Joint Estimation of Facial Action Unit Intensity

Joint modeling of the intensity of multiple facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for $ordinal$ordinal structure in output variables and their $non$non-$linear$linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.

Facial expression recognition as a candidate marker for autism spectrum disorder: how frequent and severe are deficits?

BackgroundImpairments in social communication are a core feature of Autism Spectrum Disorder (ASD). Because the ability to infer other people’s emotions from their facial expressions is critical for many aspects of social communication, deficits in expression recognition are a plausible candidate marker for ASD. However, previous studies on facial expression recognition produced mixed results, which may be due to differences in the sensitivity of the many tests used and/or the heterogeneity among individuals with ASD. To ascertain whether expression recognition may serve as a diagnostic marker (which distinguishes people with ASD from a comparison group) or a stratification marker (which helps to divide ASD into more homogeneous subgroups), a crucial first step is to move beyond identification of mean group differences and to better understand the frequency and severity of impairments.MethodsThis study tested 46 individuals with ASD and 52 age- and IQ-matched typically developing (TD) participants on the Films Expression Task, which combines three key features of real-life expression recognition: naturalistic facial expressions, a broad range of simple and complex emotions, and short presentation time. Test-retest reliability was assessed in 28 individuals who did not participate in the main study and revealed acceptable reliability (ICC r = .74).ResultsCase-control comparisons showed highly significant mean group differences for accuracy (p = 1.1 × 10− 10), with an effect size (Cohen’s d = 1.6), more than twice as large as the mean effect size reported in a previous meta-analysis (Uljarevic and Hamilton, 2012, J Autism Dev Disord). The ASD group also had significantly increased mean reaction times overall (p = .00015, d = .83) and on correct trials (p = .0002, d = .78). However, whereas 63% of people with ASD showed severe deficits (they performed below two standard deviations of the TD mean, a small subgroup (15.3%) performed normally (within one standard deviation of the mean).ConclusionThese findings indicate that the majority of people with ASD have severe expression recognition deficits and that the Films Expression Test is a sensitive task for biomarker research in ASD. Future work is needed to establish whether ASD subgroups with and without expression recognition deficits differ from one another in terms of their symptom profile or neurobiological underpinnings.

Facial expression analysis with AFFDEX and FACET: A validation study.

The goal of this study was to validate AFFDEX and FACET, two algorithms classifying emotions from facial expressions, in iMotions's software suite. In Study 1, pictures of standardized emotional facial expressions from three databases, the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP), the Amsterdam Dynamic Facial Expression Set (ADFES), and the Radboud Faces Database (RaFD), were classified with both modules. Accuracy (Matching Scores) was computed to assess and compare the classification quality. Results show a large variance in accuracy across emotions and databases, with a performance advantage for FACET over AFFDEX. In Study 2, 110 participants' facial expressions were measured while being exposed to emotionally evocative pictures from the International Affective Picture System (IAPS), the Geneva Affective Picture Database (GAPED) and the Radboud Faces Database (RaFD). Accuracy again differed for distinct emotions, and FACET performed better. Overall, iMotions can achieve acceptable accuracy for standardized pictures of prototypical (vs. natural) facial expressions, but performs worse for more natural facial expressions. We discuss potential sources for limited validity and suggest research directions in the broader context of emotion research.

Seven- to 11-Year-Olds' Developing Ability to Recognize Natural Facial Expressions of Basic Emotions.

Being able to recognize facial expressions of basic emotions is of great importance to social development. However, we still know surprisingly little about children's developing ability to interpret emotions that are expressed dynamically, naturally, and subtly, despite real-life expressions having such appearance in the vast majority of cases. The current research employs a new technique of capturing dynamic, subtly expressed natural emotional displays (happy, sad, angry, shocked, and disgusted). Children aged 7, 9, and 11 years (and adults) were systematically able to discriminate each emotional display from alternatives in a five-way choice. Children were most accurate in identifying the expression of happiness and were also relatively accurate in identifying the expression of sadness; they were far less accurate than adults in identifying shocked and disgusted. Children who performed well academically also tended to be the most accurate in recognizing expressions, and this relationship maintained independently of chronological age. Generally, the findings testify to a well-developed ability to recognize very subtle naturally occurring expressions of emotions.

An investigation on Facial Expressions using Computational Intelligence techniques and Wavelets

Face to face individual communication utilizes a range of techniques including facial signals and expressions. It would be advantageous to make use of extra natural communication means in human computer interaction and ultimately computer supported human communication. The automatic identification of natural facial expressions is an essential pace towards this goal. The research was done using four different approaches with input sets from various image databases to achieve this goal. Among the four approaches, it is found that the fourth approach which uses Gabor wavelet transforms and Gabor filters yields a better performance of 96.73% with five expressions and 94.8% with all the six natural expressions.

3D Character Animation: A Brief Review

The Augmented Reality (AR) and Virtual Reality (VR) tech -nologies provide many contents that we can barely experience in reality. Further, the developments in performance of comput-er and GPU are based on putting virtual environment into our reality efficiently. The core technologies in developing contents of AR and VR are 3D character modeling and 3D character ani -mation. These are used in various areas such as 3D film, game industry, Physics, Medical simulation and Educational contents. In this review, we discuss the method of animating contents in the virtual reality. There are two topics of 3D character animation, namely, Fa-cial animation and Body animation. Facial animation implies the representation of natural facial expressions. There are two methods of producing this type of animation. One is the Geom-etry-based approach which is based on control point in 3D char-acter face mesh and the other is Blend shape model-based ap-proach which utilizes 3D facial expression template blending. The tree-structured skeleton is usually used for animating body motion. In addition, to achieve more realistic character move-ments, not only physics but also non-rigid deformation of skin or clothes are adopted. In this paper, we discuss Geometry- and Blend shape model-based of facial animation as well as Skeleton- and Physics-based of body animation.

Adaptation to natural dynamic facial emotional expressions

Adaptation to natural dynamic facial emotional expressions

Time-Delay Neural Network for Continuous Emotional Dimension Prediction From Facial Expression Sequences.

Automatic continuous affective state prediction from naturalistic facial expression is a very challenging research topic but very important in human-computer interaction. One of the main challenges is modeling the dynamics that characterize naturalistic expressions. In this paper, a novel two-stage automatic system is proposed to continuously predict affective dimension values from facial expression videos. In the first stage, traditional regression methods are used to classify each individual video frame, while in the second stage, a time-delay neural network (TDNN) is proposed to model the temporal relationships between consecutive predictions. The two-stage approach separates the emotional state dynamics modeling from an individual emotional state prediction step based on input features. In doing so, the temporal information used by the TDNN is not biased by the high variability between features of consecutive frames and allows the network to more easily exploit the slow changing dynamics between emotional states. The system was fully tested and evaluated on three different facial expression video datasets. Our experimental results demonstrate that the use of a two-stage approach combined with the TDNN to take into account previously classified frames significantly improves the overall performance of continuous emotional state estimation in naturalistic facial expressions. The proposed approach has won the affect recognition sub-challenge of the Third International Audio/Visual Emotion Recognition Challenge.

Revealing Variations in Perception of Mental States from Dynamic Facial Expressions: A Cautionary Note

Although a great deal of research has been conducted on the recognition of basic facial emotions (e.g., anger, happiness, sadness), much less research has been carried out on the more subtle facial expressions of an individual's mental state (e.g., anxiety, disinterest, relief). Of particular concern is that these mental state expressions provide a crucial source of communication in everyday life but little is known about the accuracy with which natural dynamic facial expressions of mental states are identified and, in particular, the variability in mental state perception that is produced. Here we report the findings of two studies that investigated the accuracy and variability with which dynamic facial expressions of mental states were identified by participants. Both studies used stimuli carefully constructed using procedures adopted in previous research, and free-report (Study 1) and forced-choice (Study 2) measures of response accuracy and variability. The findings of both studies showed levels of response accuracy that were accompanied by substantial variation in the labels assigned by observers to each mental state. Thus, when mental states are identified from facial expressions in experiments, the identities attached to these expressions appear to vary considerably across individuals. This variability raises important issues for understanding the identification of mental states in everyday situations and for the use of responses in facial expression research.

Board 516 - Technology Innovations Abstract A Novel Method for Synthesizing Naturalistic Pain on Virtual Patients (Submission #1072)

Introduction/Background Since nearly all areas of clinical care involve face-to-face interaction, it is important for learners to be able to recognize pain in patients. While patient self-report (i.e. pain scales) can be a helpful tool in the clinical toolbox, in most cases they must rely on a keen level of sensory observation skills.1 In fact, in some cases the pain-rating scale may not be a reliable tool due to variability across patients, as well as across clinicians’ own interpretation of the scale.2 Thus, there is a desire within the clinical education community to help hone the pain observation skills of learners. However, few training tools exist. To help bridge this gap, we have developed technology that may one day become part of an educational tool. We have designed a method for expressing pain on virtual patients using a naturalistic database of people expressing pain. Methods We extracted source videos from the UMBC-McMaster Pain Archive, a fully labeled, naturalistic data set of 200 video sequences from 25 participants suffering from shoulder pain. The participants performed range-of-motion tests on both their affected and unaffected limbs under the instruction of a physiotherapist. Videos were labeled by facial expression recognition experts, with a pain score from zero to twelve.3 We included all source videos with a pain score greater than three. For each video, we used a CLM-based facial-feature tracker4 to track 68 points on the face. Each facial expression is comprised into certain facial segments called action units (AUs),5 and we mapped these facial movements to a virtual patient in real time using a technique called performance-driven animation. Thus, the virtual patient’s expression directly matched those of a real person in the source video. We used several avatars in the Steam Source SDK6 as our virtual patient representations. In order to validate people’s ability to identify pain on virtual avatars using our method, we conducted an online study involving 50 participants recognizing expressions of pain on virtual avatars (34 female, 16 male, mean age = 38.6). To test the flexibility of our system, we also created three avatars to see if there was any significant effect of gender on participants’ perception of pain (male, female, and androgynous). Results: Conclusion We found that people are able to accurately identify naturalistic facial expressions of pain when expressed by a virtual patient using performance-driven pain synthesis (overall pain accuracy = 67.33%). We also found no significant difference in participants’ accuracy in identifying pain across our three virtual avatar types (all p > .05, with accuracy’s of 66.67%, 65.33%, and 70% for female, male, and androgynous avatars respectively). Our Results are encouraging, suggesting that virtual patients expressing pain using a performance-driven animation method may be useful as part of a general training tool for learners. However, one limitation of this study was it was conducted on the general population, who may interpret pain differently than clinicians.7 Additional work is needed to explore this difference further, and we intend to conduct a study with clinical students in the future.

Robot Assisted Physiotherapy to Support Rehabilitation of Facial Paralysis

We have been developing the Robot Mask with shape memory alloy based actuators that follows an approach of manipulating the skin through a minimally obtrusive wires, transparent strips and tapes based pulling mechanism to enhance the expressiveness of the face. For achieving natural looking facial expressions by taking the advantage of specific characteristics of the skin, the Robot Mask follows a human anatomy based criteria in selecting these manipulation points and directions. In this paper, we describe a case study of using the Robot Mask to assist physiotherapy of a hemifacial paralyzed patient. The significant differences in shape and size of the human head between different individuals demands proper customizations of the Robot Mask. This paper briefly describes the adjusting and customizing stages employed from the design level to the implementation level of the Robot Mask. We will also introduce a depth image sensor data based analysis, which can remotely evaluate dynamic characteristics of facial expressions in a continuous manner. We then investigate the effectiveness of the Robot Mask by analyzing the range sensor data. From the case study, we found that the Robot Mask could automate the physiotherapy tasks of rehabilitation of facial paralysis. We also verify that, while providing quick responses, the Robot Mask can reduce the asymmetry of a smiling face and manipulate the facial skin to formations similar to natural facial expressions.

Humans Process Dog and Human Facial Affect in Similar Ways

Humans share aspects of their facial affect with other species such as dogs. Here we asked whether untrained human observers with and without dog experience are sensitive to these aspects and recognize dog affect with better-than-chance accuracy. Additionally, we explored similarities in the way observers process dog and human expressions. The stimulus material comprised naturalistic facial expressions of pet dogs and human infants obtained through positive (i.e., play) and negative (i.e., social isolation) provocation. Affect recognition was assessed explicitly in a rating task using full face images and images cropped to reveal the eye region only. Additionally, affect recognition was assessed implicitly in a lexical decision task using full faces as primes and emotional words and pseudowords as targets. We found that untrained human observers rated full face dog expressions from the positive and negative condition more accurately than would be expected by chance. Although dog experience was unnecessary for this effect, it significantly facilitated performance. Additionally, we observed a range of similarities between human and dog face processing. First, the facial expressions of both species facilitated lexical decisions to affectively congruous target words suggesting that their processing was equally automatic. Second, both dog and human negative expressions were recognized from both full and cropped faces. Third, female observers were more sensitive to affective information than were male observers and this difference was comparable for dog and human expressions. Together, these results extend existing work on cross-species similarities in facial emotions and provide evidence that these similarities are naturally exploited when humans interact with dogs.

The efficiency of dynamic and static facial expression recognition

Unlike frozen snapshots of facial expressions that we often see in photographs, natural facial expressions are dynamic events that unfold in a particular fashion over time. But how important are the temporal properties of expressions for our ability to reliably extract information about a person's emotional state? We addressed this question experimentally by gauging human performance in recognizing facial expressions with varying temporal properties relative to that of a statistically optimal ("ideal") observer. We found that people recognized emotions just as efficiently when viewing them as naturally evolving dynamic events, temporally reversed events, temporally randomized events, or single images frozen in time. Our results suggest that the dynamic properties of human facial movements may play a surprisingly small role in people's ability to infer the emotional states of others from their facial expressions.

A two-dimensional facial-affect estimation system for human–robot interaction using facial expression parameters

A social robot should be able to autonomously interpret human affect and adapt its behavior accordingly in order for successful social human–robot interaction to take place. This paper presents a modular non-contact automated affect-estimation system that employs support vector regression over a set of novel facial expression parameters to estimate a person’s affective states using a valence-arousal two-dimensional model of affect. The proposed system captures complex and ambiguous emotions that are prevalent in real-world scenarios by utilizing a continuous two-dimensional model, rather than a traditional discrete categorical model for affect. As the goal is to incorporate this recognition system in robots, real-time estimation of spontaneous natural facial expressions in response to environmental and interactive stimuli is an objective. The proposed system can be combined with affect detection techniques using other modes, such as speech, body language and/or physiological signals, etc., in order to develop an accurate multi-modal affect estimation system for social HRI applications. Experiments presented herein demonstrate the system’s ability to successfully estimate the affect of a diverse group of unknown individuals exhibiting spontaneous natural facial expressions.

Holistic Gaze Strategy to Categorize Facial Expression of Varying Intensities

Using faces representing exaggerated emotional expressions, recent behaviour and eye-tracking studies have suggested a dominant role of individual facial features in transmitting diagnostic cues for decoding facial expressions. Considering that in everyday life we frequently view low-intensity expressive faces in which local facial cues are more ambiguous, we probably need to combine expressive cues from more than one facial feature to reliably decode naturalistic facial affects. In this study we applied a morphing technique to systematically vary intensities of six basic facial expressions of emotion, and employed a self-paced expression categorization task to measure participants' categorization performance and associated gaze patterns. The analysis of pooled data from all expressions showed that increasing expression intensity would improve categorization accuracy, shorten reaction time and reduce number of fixations directed at faces. The proportion of fixations and viewing time directed at internal facial features (eyes, nose and mouth region), however, was not affected by varying levels of intensity. Further comparison between individual facial expressions revealed that although proportional gaze allocation at individual facial features was quantitatively modulated by the viewed expressions, the overall gaze distribution in face viewing was qualitatively similar across different facial expressions and different intensities. It seems that we adopt a holistic viewing strategy to extract expressive cues from all internal facial features in processing of naturalistic facial expressions.

근육 모델 기반 3D 얼굴 표정 생성 시스템 설계 및 구현

Facial expression has its significance in mutual communication. It is the only means to express human`s countless inner feelings better than the diverse languages human use. This paper suggests muscle model-based 3D facial expression generation system to produce easy and natural facial expressions. Based on Waters` muscle model, it adds and used necessary muscles to produce natural facial expressions. Also, among the complex elements to produce expressions, it focuses on core, feature elements of a face such as eyebrows, eyes, nose, mouth, and cheeks and uses facial muscles and muscle vectors to do the grouping of facial muscles connected anatomically. By simplifying and reconstructing AU, the basic nuit of facial expression changes, it generates easy and natural facial expressions.

The MPI facial expression database--a validated database of emotional and conversational facial expressions.

The ability to communicate is one of the core aspects of human life. For this, we use not only verbal but also nonverbal signals of remarkable complexity. Among the latter, facial expressions belong to the most important information channels. Despite the large variety of facial expressions we use in daily life, research on facial expressions has so far mostly focused on the emotional aspect. Consequently, most databases of facial expressions available to the research community also include only emotional expressions, neglecting the largely unexplored aspect of conversational expressions. To fill this gap, we present the MPI facial expression database, which contains a large variety of natural emotional and conversational expressions. The database contains 55 different facial expressions performed by 19 German participants. Expressions were elicited with the help of a method-acting protocol, which guarantees both well-defined and natural facial expressions. The method-acting protocol was based on every-day scenarios, which are used to define the necessary context information for each expression. All facial expressions are available in three repetitions, in two intensities, as well as from three different camera angles. A detailed frame annotation is provided, from which a dynamic and a static version of the database have been created. In addition to describing the database in detail, we also present the results of an experiment with two conditions that serve to validate the context scenarios as well as the naturalness and recognizability of the video sequences. Our results provide clear evidence that conversational expressions can be recognized surprisingly well from visual information alone. The MPI facial expression database will enable researchers from different research fields (including the perceptual and cognitive sciences, but also affective computing, as well as computer vision) to investigate the processing of a wider range of natural facial expressions.

Action unit classification using active appearance models and conditional random fields

In this paper, we investigate to what extent modern computer vision and machine learning techniques can assist social psychology research by automatically recognizing facial expressions. To this end, we develop a system that automatically recognizes the action units defined in the facial action coding system (FACS). The system uses a sophisticated deformable template, which is known as the active appearance model, to model the appearance of faces. The model is used to identify the location of facial feature points, as well as to extract features from the face that are indicative of the action unit states. The detection of the presence of action units is performed by a time series classification model, the linear-chain conditional random field. We evaluate the performance of our system in experiments on a large data set of videos with posed and natural facial expressions. In the experiments, we compare the action units detected by our approach with annotations made by human FACS annotators. Our results show that the agreement between the system and human FACS annotators is higher than 90% and underlines the potential of modern computer vision and machine learning techniques to social psychology research. We conclude with some suggestions on how systems like ours can play an important role in research on social signals.

Les effets de l’ingestion aiguë d’alcool sur le jugement d’expressions faciales émotionnelles spontanées et dynamiques

This study is interested in the link between acute alcohol consumption and the perception of dynamic and spontaneous emotional facial expressions (EFE). The noxious effects of alcohol on EFE recognition are now well demonstrated ( Attwood et al., 2009a). Studies showed that alcohol drinking impairs the judgment of facial expressions, especially of negative ones (EFE of anger and disgust notably). However, such effects have been observed for the judgment of artificial material (static/posed/morphed/EFE). Yet, everyday EFE are far from the archetypes used in the lab. Therefore, the present study aims at completing previous observations, this time for the identification of dynamic and spontaneous EFE. Dynamic and spontaneous EFE of amusement, interest, irritation, anxiety, and neutral were judged by 63 normal male drinkers. Buck et al.’s (1972) paradigm was adopted since it is recommended for the study of nonverbal behavior in the process of communication ( Buck, 1990; Wagner, 1990; Wagner et al., 1986; Zuckerman et al., 1976). The emotion to be identified is the one self-reported by the EFE sender. On the pretext of tasting drinks, participants were randomly assigned to one out of six experimental conditions. Two conditions are concerned with the nature of the drink (alcohol or non-alcohol drink). Three other conditions relate to the participant's experimentally manipulated belief regarding the nature of the drink (non-alcohol drink, slight alcohol drink, hard alcohol drink). Results confirm the emotion identification impairment due to alcohol usually noted, thus showing that the effects of alcohol are also observed for the perception of spontaneous and natural facial expressions. Indeed, the comparison of participants who drank alcoholic drinks versus those who did not drink alcohol shows that the formers identify less well emotions than the latter. Moreover, results show that beliefs do not play a role in this deficit since participants’ identification errors are not linked to the belief they hold regarding the nature of the drink (with/without alcohol). Finally, the confusion matrix analysis brings to light the existence of recurrent confusions amongst alcoholized participants, confusions that do not appear amongst sober participants. Thus, results show that, to the exception of EFE of amusement, participants who drank alcohol recurrently assess stimuli as displaying anxiety. To conclude, it comes out that acute alcohol drinking alters the capacity to accurately identify spontaneous emotions expressed by faces, which capacity is essential to the smooth unfolding of human interactions.