Prototypic Facial Expressions Research Articles

The Influence of Social Context on Adolescents’ use of Prototypical Facial Expressions

Adolescents’ social behaviour varies by social context (e.g., who they are with); they might ‘play it cool’ with peers while displaying vulnerability with caregivers. Yet, the way in which social context affects teenagers’ nonverbal expression is unknown. The current study addresses this knowledge gap by examining how teenagers’ facial expressions differs when discussing emotionally charged topics with a friend vs. with a caregiver. Thirty-one 11 to 16-year-old participants engaged in two separate conversations, with their caregiver and friend, respectively (social context). Within each context, they spoke about emotionally charged topics (happy vs angry) of their choosing (emotional type). Facial analysis software (iMotions) was employed to quantify the prototypicality of angry and happy facial expressions. Two separate within-subject factorial (2x2) ANOVAs were conducted to assess the effect of social context and emotional type on the prototypicality of happy and angry facial expressions. There was a significant main effect of social context on prototypical facial cues of anger (F(1, 30) = 5.45, p = .026) with adolescents exhibiting more prototypical angry expressions with their friends than with their caregivers. Moreover, there was a significant main effect of emotional type on prototypical facial cues of happiness (F(1, 30) = 4.72, p = .038) with adolescents displaying more prototypical happy facial expressions when discussing happy topics compared to angry topics. No other effects were significant. The above main effects were rendered non-significant after accounting for sex assigned at birth and age. As the current study is ongoing, the next steps will be to increase our sample size to boost statistical power and to investigate whether the change in non-verbal encoding we observe across social context relates to teens' socio-emotional functioning. With these results, we can better support teenagers in their emotional expression to foster relationship formation and social success throughout adolescence.

Emotion category-modulated interpretation bias in perceiving ambiguous facial expressions.

In contrast to prototypical facial expressions, we show less perceptual tolerance in perceiving vague expressions by demonstrating an interpretation bias, such as more frequent perception of anger or happiness when categorizing ambiguous expressions of angry and happy faces that are morphed in different proportions and displayed under high- or low-quality conditions. However, it remains unclear whether this interpretation bias is specific to emotion categories or reflects a general negativity versus positivity bias and whether the degree of this bias is affected by the valence or category of two morphed expressions. These questions were examined in two eye-tracking experiments by systematically manipulating expression ambiguity and image quality in fear- and sad-happiness faces (Experiment 1) and by directly comparing anger-, fear-, sadness-, and disgust-happiness expressions (Experiment 2). We found that increasing expression ambiguity and degrading image quality induced a general negativity versus positivity bias in expression categorization. The degree of negativity bias, the associated reaction time and face-viewing gaze allocation were further manipulated by different expression combinations. It seems that although we show a viewing condition-dependent bias in interpreting vague facial expressions that display valence-contradicting expressive cues, it appears that the perception of these ambiguous expressions is guided by a categorical process similar to that involved in perceiving prototypical expressions.

Understanding Human Emotions and Detecting Stress Levels using YOLO

Abstract: The emotion recognition system can detect the emotions of a person through their facial expression. Detected emotions can fall into any of the six main data of emotions can fall into any of the six main data of emotions such as happiness, sadness, fear, surprise, disgust, and anger. It is the process of using datasets in recognizing the emotion detected by the system through image processing. Emotion detection leads to maintain a healthy environment in lives. We will use the deep learning technology to generate models with emotion based facial expression to recognized emotions. The datasets contain a large number of labelled facial images or videos with different emotions and poses, enabling researchers to train and test their models effectively. To detect the emotions we have used the YOLOV3 for implementing the existing system and YOLOV7 is used to implement the proposed system. We know that facial expression is one of the most powerful, natural and universal signals for human to convey their emotions states and intentions. Numerous studies have been conducted on automatic facial expression analysis because of its practical important in sociable robotics, medical treatment, driver fatigue surveillance, and many other human-computer interaction systems. In the field of computer vision and machine learning various facial expression recognition (FER) systems have been explored to encode expression information from facial representation. As early as the 21th Ekman and Frisen defined six basic emotions based on cross-culture study, which indicated that humans perceive certain basic emotions in the same way regardless of culture. These prototypical facial expressions are anger, disgust, fear, happiness, sadness, and surprise. Recently advanced research on neuroscience and psychology argued that the model of six basic emotions are culture-specific and not universal.

An examination of autonomic and facial responses to prototypical facial emotion expressions in psychopathy

An examination of autonomic and facial responses to prototypical facial emotion expressions in psychopathy

Training machine learning algorithms for automatic facial coding: The role of emotional facial expressions' prototypicality.

Automatic facial coding (AFC) is a promising new research tool to efficiently analyze emotional facial expressions. AFC is based on machine learning procedures to infer emotion categorization from facial movements (i.e., Action Units). State-of-the-art AFC accurately classifies intense and prototypical facial expressions, whereas it is less accurate for non-prototypical and less intense facial expressions. A potential reason might be that AFC is typically trained with standardized and prototypical facial expression inventories. Because AFC would be useful to analyze less prototypical research material as well, we set out to determine the role of prototypicality in the training material. We trained established machine learning algorithms either with standardized expressions from widely used research inventories or with unstandardized emotional facial expressions obtained in a typical laboratory setting and tested them on identical or cross-over material. All machine learning models' accuracies were comparable when trained and tested with held-out dataset from the same dataset (acc. = [83.4% to 92.5%]). Strikingly, we found a substantial drop in accuracies for models trained with the highly prototypical standardized dataset when tested in the unstandardized dataset (acc. = [52.8%; 69.8%]). However, when they were trained with unstandardized expressions and tested with standardized datasets, accuracies held up (acc. = [82.7%; 92.5%]). These findings demonstrate a strong impact of the training material's prototypicality on AFC's ability to classify emotional faces. Because AFC would be useful for analyzing emotional facial expressions in research or even naturalistic scenarios, future developments should include more naturalistic facial expressions for training. This approach will improve the generalizability of AFC to encode more naturalistic facial expressions and increase robustness for future applications of this promising technology.

Perceived emotions and AU combinations in ambiguous facial expressions

• Network analysis depicts the emotion-network and the surprise is the most repulsive to other emotions. • AU25 and related mouth-open actions are the center of the AU-network. • Matrix calculation found that AU25 weights first in contributing to perceived emotion. • AU4 and AU25 are most frequently presented in the common ambiguous facial expressions Ambiguous facial expressions are common and cannot be classified as specifically prototypical or compound in nature. We have very little understanding of the relationships connecting such expressions to perceived human emotions, such as in terms of compatibility or repulsion. This ignorance also exists with regards to their relationship to action units (AUs). This research employed network analysis to depict the network of perceived emotions and AUs in nearly 5,000 facial expressions obtained from the RAF-AU database, and calculated the centrality indices. We then used a matrix calculation to analyze the relationships between AU combinations and perceived emotions to better understand how people interpret the actions appearing on faces. The results showed that: (1) surprise was the most repulsive to other emotions in the emotion network, (2) AU25 and related open-mouth actions comprised the center of the AU network, (3) AU25 was weighted first in terms of contributing to perceived emotion, and (4) AU4 and AU25 were the most frequently presented in common ambiguous facial expressions. The results were not consistent with those of previous research, mainly due to differences in research methods and materials. The results imply that emotions perceived from ambiguous facial expressions cannot be predicted by core AUs of prototypical facial expressions. The implications and limitations of these conclusions are also discussed herein.

An examination of autonomic and facial responses to prototypical facial emotion expressions in psychopathy.

Meta-analyses have found that people high in psychopathy categorize (or "recognize") others' prototypical facial emotion expressions with reduced accuracy. However, these have been contested with remaining questions regarding the strength, specificity, and mechanisms of this ability in psychopathy. In addition, few studies have tested holistically whether psychopathy is related to reduced facial mimicry or autonomic arousal in response to others' dynamic facial expressions. Therefore, the current study presented 6 s videos of a target person making prototypical emotion expressions (anger, fear, disgust, sadness, joy, and neutral) to N = 88 incarcerated adult males while recording facial electromyography, skin conductance response (SCR), and heart rate. Participants identified the emotion category and rated the valence and intensity of the target person's emotion. Psychopathy was assessed via the Psychopathy Checklist-Revised (PCL-R). We predicted that overall PCL-R scores and scores for the interpersonal/affective traits, in particular, would be related to reduced emotion categorization accuracy, valence ratings, intensity ratings, facial mimicry, SCR amplitude, and cardiac deceleration in response to the prototypical facial emotion expressions. In contrast to our hypotheses, PCL-R scores were unrelated to emotion categorization accuracy, valence ratings, and intensity ratings. Stimuli failed to elicit facial mimicry from the full sample, which does not allow drawing conclusions about the relationship between psychopathy and facial mimicry. However, participants displayed general autonomic arousal responses, but not to prototypical emotion expressions per se. PCL-R scores were also unrelated to SCR and cardiac deceleration. These findings failed to identify aberrant behavioral and physiological responses to prototypical facial emotion expressions in relation to psychopathy.

Reading Your Emotions in My Physiology? Reliable Emotion Interpretations in Absence of a Robust Physiological Resonance.

Affective states are expressed in an individual’s physical appearance, ranging from facial expressions and body postures, to indicators of physiological arousal (e.g., a blush). Confirming the claimed communicative function of these markers, humans are capable of distinguishing between a variety of discrete emotion displays. In an attempt to explain the underlying mechanism, characteristic bodily changes within the observer, including physiological arousal and mimicry, have been suggested to facilitate the interpretation of an expression. The current study aims to create a holistic picture of emotion perception by (1) using three different sources of emotional information (prototypical facial expressions, bodily expressions, and subtle facial cues) and (2) measuring changes in multiple physiological signals (facial electromyography, skin conductance level, skin temperature, and pupil size). While participants clearly discriminated between perceived emotional expressions, there was no overall 1–1 correspondence with their physiological responses. Some specific but robust effects were observed. Angry facial expressions were consistently responded to with a peak in skin conductance level. Furthermore, sad body expressions were associated with a drop in skin temperature. In addition to being the best recognized expression, viewing happy faces elicited congruent facial muscle responses, which supports the potential role of embodied simulation in emotion recognition. Lastly, tears were not only rated as highly emotional intense but also evoked a peak in skin conductance level in the observer. The absence of distinct physiological responses to other expressions could be explained by the lacking functionality of affect sharing in a non-interactive experimental context. Consequentially, emotional alignment in body and mind might especially take place in real social situations, which should be considered in future research.Supplementary InformationThe online version contains supplementary material available at 10.1007/s42761-021-00083-5.

An Android for Emotional Interaction: Spatiotemporal Validation of Its Facial Expressions.

Android robots capable of emotional interactions with humans have considerable potential for application to research. While several studies developed androids that can exhibit human-like emotional facial expressions, few have empirically validated androids’ facial expressions. To investigate this issue, we developed an android head called Nikola based on human psychology and conducted three studies to test the validity of its facial expressions. In Study 1, Nikola produced single facial actions, which were evaluated in accordance with the Facial Action Coding System. The results showed that 17 action units were appropriately produced. In Study 2, Nikola produced the prototypical facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and naïve participants labeled photographs of the expressions. The recognition accuracy of all emotions was higher than chance level. In Study 3, Nikola produced dynamic facial expressions for six basic emotions at four different speeds, and naïve participants evaluated the naturalness of the speed of each expression. The effect of speed differed across emotions, as in previous studies of human expressions. These data validate the spatial and temporal patterns of Nikola’s emotional facial expressions, and suggest that it may be useful for future psychological studies and real-life applications.

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions.

Emotional facial expressions can inform researchers about an individual's emotional state. Recent technological advances open up new avenues to automatic Facial Expression Recognition (FER). Based on machine learning, such technology can tremendously increase the amount of processed data. FER is now easily accessible and has been validated for the classification of standardized prototypical facial expressions. However, applicability to more naturalistic facial expressions still remains uncertain. Hence, we test and compare performance of three different FER systems (Azure Face API, Microsoft; Face++, Megvii Technology; FaceReader, Noldus Information Technology) with human emotion recognition (A) for standardized posed facial expressions (from prototypical inventories) and (B) for non-standardized acted facial expressions (extracted from emotional movie scenes). For the standardized images, all three systems classify basic emotions accurately (FaceReader is most accurate) and they are mostly on par with human raters. For the non-standardized stimuli, performance drops remarkably for all three systems, but Azure still performs similarly to humans. In addition, all systems and humans alike tend to misclassify some of the non-standardized emotional facial expressions as neutral. In sum, emotion recognition by automated facial expression recognition can be an attractive alternative to human emotion recognition for standardized and non-standardized emotional facial expressions. However, we also found limitations in accuracy for specific facial expressions; clearly there is need for thorough empirical evaluation to guide future developments in computer vision of emotional facial expressions.

Facial Expression Ambiguity and Face Image Quality Affect Differently on Expression Interpretation Bias.

We often show an invariant or comparable recognition performance for perceiving prototypical facial expressions, such as happiness and anger, under different viewing settings. However, it is unclear to what extent the categorisation of ambiguous expressions and associated interpretation bias are invariant in degraded viewing conditions. In this exploratory eye-tracking study, we systematically manipulated both facial expression ambiguity (via morphing happy and angry expressions in different proportions) and face image clarity/quality (via manipulating image resolution) to measure participants' expression categorisation performance, perceived expression intensity, and associated face-viewing gaze distribution. Our analysis revealed that increasing facial expression ambiguity and decreasing face image quality induced the opposite direction of expression interpretation bias (negativity vs. positivity bias, or increased anger vs. increased happiness categorisation), the same direction of deterioration impact on rating expression intensity, and qualitatively different influence on face-viewing gaze allocation (decreased gaze at eyes but increased gaze at mouth vs. stronger central fixation bias). These novel findings suggest that in comparison with prototypical facial expressions, our visual system has less perceptual tolerance in processing ambiguous expressions which are subject to viewing condition-dependent interpretation bias.

Fine-grained facial expression analysis using dimensional emotion model

Automated facial expression analysis has a variety of applications in human-computer interaction. Traditional methods mainly analyze prototypical facial expressions of no more than eight discrete emotions as a classification task. However, in practice, spontaneous facial expressions in a naturalistic environment can represent not only a wide range of emotions, but also different intensities within an emotion family. In such situations, these methods are not reliable or adequate. In this paper, we propose to train deep convolutional neural networks (CNNs) to analyze facial expressions explainable in a dimensional emotion model. The proposed method accommodates not only a set of basic emotion expressions, but also a full range of other emotions and subtle emotion intensities that we both feel in ourselves and perceive in others in our daily life. Specifically, we first mapped facial expressions into dimensional measures so that we transformed facial expression analysis from a classification problem to a regression one. We then tested our CNN-based methods for facial expression regression and these methods demonstrated promising performance. Moreover, we improved our method by a bilinear pooling which encodes second-order statistics of features. We showed that such bilinear-CNN models significantly outperformed their respective baseline models.

Hemispheric asymmetries in categorical facial expression perception.

Although many sensory phenomena vary continuously, humans tend to divide them into discrete categories with facial expressions being divided into categories such as happy, sad, anger, and fear. A critical effect of category use is categorical perception: same sized physical differences are better perceived if the difference occurs between two categories rather than within the same category. Here we investigate the lateralization of categorical perception of facial expressions. Categorical perception of facial expressions may be lateralized to the right hemisphere due to the intimacy of basic, possibly universal and innate, prototypical facial expression categories and face processing which is predominantly lateralized to the right hemisphere. Alternatively, facial expression categories may be a linguistic phenomenon and thus predominantly involve the left hemisphere. A visual half-field paradigm was used to assess categorical perception of morphed facial expressions running between happy-sad, and anger-fear. Across two experiments, the lateralization of categorical perception was found to be contingent upon previous exposure. Better between-than within-category discrimination of facial expressions was found in both visual half-fields when participants had previously been exposed to facial expression categories; but for naïve participants, categorical perception was restricted to the right visual hemifield, corresponding to the left cerebral hemisphere. Facial expressions presented to the right hemisphere are therefore not necessarily encoded in terms of their emotional expression category and such category-based encoding, when it does occur, may be due to the left hemisphere and so is likely to be subject to biases related to language. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Facial Expressions of Basic Emotions in Japanese Laypeople.

Facial expressions that show emotion play an important role in human social interactions. In previous theoretical studies, researchers have suggested that there are universal, prototypical facial expressions specific to basic emotions. However, the results of some empirical studies that tested the production of emotional facial expressions based on particular scenarios only partially supported the theoretical predictions. In addition, all of the previous studies were conducted in Western cultures. We investigated Japanese laypeople (n = 65) to provide further empirical evidence regarding the production of emotional facial expressions. The participants produced facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise) in specific scenarios. Under the baseline condition, the participants imitated photographs of prototypical facial expressions. The produced facial expressions were automatically coded using FaceReader in terms of the intensities of emotions and facial action units. In contrast to the photograph condition, where all target emotions were shown clearly, the scenario condition elicited the target emotions clearly only for happy and surprised expressions. The photograph and scenario conditions yielded different profiles for the intensities of emotions and facial action units associated with all of the facial expressions tested. These results provide partial support for the theory of universal, prototypical facial expressions for basic emotions but suggest the possibility that the theory may need to be modified based on empirical evidence.

CGAN Based Facial Expression Recognition for Human-Robot Interaction

As an emerging research topic for proximity service (ProSe), automatic emotion recognition enables the machines to understand the emotional changes of human beings which can not only facilitate natural, effective, seamless, and advanced human-robot interaction or human-computer interface but also promote emotional health. Facial expression recognition (FER) is a vital task for emotion recognition. However, significant gap between human and machine exists in FER task. In this paper, we present a conditional generative adversarial network-based approach to alleviate the intra-class variations by individually controlling the facial expressions and learning the generative and discriminative representations simultaneously. The proposed framework consists of a generator G and three discriminators (Di, Da, and Dexp). The generator G transforms any query face image into another prototypic facial expression image with other factors preserved. Armed with action units condition, the generator G pays more attention to information relevant to facial expression. Three loss functions ($L_{I}$ , La, and Lexp) corresponding to the three discriminators (Di, Da, and Dexp) were designed to learn generative and discriminative representations. Moreover, after rendering the generated expression back to its original facial expression, cycle consistency loss is also applied to guarantee the identity and produce more constrained visual representations. Optimized by combining both synthesis and classification loss functions, the learnt representation is explicitly disentangled from other variations such as identity, head pose, and illumination. Qualitative and quantitative experimental results demonstrate the proposed FER system is effective for expression recognition.

Differences in facial affect recognition between non-offending and offending drivers

It is assumed that emotion recognition is a complex process related to prosocial and antisocial behaviour (Marsh & Blair, 2008). The present study focuses on the connection between recognizing emotions and safe/unsafe driving. We studied whether there are differences in response time in facial emotion recognition. Fifty-one non-offenders and 41 offenders completed a Pictures of Facial Affect test (Ekman & Friesen, 1976) wherein photographs of prototypical facial emotional expressions were presented. Results show differences between the groups in response time to all emotions whether answers were correct or incorrect. Data show that non-offenders are faster in recognizing emotions than are offenders. These findings demonstrate that offenders exhibit specific deficits in response time for facial affect expressions.

Children's prototypic facial expressions during emotion-eliciting conversations with their mothers.

Despite theoretical claims that emotions are primarily communicated through prototypic facial expressions, empirical evidence is surprisingly scarce. This study aimed to (a) test whether children produced more components of a prototypic emotional facial expression during situations judged or self-reported to involve the corresponding emotion than situations involving other emotions (termed "intersituational specificity"), (b) test whether children produced more components of the prototypic expression corresponding to a situation's judged or self-reported emotion than components of other emotional expressions (termed "intrasituational specificity"), and (c) examine coherence between children's self-reported emotional experience and observers' judgments of children's emotions. One hundred and 20 children (ages 7-9) were video-recorded during a discussion with their mothers. Emotion ratings were obtained for children in 441 episodes. Children's nonverbal behaviors were judged by observers and coded by FACS-trained researchers. Children's self-reported emotion corresponded significantly to observers' judgments of joy, anger, fear, and sadness but not surprise. Multilevel modeling results revealed that children produced joy facial expressions more in joy episodes than nonjoy episodes (supporting intersituational specificity for joy) and more joy and surprise expressions than other emotional expressions in joy and surprise episodes (supporting intrasituational specificity for joy and surprise). However, children produced anger, fear, and sadness expressions more in noncorresponding episodes and produced these expressions less than other expressions in corresponding episodes. Findings suggest that communication of negative emotion during social interactions-as indexed by agreement between self-report and observer judgments-may rely less on prototypic facial expressions than is often theoretically assumed. (PsycINFO Database Record

Facial expression analysis with AFFDEX and FACET: A validation study.

The goal of this study was to validate AFFDEX and FACET, two algorithms classifying emotions from facial expressions, in iMotions's software suite. In Study 1, pictures of standardized emotional facial expressions from three databases, the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP), the Amsterdam Dynamic Facial Expression Set (ADFES), and the Radboud Faces Database (RaFD), were classified with both modules. Accuracy (Matching Scores) was computed to assess and compare the classification quality. Results show a large variance in accuracy across emotions and databases, with a performance advantage for FACET over AFFDEX. In Study 2, 110 participants' facial expressions were measured while being exposed to emotionally evocative pictures from the International Affective Picture System (IAPS), the Geneva Affective Picture Database (GAPED) and the Radboud Faces Database (RaFD). Accuracy again differed for distinct emotions, and FACET performed better. Overall, iMotions can achieve acceptable accuracy for standardized pictures of prototypical (vs. natural) facial expressions, but performs worse for more natural facial expressions. We discuss potential sources for limited validity and suggest research directions in the broader context of emotion research.

Gently does it: Humans outperform a software classifier in recognizing subtle, nonstereotypical facial expressions.

According to dominant theories of affect, humans innately and universally express a set of emotions using specific configurations of prototypical facial activity. Accordingly, thousands of studies have tested emotion recognition using sets of highly intense and stereotypical facial expressions, yet their incidence in real life is virtually unknown. In fact, a commonplace experience is that emotions are expressed in subtle and nonprototypical forms. Such facial expressions are at the focus of the current study. In Experiment 1, we present the development and validation of a novel stimulus set consisting of dynamic and subtle emotional facial displays conveyed without constraining expressers to using prototypical configurations. Although these subtle expressions were more challenging to recognize than prototypical dynamic expressions, they were still well recognized by human raters, and perhaps most importantly, they were rated as more ecological and naturalistic than the prototypical expressions. In Experiment 2, we examined the characteristics of subtle versus prototypical expressions by subjecting them to a software classifier, which used prototypical basic emotion criteria. Although the software was highly successful at classifying prototypical expressions, it performed very poorly at classifying the subtle expressions. Further validation was obtained from human expert face coders: Subtle stimuli did not contain many of the key facial movements present in prototypical expressions. Together, these findings suggest that emotions may be successfully conveyed to human viewers using subtle nonprototypical expressions. Although classic prototypical facial expressions are well recognized, they appear less naturalistic and may not capture the richness of everyday emotional communication. (PsycINFO Database Record

Supplemental Material for Children’s Prototypic Facial Expressions During Emotion-Eliciting Conversations With Their Mothers

Supplemental Material for Children’s Prototypic Facial Expressions During Emotion-Eliciting Conversations With Their Mothers