Facial Cues Research Articles

Face and context integration in emotion inference is limited and variable across categories and individuals

The ability to make nuanced inferences about other people’s emotional states is central to social functioning. While emotion inferences can be sensitive to both facial movements and the situational context that they occur in, relatively little is understood about when these two sources of information are integrated across emotion categories and individuals. In a series of studies, we use one archival and five empirical datasets to demonstrate that people could be integrating, but that emotion inferences are just as well (and sometimes better) captured by knowledge of the situation alone, while isolated facial cues are insufficient. Further, people integrate facial cues more for categories for which they most frequently encounter facial expressions in everyday life (e.g., happiness). People are also moderately stable over time in their reliance on situational cues and integration of cues and those who reliably utilize situation cues more also have better situated emotion knowledge. These findings underscore the importance of studying variability in reliance on and integration of cues.

Attentional Bias of Individuals with Social Anxiety towards Facial and Somatic Emotional Cues in a Holistic Manner.

Attentional bias towards threatening information is a crucial factor contributing to the development and persistence of social anxiety. However, the attentional bias towards threat information and the preferential processing pattern of emotional cues in individuals with social anxiety disorder during integrated facial and physical stimuli processing remain unclear. In this study, we employed a dot-probe paradigm to investigate the attentional bias towards integrated emotions (facial-body) among students with high and low levels of social anxiety (Experiment 1). Experiments 2 and 3 examined the attentional bias of socially anxious individuals when faced with conflicting emotional cues from faces or bodies in relation to integrated emotions. The data revealed that participants both high and low levels of social anxiety participants exhibited accelerated orienting and biased attention towards facial-body emotional processing. When there was inconsistency between emotional cues from faces or bodies and integrated emotions, higher levels of social anxiety were associated with increased vigilance towards threatening faces or bodies. These findings underscore that individuals with social anxiety possess an ability to rapidly capture threatening cues during the processing of facial-body emotional stimuli while also demonstrating a tendency to avoid relying solely on facial cues by compensating through bodily cues for emotion perception.

Emotional cues reduce Pavlovian interference in feedback-based go and nogo learning

It is easier to execute a response in the promise of a reward and withhold a response in the promise of a punishment than vice versa, due to a conflict between cue-related Pavlovian and outcome-related instrumental action tendencies in the reverse conditions. This robust learning asymmetry in go and nogo learning is referred to as the Pavlovian bias. Interestingly, it is similar to motivational tendencies reported for affective facial expressions, i.e., facilitation of approach to a smile and withdrawal from a frown. The present study investigated whether and how learning from emotional faces instead of abstract stimuli modulates the Pavlovian bias in reinforcement learning. To this end, 137 healthy adult participants performed an orthogonalized Go/Nogo task that fully decoupled action (go/nogo) and outcome valence (win points/avoid losing points). Three groups of participants were tested with either emotional facial cues whose affective valence was either congruent (CON) or incongruent (INC) to the required instrumental response, or with neutral facial cues (NEU). Relative to NEU, the Pavlovian bias was reduced in both CON and INC, though still present under all learning conditions. Importantly, only for CON, the reduction of the Pavlovian bias effect was adaptive by improving learning performance in one of the conflict conditions. In contrast, the reduction of the Pavlovian bias in INC was completely driven by decreased learning performance in non-conflict conditions. These results suggest a potential role of arousal/salience in Pavlovian-instrumental regulation and cue-action congruency in the adaptability of goal-directed behavior. Implications for clinical application are discussed.

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

In this study, the focus is primarily on the diverse methods for recognizing human emotions through language, text, and facial expressions via computational technology. Emphasizing the real-world applicability of these techniques, the paper underscores the significance of multimodal emotion recognition in areas such as human-computer interaction, psychology, and emotion analytics. Multimodal methods, which combine data from various sources like voice tone, facial cues, and textual context, offer a robust approach for discerning nuanced emotional states. Compared to single-mode analysis, these multimodal techniques tend to produce more accurate and comprehensive results, bridging the gaps left by any one mode in isolation. As technology increasingly integrates with daily human activity, the importance of nuanced, reliable emotion recognition is becoming paramount for fostering more natural and empathic machine-human interactions. Moreover, in the realm of psychology, these methods offer groundbreaking possibilities for diagnosis and treatment. By discussing the future applications and methodologies of multimodal emotion recognition, this paper aims to provide a comprehensive roadmap for both academic research and practical applications in the evolving landscape of emotion-aware computing.

The pupil dilation response as an indicator of visual cue uncertainty and auditory outcome surprise.

In everyday perception, we combine incoming sensory information with prior expectations. Expectations can be induced by cues that indicate the probability of following sensory events. The information provided by cues may differ and hence lead to different levels of uncertainty about which event will follow. In this experiment, we employed pupillometry to investigate whether the pupil dilation response to visual cues varies depending on the level of cue-associated uncertainty about a following auditory outcome. Also, we tested whether the pupil dilation response reflects the amount of surprise about the subsequently presented auditory stimulus. In each trial, participants were presented with a visual cue (face image) which was followed by an auditory outcome (spoken vowel). After the face cue, participants had to indicate by keypress which of three auditory vowels they expected to hear next. We manipulated the cue-associated uncertainty by varying the probabilistic cue-outcome contingencies: One face was most likely followed by one specific vowel (low cue uncertainty), another face was equally likely followed by either of two vowels (intermediate cue uncertainty) and the third face was followed by all three vowels (high cue uncertainty). Our results suggest that pupil dilation in response to task-relevant cues depends on the associated uncertainty, but only for large differences in the cue-associated uncertainty. Additionally, in response to the auditory outcomes, the pupil dilation scaled negatively with the cue-dependent probabilities, likely signalling the amount of surprise.

Effects of chronic intranasal oxytocin on visual attention to faces vs. natural scenes in older adults

Aging is associated with changes in face processing, including desensitization to face cues like gaze direction and an attentional preference to faces with positive over negative emotional valence. A parallel line of research has shown that acute administration of oxytocin (OT) increases visual attention to social stimuli such as human faces. The current study examined effects of chronic OT administration among older adults on fixation duration to faces that varied in emotional expression, gaze direction, age, and sex. One hundred and twelve generally healthy older adults (aged 55–95 years) underwent a randomized, placebo-controlled, double-blind, between-subject clinical trial in which they self-administered either OT or placebo (P) intranasally twice a day for 4 weeks. The behavioral task involved rating the trustworthiness of faces (i.e., social stimuli) and natural scenes (i.e., non-social control stimuli) during eye tracking and was conducted before and after the intervention. Fixation duration to both the faces and the natural scenes declined from pre- to post-intervention, however this decline was less pronounced among older adults in the OT compared to the P group for faces but not scenes. Further, face cues (emotional expression, gaze direction, age, sex) did not moderate the treatment effect. This study provides first evidence that chronic intranasal OT maintains salience of social cues over time in older adults, perhaps buffering effects of habituation. These findings enhance understanding of OT effects on social cognition among older adults, and would benefit from follow up with a young adult comparison group to directly speak to specificity of observed effects to older adults and reflection of the aging process.

Cultural Predictors of Facial Ethnicity Preference in the Miskitu and Mestizos of Rural Nicaragua

Both basic visual experience and cultural associations with race and ethnicity may contribute to the extent observers do or do not favor some facial ethnicity cues over others. Given that visual media contain a highly biased selection of faces, with Whiteness both over-represented and strongly privileged in film and television, communities for whom visual media are relatively novel may experience an additional, pervasive source of attitudes to facial ethnicity markers. In the current research, we compared individuals of Mestizo and Miskitu identities living in communities on the Caribbean Coast of Nicaragua with, and without, regular access to television on their relative preference for facial stimuli manipulated to look more or less White (Black vs White, Black vs Mestizo, Mestizo vs White). Results showed that all communities showed an overall preference for images with lighter skin, although changes in facial shape did not affect preferences. Those who had attended more years of education preferred whiter faces than those with less education, and those who watched more television preferred whiter faces more only where color (rather than shape) had been manipulated. Results are discussed in terms of the broader relations around ethnicity, status, and technological transition in this area.

Heightened response to positive facial cues as a potential marker of resilience following childhood adversity

ABSTRACT Background: Childhood maltreatment profoundly influences social and emotional development, increasing psychiatric risk. Alterations in the implicit processing of threat-related cues following early abuse and neglect represent a marker of mental health vulnerability. Less is known about how early adversity influences the perception of positive social cues, despite their central role in establishing and maintaining social interactions and their association with better mental health outcomes. Methods: The sample consisted of 42 children and adolescents with substantiated childhood maltreatment experiences and 32 peers (mean age 13.3), matched on age, pubertal status, gender, socioeconomic status, ethnicity, and cognitive ability. A computerised experimental task assessed the perceived emotional intensity of positive (happy) and negative (fearful) facial expressions. Mental health symptoms were measured via self- and parental reports, and perceived social support was self-reported. Results: The experience of abuse and neglect was associated with heightened perceived intensity of positive facial cues. Cross-sectional post-hoc moderation and mediation analyses, employing a model-building approach, revealed that in maltreatment-exposed participants: (i) their increased response to positive facial cues was associated with lower symptoms; (ii) the presence of social support accounted for their heightened perceived intensity of positive facial cues; (iii) the presence of social support putatively contributed to lower symptoms by increasing the perceived intensity of positive facial cues. No group differences in perceived intensity of negative expressions were observed. Conclusions: These findings provide fresh insight into how positive faces are processed following maltreatment experience in childhood. Maltreatment experience was found to be associated with heightened perceived intensity of happy faces, which in turn was associated with better mental health and greater levels of social support. This suggests that heightened saliency of positive emotions acts protectively in children with maltreatment experience.

Abstract 123: Emotional Prosody and Facial Expressions Differentially Augment Comprehension of Discourse for Left and Right Hemisphere Stroke Survivors

Deficits in spoken discourse comprehension are common after left hemisphere (LH) and right hemisphere (RH) stroke. Thus, it is important to identify factors that facilitate comprehension and enhance communication between stroke survivors and their care partners. Emotional prosody and facial expression cues can facilitate comprehension, but this may depend on an intact RH. To determine if such cues are facilitatory to LH and RH stroke survivors, we compared discourse comprehension between the groups across 3 conditions. We hypothesized that RH participants would not benefit as much from these cues as LH participants due to concomitant impairments in recognition of prosody and facial expression. We assessed 72 stroke survivors within 3 months of unilateral ischemic stroke, including 40 LH (15 female; aged 62±13 yrs) and 32 RH (17 female; aged 68±10 yrs), and 29 healthy controls (19 female; aged 72±15 yrs). Participants listened to a series of recorded spoken descriptions of scenarios and selected 1 picture from 4 to match each scenario. Targets and foils were balanced across spatial positions. Each description was presented in 3 conditions: read with neutral prosody (Trial 1), read with emotional prosody (Trial 2), and read with emotional prosody + facial expressions (Trial 3, audio + video). If 100% accuracy was achieved on a trial, the next trial was not presented. We used a Kruskal-Wallace test and paired samples t -tests for analyses and corrected for multiple comparisons. Accuracy differed significantly between groups on Trial 1, H (2) = 26.68, p < .001. Controls achieved higher accuracy than LH ( p < .01) and RH participants ( p < .01). LH and RH participants did not differ, p = .51. LH participants achieved significant gains in accuracy from Trial 1 to 2 ( t (32) = 4.16, p = .001) and from Trial 1 to 3 ( t (32) = 5.49, p < .001), but not from Trial 2 to 3 ( t (32) = 1.79, p = .08). RH participants did not achieve significant gains in accuracy from Trial 1 to 2 ( t (23) = 1.59, p = .13), Trial 2 to 3 ( t (23) = 0, p = 1.0), or Trial 1 to 3 ( t (23) = 1.21, p = .24). To conclude, emotional prosody and facial expressions significantly improved discourse comprehension for LH but not for RH stroke survivors. Different/additional strategies to improve discourse comprehension after RH stroke are needed.

Biracial Faces Offer Visual Cues of Successful Intergroup Contact: Genetic Admixture and Coalition Detection.

This research explores how biracial facial cues affect racial perception and social judgment. We tested a coalition-signaling hypothesis of biracial cues in two studies conducted in the United States (n = 227) and China (n = 116). From the perspective of intergroup and interpersonal relations theories in social psychology, biracial features would likely be perceived as cues of threat or resource competition. In contrast, we propose an evolutionary hypothesis that biracial facial cues reveal the ancestral history of intergroup alliances between members of two races or ethnic groups. When racial cues are mixed, we predict that biracial individuals may be viewed more positively than other-race or even own-race members who often compete for limited ingroup resources. The participants observed facial images that ranged from 100% Asian to 100% Caucasian, including morphed biracial composites of 30%, 40%, 50%, 60%, and 70% Caucasian or Asian. The participants evaluated each image regarding perceived Caucasianness (Asianness), attractiveness, trustworthiness, health, intelligence, and career prospects. The US and Chinese samples yielded a similar pattern of own-race bias in racial perception and biracial favoritism in social judgment. The social judgment ratings were not correlated with the racial perception scores and were independent of the sex of the participants or biracial images, indicating a coalitional motive, instead of a mating motive, underlying social perception of biracial individuals. Overall, the results suggest that biracial facial features signal a successful genetic admixture and coalition in parental generations and thus increase the trustworthiness and cooperative potential of a biracial person.

Implicit self-esteem and self-esteem discrepancy predict sensitivity to perceive dynamic facial expressions

The current study examined the relationship between Explicit Self-Esteem (ESE), Implicit Self-Esteem (ISE), and the size of discrepancy between them (SEDS) in relation to sensitivity to emotional facial cues signifying acceptance (happy facial expressions), rejection (angry and disgusted facial expressions), and fear. The results reveal a significant association between ISE and sensitivity to negative emotions, with the SEDS significantly predicting rejection cue sensitivity. Moreover, higher scores in psychological wellbeing were observed in individuals with higher ESE, higher ISE, and smaller ESE-ISE discrepancies. These findings not only support the sociometer theory, but also highlight the complex relationship between self-esteem dimensions, emotional sensitivity, and psychological wellbeing, highlighting the need for a more integrative approach in self-esteem research.

Fusing facial and speech cues for enhanced multimodal emotion recognition

Fusing facial and speech cues for enhanced multimodal emotion recognition

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Joint Audio-Visual Attention with Contrastive Learning for More General Deepfake Detection

With the continuous advancement of deepfake technology, there has been a surge in the creation of realistic fake videos. Unfortunately, the malicious utilization of deepfake poses a significant threat to societal morality and political security. Therefore, numerous researchers have proposed various deepfake detection methods. However, traditional deepfake approaches tend to focus on specific forgery features, such as artifacts or inconsistent actions, which can be vulnerable to specialized countermeasures. Recent studies show an intrinsic correlation between facial and audio cues, which can be exploited for deepfake detection. To address these challenges and enhance the robustness and generalization of deepfake detection algorithms, we propose a novel joint audio-visual deepfake detection model named AVA-CL, which is capable of detecting deepfakes in both audio and visual domains. Furthermore, exploiting the inherent correlation and consistency between audio and visual enhances the effectiveness of deepfake detection significantly. Through extensive experiments, we demonstrate that our proposed AVA-CL model outperforms many state-of-the-art (SOTA) methods with superior robustness and generalization capabilities. This research presents a promising approach for deepfake detection and reducing the harm caused by malicious use.

The Relative Contribution of Facial and Body Information to the Perception of Cuteness.

Faces and bodies both provide cues to age and cuteness, but little work has explored their interaction in cuteness perception. This study examines the interplay of facial and bodily cues in the perception of cuteness, particularly when these cues convey conflicting age information. Participants rated the cuteness of face-body composites that combined either a child or adult face with an age-congruent or incongruent body alongside manipulations of the head-to-body height ratio (HBR). The findings from two experiments indicated that child-like facial features enhanced the perceived cuteness of adult bodies, while child-like bodily features generally had negative impacts. Furthermore, the results showed that an increased head size significantly boosted the perceived cuteness for child faces more than for adult faces. Lastly, the influence of the HBR was more pronounced when the outline of a body's silhouette was the only available information compared to when detailed facial and bodily features were presented. This study suggests that body proportion information, derived from the body's outline, and facial and bodily features, derived from the interior surface, are integrated to form a unitary representation of a whole person in cuteness perception. Our findings highlight the dominance of facial features over bodily information in cuteness perception, with facial attributes serving as key references for evaluating face-body relationships and body proportions. This research offers significant insights into social cognition and character design, particularly in how people perceive entities with mixed features of different social categories, underlining the importance of congruency in perceptual elements.

Perceptual integration of bodily and facial emotion cues in chimpanzees and humans.

For highly visual species like primates, facial and bodily emotion expressions play a crucial role in emotion perception. However, most research focuses on facial expressions, while the perception of bodily cues is still poorly understood. Using a novel comparative priming eye-tracking design, we examined whether our close primate relatives, the chimpanzees (Pan troglodytes), and humans infer emotions from bodily cues through subsequent perceptual integration with facial expressions. In experiment 1, we primed chimpanzees with videos of bodily movements of unfamiliar conspecifics engaged in social activities of opposite valence (play and fear) against neutral control scenes to examine attentional bias toward succeeding congruent or incongruent facial expressions. In experiment 2, we assessed the same attentional bias in humans yet using stimuli showing unfamiliar humans. In experiment 3, humans watched the chimpanzee stimuli of experiment 1, to examine cross-species emotion perception. Chimpanzees exhibited a persistent fear-related attention bias but did not associate bodily with congruent facial cues. In contrast, humans prioritized conspecifics' congruent facial expressions (matching bodily scenes) over incongruent ones (mismatching). Nevertheless, humans exhibited no congruency effect when viewing chimpanzee stimuli, suggesting difficulty in cross-species emotion perception. These results highlight differences in emotion perception, with humans being greatly affected by fearful and playful bodily cues and chimpanzees being strongly drawn toward fearful expressions, regardless of the preceding bodily priming cue. These data advance our understanding of the evolution of emotion signaling and the presence of distinct perceptual patterns in hominids.

Predicting Facial Attractiveness from Colour Cues: A New Analytic Framework.

Various facial colour cues were identified as valid predictors of facial attractiveness, yet the conventional univariate approach has simplified the complex nature of attractiveness judgement for real human faces. Predicting attractiveness from colour cues is difficult due to the high number of candidate variables and their inherent correlations. Using datasets from Chinese subjects, this study proposed a novel analytic framework for modelling attractiveness from various colour characteristics. One hundred images of real human faces were used in experiments and an extensive set of 65 colour features were extracted. Two separate attractiveness evaluation sets of data were collected through psychophysical experiments in the UK and China as training and testing datasets, respectively. Eight multivariate regression strategies were compared for their predictive accuracy and simplicity. The proposed methodology achieved a comprehensive assessment of diverse facial colour features and their role in attractiveness judgements of real faces; improved the predictive accuracy (the best-fit model achieved an out-of-sample accuracy of 0.66 on a 7-point scale) and significantly mitigated the issue of model overfitting; and effectively simplified the model and identified the most important colour features. It can serve as a useful and repeatable analytic tool for future research on facial impression modelling using high-dimensional datasets.

Male facial cues to physical strength in Europe: Medium-strength preference and higher aggressiveness attribution to the weakest.

The capacity to assess male physical strength from facial cues may be adaptive given health and fitness-related associations with muscular strength. Our study complements recent research on strength-related face perceptions of male Maasai by applying the protocol to male European faces and assessors. Five distinct facial morphs calibrated for handgrip strength (HGS) were manufactured with geometric morphometrics performing regressions of the Procrustes shape coordinates on HGS in a sample of 26 European men (18-32 years). Young adult men and women (n = 445) rated these morphs on physical strength, attractiveness, and aggressiveness. Facial morphs calibrated to lower HGS were rated as less strong, less attractive, and more aggressive than those calibrated to higher HGS. Medium levels of HGS were associated with the highest attractiveness ratings. The rating patterns of physical strength, attractiveness, and aggressiveness for European male facial morphs exhibit similarity to previous ratings of Maasai male faces. Therefore, the current findings corroborate the suggestion of a common mechanism for social attributions based on facial cues to physical strength, modulated by local ecology and societal context.

Age-group differences in trust-related decision-making and learning

Facial impressions contribute to evaluations of trustworthiness. Older adults are especially vulnerable to trust violations, incurring risks for deception and exploitation. Using the newly developed social Iowa Gambling Task (S-IGT), we examined age-group differences in the impact of facial trustworthiness on decision-making and learning. In the congruent condition (CS-IGT), advantageous decks were paired with trustworthy faces and disadvantageous decks with untrustworthy faces. In the incongruent condition (IS-IGT), this pairing was reversed. Younger (n = 143) and older (n = 129) participants completed either the standard Iowa Gambling Task (IGT), CS-IGT, or IS-IGT. Both age groups preferred trustworthy faces in their initial choices. Older adults performed worse than younger adults across all tasks over time. Further, compared to younger adults, older adults performed worse on the IS-IGT, suggesting that incongruent facial cues interfered with older adults’ performance, which aligns with reduced sensitivity to negative social reputations in aging. Multilevel modeling also indicated that age-group differences were most pronounced across all tasks in the last 40 trials. Together these findings suggest that differences between younger and older adults in experience-dependent decision-making are magnified in social contexts that involve a “wolf in sheep’s clothing,” which may reflect age-related difficulties in integrating incongruent information.

Men with high dark triad personality traits can accurately infer dark triad traits from other people's faces.

The literature suggests that people can accurately infer dark triad (DT) personality traits from other peoples' faces. Using a self-report scale, this study investigated the impact of participants' DT personality traits on their ability to accurately infer other peoples' DT traits from facial cues. We created composite facial photographs of Japanese people with varying Machiavellianism, psychopathy, and narcissism scores. The Japanese participants (N = 170) assessed these three DT traits in the facial photographs and completed a questionnaire that assessed their own DT traits. The results indicated that the participants could accurately infer all three DT traits from female faces but not from male faces. Male participants showed a positive correlation between accurate inferences of other men's Machiavellianism and their own DT traits. In contrast, female participants showed a negative correlation between correct inferences of men's DT traits and their own DT traits. These findings offer novel insights into human evolutionary and social adaptations.