Facial Cues Research Articles

A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face.

Multimodal emotion recognition (MER) refers to the identification and understanding of human emotional states by combining different signals, including-but not limited to-text, speech, and face cues. MER plays a crucial role in the human-computer interaction (HCI) domain. With the recent progression of deep learning technologies and the increasing availability of multimodal datasets, the MER domain has witnessed considerable development, resulting in numerous significant research breakthroughs. However, a conspicuous absence of thorough and focused reviews on these deep learning-based MER achievements is observed. This survey aims to bridge this gap by providing a comprehensive overview of the recent advancements in MER based on deep learning. For an orderly exposition, this paper first outlines a meticulous analysis of the current multimodal datasets, emphasizing their advantages and constraints. Subsequently, we thoroughly scrutinize diverse methods for multimodal emotional feature extraction, highlighting the merits and demerits of each method. Moreover, we perform an exhaustive analysis of various MER algorithms, with particular focus on the model-agnostic fusion methods (including early fusion, late fusion, and hybrid fusion) and fusion based on intermediate layers of deep models (encompassing simple concatenation fusion, utterance-level interaction fusion, and fine-grained interaction fusion). We assess the strengths and weaknesses of these fusion strategies, providing guidance to researchers to help them select the most suitable techniques for their studies. In summary, this survey aims to provide a thorough and insightful review of the field of deep learning-based MER. It is intended as a valuable guide to aid researchers in furthering the evolution of this dynamic and impactful field.

Neural sensitivity to facial identity and facial expression discrimination in adults with autism.

The fluent processing of faces can be challenging for autistic individuals. Here, we assessed the neural sensitivity to rapid changes in subtle facial cues in 23 autistic men and 23 age and IQ matched non-autistic (NA) controls using frequency-tagging electroencephalography (EEG). In oddball paradigms examining the automatic and implicit discrimination of facial identity and facial expression, base rate images were presented at 6 Hz, periodically interleaved every fifth image with an oddball image (i.e. 1.2 Hz oddball frequency). These distinctive frequency tags for base rate and oddball stimuli allowed direct and objective quantification of the neural discrimination responses. We found no large differences in the neural sensitivity of participants in both groups, not for facial identity discrimination, nor for facial expression discrimination. Both groups also showed a clear face-inversion effect, with reduced brain responses for inverted versus upright faces. Furthermore, sad faces generally elicited significantly lower neural amplitudes than angry, fearful and happy faces. The only minor group difference is the larger involvement of high-level right-hemisphere visual areas in NA men for facial expression processing. These findings are discussed from a developmental perspective, as they strikingly contrast with robust face processing deficits observed in autistic children using identical EEG paradigms.

Representation constraint‐based dual‐channel network for face antispoofing

AbstractAlthough multimodal face data have obvious advantages in describing live and spoofed features, single‐modality face antispoofing technologies are still widely used when it is difficult to obtain multimodal face images or inconvenient to integrate and deploy multimodal sensors. Since the live/spoofed representations in visible light facial images include considerable face identity information interference, existing deep learning‐based face antispoofing models achieve poor performance when only the visible light modality is used. To address the above problems, the authors design a dual‐channel network structure and a constrained representation learning method for face antispoofing. First, they design a dual‐channel attention mechanism‐based grouped convolutional neural network (CNN) to learn important deceptive cues in live and spoofed faces. Second, they design inner contrastive estimation‐based representation constraints for both live and spoofed samples to minimise the sample similarity loss to prevent the CNN from learning more facial appearance information. This increases the distance between live and spoofed faces and enhances the network's ability to identify deceptive cues. The evaluation results indicate that the framework we designed achieves an average classification error rate (ACER) of 2.37% on the visible light modality subset of the CASIA‐SURF dataset and an ACER of 2.4% on the CASIA‐SURF CeFA dataset, outperforming existing methods. The proposed method achieves low ACER scores in cross‐dataset testing, demonstrating its advantage in domain generalisation.

An Honest Joker reveals stereotypical beliefs about the face of deception

Research on deception detection has mainly focused on Simple Deception, in which false information is presented as true. Relatively few studies have examined Sophisticated Deception, in which true information is presented as false. Because Sophisticated Deception incentivizes the appearance of dishonesty, it provides a window onto stereotypical beliefs about cues to deception. Here, we adapted the popular Joker Game to elicit spontaneous facial expressions under Simple Deception, Sophisticated Deception, and Plain Truth conditions, comparing facial behaviors in static, dynamic nonspeaking, and dynamic speaking presentations. Facial behaviors were analysed via machine learning using the Facial Action Coding System. Facial activations were more intense and longer lasting in the Sophisticated Deception condition than in the Simple Deception and Plain Truth conditions. More facial action units intensified in the static condition than in the dynamic speaking condition. Simple Deception involved leaked facial behaviors of which deceivers were unaware. In contrast, Sophisticated Deception involved deliberately leaked facial cues, including stereotypical cues to lying (e.g., gaze aversion). These stereotypes were inaccurate in the sense that they diverged from cues in the Simple Deception condition—the actual appearance of deception in this task. Our findings show that different modes of deception can be distinguished via facial action analysis. They also show that stereotypical beliefs concerning cues to deception can inform behavior. To facilitate future research on these topics, the multimodal stimuli developed in this study are available free for scientific use.

Who do you think they are? An exploratory analysis of face-based impressions formed by cisgender, transgender and gender nonbinary individuals.

Background: Individuals make first impressions of others based on how they look. Facial cues trigger gender social categorisation and elicit gender stereotyping. However, it remains unclear to what extent such impressions are influenced by the perceivers’ gender identity and gender beliefs. Methods: We recruited cisgender and transgender and nonbinary (TGNB) participants (N = 195). Participants were shown three morphed faces (prototypical male versus. prototypical female versus. androgynous) and, for each target, were asked to create a sentence describing their first impression. (within-participants design). To do so, they had to choose from a list of pronouns, professions, and personality traits. Participants also reported how confident they were in their first impression and rated the gender of the targets. Results: Results showed that overall participants perceived the prototypical male target as masculine while both the prototypical female and androgynous targets were seen as feminine. Participants also felt more confident when forming an impression about the prototypical male target. Further, impressions were influenced by perceivers’ gender identity. Cisgender participants formed stronger stereotypical impressions of the prototypical male and female target than TGNB participants. In particular, cisgender participants were more likely to use masculine pronouns for the prototypical male target and more feminine pronouns for the prototypical female target, but less neutral pronouns for the androgynous target, than TGNB participants. The difference is pronoun use was mediated by cisgender participants stronger endorsement of gender binary beliefs. Conclusions: These findings provide new insights into face-based stereotyping and show the importance of considering different stereotype components and individual differences such as gender identity and beliefs.

When men are wonderful: A larger happy face facilitation effect for male (vs. female) faces for male participants.

Facial cues for age, race, and sex influence how we recognize facial expressions. For example, the faster recognition of happy compared to sad expression increases in magnitude when the faces are female compared to male-an effect termed Researchers have argued that presenting expressions of opposite valence (e.g., sad vs. happy expressions) creates an evaluative mindset and consequently, face sex affects emotion recognition via evaluative rather than stereotype associations. For the comparison between anger and happiness, recent results indicate that the effects of face sex are larger for female participants. However, for the critical comparison between sad and happy expressions-used to support the evaluative over the stereotype account-moderation by participant sex has not been adequately examined because the sample size of male participants has been too small. Here, I increased the number of male participants relative to previous studies. For male participants, the usual facilitation effect for female faces was reversed-the happy face facilitation effect was larger for male compared to female faces. The novel pattern for male participants-supporting an in-group bias-was replicated in Study 2, a preregistered study. Finally, ex-Gaussian analyses of the results of Study 1 and Study 2 helped identify differences between the current research and previous studies that had reported participant sex differences. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Maternal depressive symptoms and affective responses to infant crying and laughing.

Depressive symptoms are common in the postpartum period and can affect mother-infant interaction. To better understand the role of depressive symptoms in the mother-infant interchange, this study examined whether maternal depressive symptoms are associated with self-reported, physiological, and facial expressive responses to infant crying and laughing sounds. A nonclinical sample was used, consisting of 101 mothers (Age M = 30.88 years, 33% scored 7 or higher on the Edinburgh Postnatal Depression Scale) with a young child. Mothers were exposed to standard infant crying and laughing sounds. Affect, perception of crying and laughing, intended caregiving responses, skin conductance level reactivity, and facial expressive responses to infant crying and laughing were measured. Higher levels of depressive symptoms were associated with more self-reported negative affect in general and a more negative perception of infant crying. Depressive symptoms were not associated with intended caregiving responses and physiological responses to infant crying. Infant laughing increased self-reported positive affect and happy facial expressions in mothers with all levels of depressive symptoms. Higher levels of depressive symptoms were associated with higher sad facial expressivity in general. Depressive symptoms were not related to positive perception of infant laughing, intended caregiving responses, and physiological responses to infant laughing. The findings suggest that mothers who score high on depressive symptoms send subtle facial cues showing sadness, which may overshadow happy facial expressions during infant laughing and may affect mother-infant interaction. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Emotional modulation of gaze cueing does not depend on a global perceptual processing strategy.

Research on emotional modulation of attention in gaze cueing has resulted in contradictory findings. Some studies found larger gaze cueing effects (GCEs) in response to a fearful gaze cue, whereas others did not. A recent study explained this discrepancy within a cognitive resource account, in which perceptual demands of the task promote a bias toward either a local (discrimination task) or global (localization task) processing strategy. During local processing, the integration of emotional expression with gaze direction is assumed to be impaired, whereas during global processing integration is assumed to be facilitated. In the current study, we investigated the cognitive resource account in three experiments. In Experiment 1, we manipulated task demands by adopting a detection or a localization task whilst both should allow global processing. In Experiments 2 and 3, we induced either a local or global perceptual processing strategy by presenting local or global targets (Experiment 2) or by priming local or global perception prior to the gaze cueing task (Experiment 3). Results showed faster orienting in response to a fearful face cue independent of task demands in Experiment 1. Inducing local and global processing strategies in Experiments 2 and 3 did not affect emotional modulation of the GCE. In contrast, Bayesian analyses provided evidence of absence of such an effect, demonstrating that local or global processing strategies cannot explain the mixed findings obtained in emotional modulation of gaze cueing. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Effects of wearing an opaque or transparent face mask on the perception of facial expressions: A comparative study between Japanese school-aged children and adults.

The negative side effects of mask-wearing on reading facial emotional cues have been investigated in several studies with adults post-2020. However, little is known about children. This study aimed to determine the negative influence of mask-wearing on reading emotions of adult faces by Japanese school-aged children, compared to Japanese adults. We also examined whether this negative influence could be alleviated by using a transparent face mask instead of an opaque one (surgical mask). The performance on reading emotions was measured using emotion categorization and emotion intensity rating tasks for adult faces. As per the findings, the accuracy of emotion recognition in children was impaired for various facial expressions (disgust, fear, happy, neutral, sad, and surprise faces), except for angry faces. Conversely, in adults, it was impaired for a few facial expressions. The perceived intensity for happy faces with a surgical mask was weaker in both children and adults than in those without the mask. A negative influence of wearing surgical masks was generally not observed for faces wearing a transparent mask in both children and adults. Thus, negative side effects of mask-wearing on reading emotions are observed for more facial expressions in children than in adults; transparent masks can help remedy these.

Sign language experience has little effect on face and biomotion perception in bimodal bilinguals

Sensory and language experience can affect brain organization and domain-general abilities. For example, D/deaf individuals show superior visual perception compared to hearing controls in several domains, including the perception of faces and peripheral motion. While these enhancements may result from sensory loss and subsequent neural plasticity, they may also reflect experience using a visual-manual language, like American Sign Language (ASL), where signers must process moving hand signs and facial cues simultaneously. In an effort to disentangle these concurrent sensory experiences, we examined how learning sign language influences visual abilities by comparing bimodal bilinguals (i.e., sign language users with typical hearing) and hearing non-signers. Bimodal bilinguals and hearing non-signers completed online psychophysical measures of face matching and biological motion discrimination. No significant group differences were observed across these two tasks, suggesting that sign language experience is insufficient to induce perceptual advantages in typical-hearing adults. However, ASL proficiency (but not years of experience or age of acquisition) was found to predict performance on the motion perception task among bimodal bilinguals. Overall, the results presented here highlight a need for more nuanced study of how linguistic environments, sensory experience, and cognitive functions impact broad perceptual processes and underlying neural correlates.

FacialCueNet: unmasking deception - an interpretable model for criminal interrogation using facial expressions

Polygraphs are used in criminal interrogations to detect deception. However, polygraphs can be difficult to administer under circumstances that prevent the use of biosensors. To address the shortcomings of the biosensors, deception-detection technology without biosensors is needed. We propose a deception-detection method, FacialCueNet, which is a multi-modal network that utilizes both facial images and facial cues based on deep-learning technology. FacialCueNet incorporates facial cues that indicate deception, such as action-unit frequency, symmetry, gaze pattern, and micro-expressions extracted from videos. Additionally, the spatial-temporal attention module, based on convolutional neural network and convolutional long short-term memory, is applied to FacialCueNet to provide interpretable information from interrogations. Because our goal was developing an algorithm applicable to criminal interrogations, we trained and evaluated FacialCueNet using the DDCIT dataset, which was collected using a data acquisition protocol similar to those used in actual investigations. To compare deception-detection performance with state-of-the-art works, a public dataset was also used. As a result, the mean deception-detection F1 score using the DDCIT dataset was 81.22%, with an accuracy of 70.79%, recall of 0.9476, and precision of 0.7107. When evaluating against the public database, our method demonstrated an evaluation accuracy of 88.45% and achieved an AUC of 0.9541, indicating a improvement of 1.25% compared to the previous results. We also present interpretive results of deception detection by analyzing the influence of spatial and temporal factors. These results show that FacialCueNet has the potential to detect deception using only facial videos. By providing interpretation of predictions, our system could be useful tool for criminal interrogation.

Detecting depression based on facial cues elicited by emotional stimuli in video

Recently, depression research has received considerable attention and there is an urgent need for objective and validated methods to detect depression. Depression detection based on facial expressions may be a promising adjunct to depression detection due to its non-contact nature. Stimulated facial expressions may contain more information that is useful in detecting depression than natural facial expressions. To explore facial cues in healthy controls and depressed patients in response to different emotional stimuli, facial expressions of 62 subjects were collected while watching video stimuli, and a local face reorganization method for depression detection is proposed. The method extracts the local phase pattern features, facial action unit (AU) features and head motion features of a local face reconstructed according to facial proportions, and then fed into the classifier for classification. The classification accuracy was 76.25%, with a recall of 80.44% and a specificity of 83.21%. The results demonstrated that the negative video stimuli in the single-attribute stimulus analysis were more effective in eliciting changes in facial expressions in both healthy controls and depressed patients. Fusion of facial features under both neutral and negative stimuli was found to be useful in discriminating between healthy controls and depressed individuals. The Pearson correlation coefficient (PCC) showed that changes in the emotional stimulus paradigm were more strongly correlated with changes in subjects’ facial AU when exposed to negative stimuli compared to stimuli of other attributes. These results demonstrate the feasibility of our proposed method and provide a framework for future work in assisting diagnosis.

The Benefit of Bimodal Training in Voice Learning.

It is known that talkers can be recognized by listening to their specific vocal qualities-breathiness and fundamental frequencies. However, talker identification can also occur by focusing on the talkers' unique articulatory style, which is known to be available auditorily and visually and can be shared across modalities. Evidence shows that voices heard while seeing talkers' faces are later recognized better on their own compared to the voices heard alone. The present study investigated whether the facilitation of voice learning through facial cues relies on talker-specific articulatory or nonarticulatory facial information. Participants were initially trained to learn the voices of ten talkers presented either on their own or together with (a) an articulating face, (b) a static face, or (c) an isolated articulating mouth. Participants were then tested on recognizing the voices on their own regardless of their training modality. Consistent with previous research, voices learned with articulating faces were recognized better on their own compared to voices learned alone. However, isolated articulating mouths did not provide an advantage in learning the voices. The results demonstrated that learning voices while seeing faces resulted in better voice learning compared to the voices learned alone.

Looking more criminal: It's not so black and white.

Prior research regarding the influence of face structure on character judgments and first impressions reveals that bias for certain face-types is ubiquitous, but these studies primarily used decontextualized White faces for stimuli. Given the disadvantages Black men face in the legal system, this study aimed to investigate whether the criminal face-type presented in the context of crime influenced different legal system-type judgments as a function of perpetrator race. In a mixed-model design, participants saw Black and White computer-generated faces that varied in criminality presented with either violent or nonviolent crime scenarios. At test, participants attempted to identify the original perpetrator from a photo array, along with providing penalty severity judgments for the crime committed. Results indicate that when crimes were violent, participants meted harsher penalties overall to Black faces or to high-criminality faces identified as the perpetrator. Furthermore, for violent crimes, participants were more likely to select a face from the photo array that was higher/equally as high in criminality rating relative to the actual perpetrator when memory failed or when the perpetrator was Black. Overall, the findings suggest that when people are making judgments that could influence another's livelihood, they may rely heavily on facial cues to criminality and the nature of the crime; and this is especially the case for Black faces presented in the context of violent crime. The pattern of results provides further support for the pervasive stereotype of Black men as criminal, even in our racially diverse sample wherein 36% identified as Black.

Does issue importance attenuate partisan cue-taking?

AbstractAre cues from party leaders so important that they can cause individuals to change their own issue positions to align with the party's position? Recent work on the importance of party cues suggests they do, especially given the literature on partisanship as a strong and persistent group identity. However, in this paper we test the limits of those partisan cues. Using a unique two-wave panel survey design we find that the effect of party cues is moderated by the prior level of importance individuals place on an issue. We find that when a person believes an issue area to be more important, party cues are less likely to move that citizen's position, particularly when the cue goes against partisan ideological norms. Our results show evidence that an individual's own issue positions—at least the important ones—can be resilient in the face of party cues.

Congruence in European and Asian perception of Vietnamese facial attractiveness, averageness, symmetry and sexual dimorphism

Attractiveness is a proposed universal cue to overall biological quality. Nonetheless, local raters and raters of the same ethnicity may be more accurate in assessing the cues for attractiveness than distant and unfamiliar raters. Shared ethnicity and shared environment may both affect rating accuracy: our aim was to compare their relative influence. Therefore, we photographed young Vietnamese participants (N = 93, 33 women) from Hanoi, Vietnam. The photographs were rated by Czechs, Asian Vietnamese, and Czech Vietnamese (raters of Vietnamese origin who lived in Czechia for all or most of their life). Using geometric morphometrics, we measured facial shape cues to biological quality: averageness, asymmetry, and sexual dimorphism. We expected that Vietnamese raters residing in Czechia and Vietnam would agree on perceived attractiveness and use shape-related facial cues to biological quality better than Czech European raters, who are less familiar with East Asians. Surprisingly, mixed-effect models and post hoc comparisons identified no major cross-group differences in attributed attractiveness and path analyses revealed that the three groups based their rating on shape-related characteristics in a similar way. However, despite the considerable cross-cultural agreement regarding perceived attractiveness, Czech European raters associated attractiveness with facial shape averageness significantly more than Vietnamese raters.

Colour information biases facial age estimation and reduces inter-observer variability

Age estimation is a perceptual task that people perform automatically and effortlessly on a daily basis. Colour has been identified as one of the facial cues that contributes to age perception. To investigate further the role of colour in age perception, we manipulated the chromatic content of facial images holistically. In Experiment 1, images were shown in colour or grey scale; in Experiment 2, images were shown with red–green contrast increased or decreased; in Experiment 3, images were shown with modified yellow–blue contrast. We examined whether the presence of chromatic information biases the perception of age and/or affects inter-observer variability in age judgements, and whether specific chromatic information affects the perception of age. We found that the same face tended to be judged as younger with increased red–green contrast compared to decreased red–green contrast, suggesting that red–green contrast directly affects age perception. Inter-observer variability in age ratings was significantly lower when participants were asked to rate colour compared with grey scale versions of images. This finding indicates that colour carries information useful cues for age estimation.

Detecting Deceptive Behaviours through Facial Cues from Videos: A Systematic Review

Interest in detecting deceptive behaviours by various application fields, such as security systems, political debates, advanced intelligent user interfaces, etc., makes automatic deception detection an active research topic. This interest has stimulated the development of many deception-detection methods in the literature in recent years. This work systematically reviews the literature focused on facial cues of deception. The most relevant methods applied in the literature of the last decade have been surveyed and classified according to the main steps of the facial-deception-detection process (video pre-processing, facial feature extraction, and decision making). Moreover, datasets used for the evaluation and future research directions have also been analysed.

Facial cues to physical strength are not always associated with facial masculinity: Comparative study of Europeans and Southern Siberians.

The aim of our study was to compare morphometric patterns of facial sexual dimorphism with strength-face relationship in members of two distinct populations of European and Central Asian origin: Russians and Tuvans. Handgrip strength (HGS) measures and facial photographs were collected from Russian (n = 233) and Tuvan (n = 187) men and women. We digitized 70 landmarks and semilandmarks on full-face and 54 landmarks and semilandmarks on profile photos. This was done to capture variation in facial morphology. After that, we performed the shape regressions of landmarks' coordinates upon sex and HGS. Results were visualized in forms of thin-plate deformation grids and geometric morphometric morphs. In both populations, HGS was associated significantly with male facial shape only. In Russian men, strength-related changes of facial shape were almost completely in direction of increase in male-typicality. This was especially evident for the relative lower facial width, which was higher in men compared to women, as well as in stronger men compared to weaker ones. On the contrary, in Tuvans the lower face was relatively narrower in men than in women. However, the facial shape of strong Tuvan men was also associated with relatively wider lower face. Our results indicate that the effect of strength on facial shape is relatively independent of facial sexual dimorphism. Our findings clearly demonstrate that physical strength is associated with the shape of the lower part of male faces even in populations with a mismatched direction of lower face sexual dimorphism.

The importance of face-shape masculinity for perceptions of male dominance depends on study design

Dominance perceptions play an important role in social interactions. Although many researchers have proposed that shape masculinity is an important facial cue for dominance perceptions, evidence for this claim has come almost exclusively from studies that assessed perceptions of experimentally manipulated faces using forced-choice paradigms. Consequently, we investigated the role of masculine shape characteristics in perceptions of men’s facial dominance (1) when shape-manipulated stimuli were presented in a forced-choice paradigm and (2) when unmanipulated face images were rated for dominance and shape masculinity was measured from face images. Although we observed large effects of masculinity on dominance perceptions when we used the forced-choice method (Cohen’s ds = 2.51 and 3.28), the effect of masculinity on dominance perceptions was considerably smaller when unmanipulated face images were rated and shape masculinity measured from face images (Cohen’s ds = 0.44 and 0.62). This pattern was observed when faces were rated separately for physical dominance, social dominance, and masculinity, and was seen for two different sets of stimuli. Collectively, these results suggest that shape masculinity may not be a particularly important cue for dominance perceptions when faces vary simultaneously on multiple dimensions, as is the case during everyday social interactions.