Processing Of Facial Expressions Research Articles

Analysis of convolutional neural networks reveals the computational properties essential for subcortical processing of facial expression

Perception of facial expression is crucial for primate social interactions. This visual information is processed through the ventral cortical pathway and the subcortical pathway. However, the subcortical pathway exhibits inaccurate processing, and the responsible architectural and physiological properties remain unclear. To investigate this, we constructed and examined convolutional neural networks with three key properties of the subcortical pathway: a shallow layer architecture, concentric receptive fields at the initial processing stage, and a greater degree of spatial pooling. These neural networks achieved modest accuracy in classifying facial expressions. By replacing these properties, individually or in combination, with corresponding cortical features, performance gradually improved. Similar to amygdala neurons, some units in the final processing layer exhibited sensitivity to retina-based spatial frequencies (SFs), while others were sensitive to object-based SFs. Replacement of any of these properties affected the coordinates of the SF encoding. Therefore, all three properties limit the accuracy of facial expression information and are essential for determining the SF representation coordinate. These findings characterize the role of the subcortical computational processes in facial expression recognition.

Semantic satiation of emotional words impedes facial expression processing in two stages

To explore the mechanism of emotional words semantic satiation effect on facial expression processing, participants were asked to judge the facial expression (happiness or sadness) after an emotional word (哭(cry) or 笑(smile)) or a neutral word (阿(Ah), baseline condition) was presented for 20 s. The results revealed that participants were slower in judging valence-congruent facial expressions and reported a more enlarged (Experiment 1) and prolonged (Experiment 2) N170 component than the baseline condition. No significant difference in behavior and N170 appeared between the valence-incongruent and the baseline condition. However, the amplitude of LPC (Late Positive Complex) under both valence-congruent/incongruent conditions was smaller than the baseline condition. It indicates that, in the early stage, the impeding effect of satiated emotional words is specifically constrained to facial expressions with the same emotional valence; in the late stage, such an impeding effect might spread to facial expressions with the opposite valence of the satiated emotional word.

EEG alpha band functional connectivity reveals distinct cortical dynamics for overt and covert emotional face processing

Current knowledge regarding how the focus of our attention during face processing influences neural responses largely comes from neuroimaging studies reporting on regional brain activations. The present study was designed to add novel insights to this research by studying how attention can differentially impact the way cortical regions interact during emotional face processing. High-density electroencephalogram was recorded in a sample of fifty-two healthy participants during an emotional face processing task. The task required participants to either attend to the expressions (i.e., overt processing) or attend to a perceptual distractor, which rendered the expressions task-irrelevant (i.e., covert processing). Functional connectivity in the alpha band was estimated in source space and modeled using graph theory to quantify whole-brain integration and segregation. Results revealed that overt processing of facial expressions is linked to reduced cortical segregation and increased cortical integration, this latter specifically for negative expressions of fear and sadness. Furthermore, we observed increased communication efficiency during overt processing of negative expressions between the core and the extended face processing systems. Overall, these findings reveal that attention makes the interaction among the nodes involved in face processing more efficient, also uncovering a connectivity signature of the prioritized processing mechanism of negative expressions, that is an increased cross-communication within the nodes of the face processing network.

Oxytocin normalizes the implicit processing of fearful faces in psychopathy: a randomized crossover study using fMRI

Adults with antisocial personality disorder with (ASPD + P) and without (ASPD – P) psychopathy commit the majority of violent crimes. Empathic processing abnormalities are particularly prominent in psychopathy, but effective pharmacological interventions have yet to be identified. Oxytocin modulates neural responses to fearful expressions in healthy populations. The current study investigates its effects in violent antisocial men. In a placebo-controlled, randomized crossover design, 34 violent offenders (19 ASPD + P; 15 ASPD – P) and 24 healthy non-offenders received 40 IU intranasal oxytocin or placebo and then completed an fMRI morphed faces task examining the implicit processing of fearful facial expressions. Increasing intensity of fearful facial expressions failed to appropriately modulate activity in the bilateral mid-cingulate cortex in violent offenders with ASPD + P, compared with those with ASPD – P. Oxytocin abolished these group differences. This represents evidence of neurochemical modulation of the empathic processing of others’ distress in psychopathy.

Testing EEG functional connectivity between sensorimotor and face processing visual regions in individuals with congenital facial palsy.

Moebius syndrome (MBS) is characterized by the congenital absence or underdevelopment of cranial nerves VII and VI, leading to facial palsy and impaired lateral eye movements. As a result, MBS individuals cannot produce facial expressions and did not develop motor programs for facial expressions. In the latest model of sensorimotor simulation, an iterative communication between somatosensory, motor/premotor cortices, and visual regions has been proposed, which should allow more efficient discriminations among subtle facial expressions. Accordingly, individuals with congenital facial motor disability, specifically with MBS, should exhibit atypical communication within this network. Here, we aimed to test this facet of the sensorimotor simulation models. We estimated the functional connectivity between the visual cortices for face processing and the sensorimotor cortices in healthy and MBS individuals. To this aim, we studied the strength of beta band functional connectivity between these two systems using high-density EEG, combined with a change detection task with facial expressions (and a control condition involving non-face stimuli). The results supported our hypothesis such that when discriminating subtle facial expressions, participants affected by congenital facial palsy (compared to healthy controls) showed reduced connectivity strength between sensorimotor regions and visual regions for face processing. This effect was absent for the condition with non-face stimuli. These findings support sensorimotor simulation models and the communication between sensorimotor and visual areas during subtle facial expression processing.

Investigating mechanism of the effect of emotional facial expressions on attentional processing by data clustering approach

To explore the mechanism of the effect of emotional facial expression on attentional process, time course and topographic map of Electroencephalographic activities affected by emotional stimuli were investigated. Emotional Stroop task was used to collect 64-channel event-related potentials (ERP) in nonclinical participants, and data clustering was applied to find significant effect of sad and happy facial expression on ERP. Several significant ERP clusters were found in the sad and happy conditions respectively. In the sad condition, the decreased N170 in the bilateral parietooccipital areas, the increased P3 in the right centroparietal region and the increased negative deflection between 600 and 650 ms in the prefrontal regions were observed, these alterations reflected inhibited perceptual processing of sad facial expression, and increased activations of the orienting network and the executive control network in attentional system, respectively. In the happy condition, increased negative slow wave was found in the left centroparietal region indicating strengthened awareness and readiness for successive trials. Importantly, nonpathological attentional bias to sad facial expression in nonclinical participants was associated with inhibited perceptual processing and increased activations of the orienting and executive control networks. It provides the basis for better understanding and application of attentional bias in psychiatric clinical utilization.

Effects of aging on face processing: An ERP study of the own-age bias with neutral and emotional faces

Older adults systematically show an enhanced N170 amplitude during the visualization of facial expressions of emotion. The present study aimed to replicate this finding, further investigating if this effect is specific to facial stimuli, present in other neural correlates of face processing, and modulated by own-age faces. To this purpose, younger (n=25; Mage=28.36), middle-aged (n=23; Mage=48.74), and older adults (n=25; Mage=67.36) performed two face/emotion identification tasks during an EEG recording. The results showed that groups did not differ regarding P100 amplitude, but older adults had increased N170 amplitude for both facial and non-facial stimuli. The event-related potentials analysed were not modulated by an own-age bias, but older faces elicited larger N170 in the Emotion Identification Task for all groups. This increased amplitude may reflect a higher ambiguity of older faces due to age-related changes in their physical features, which may elicit higher neural resources to decode. Regarding P250, older faces elicited decreased amplitudes than younger faces, which may reflect a reduced processing of the emotional content of older faces. This interpretation is consistent with the lower accuracy obtained for this category of stimuli across groups. These results have important social implications and suggest that aging may hamper the neural processing of facial expressions of emotion, especially for own-age peers.

Face-Selective Patches in Marmosets Are Involved in Dynamic and Static Facial Expression Processing.

The correct identification of facial expressions is critical for understanding the intention of others during social communication in the daily life of all primates. Here we used ultra-high-field fMRI at 9.4 T to investigate the neural network activated by facial expressions in awake New World common marmosets from both male and female sex, and to determine the effect of facial motions on this network. We further explored how the face-patch network is involved in the processing of facial expressions. Our results show that dynamic and static facial expressions activate face patches in temporal and frontal areas (O, PV, PD, MD, AD, and PL) as well as in the amygdala, with stronger responses for negative faces, also associated with an increase of the respiration rates of the monkey. Processing of dynamic facial expressions involves an extended network recruiting additional regions not known to be part of the face-processing network, suggesting that face motions may facilitate the recognition of facial expressions. We report for the first time in New World marmosets that the perception and identification of changeable facial expressions, vital for social communication, recruit face-selective brain patches also involved in face detection processing and are associated with an increase of arousal.SIGNIFICANCE STATEMENT Recent research in humans and nonhuman primates has highlighted the importance to correctly recognize and process facial expressions to understand others' emotions in social interactions. The current study focuses on the fMRI responses of emotional facial expressions in the common marmoset (Callithrix jacchus), a New World primate species sharing several similarities of social behavior with humans. Our results reveal that temporal and frontal face patches are involved in both basic face detection and facial expression processing. The specific recruitment of these patches for negative faces associated with an increase of the arousal level show that marmosets process facial expressions of their congener, vital for social communication.

Emotion recognition from ambiguous facial expressions and utterances: Relationship between implicit and explicit processing

Previous studies have differentiated between implicit and explicit processing in emotion recognition using facial expressions and utterances. Implicit processing relates to observed cues, and explicit processing to inference of the expressed emotion. However, the relationships between implicit and explicit processing, and between emotion recognition from facial expressions and from utterances, have not yet been empirically determined. To investigate this issue, the present study examined the more important cues in judging the emotion of the expressor by presenting facial expressions that expressed different emotions in the eyes and mouth, and utterances that expressed different emotions in prosody and lexical content. Additionally, the study examined the cues the participants were more likely to focus their attention on. The results indicated no relationship between implicit and explicit processing. However, an association between the explicit processing of facial expressions and that of utterances was observed, suggesting that explicit processing is independent of implicit processing, and influenced by social conventions. These findings expand our knowledge regarding the processes involved in emotion recognition.

Equivalent processing of facial expression and identity by macaque visual system and task-optimized neural network

Both the primate visual system and artificial deep neural network (DNN) models show an extraordinary ability to simultaneously classify facial expression and identity. However, the neural computations underlying the two systems are unclear. Here, we developed a multi-task DNN model that optimally classified both monkey facial expressions and identities. By comparing the fMRI neural representations of the macaque visual cortex with the best-performing DNN model, we found that both systems: (1) share initial stages for processing low-level face features which segregate into separate branches at later stages for processing facial expression and identity respectively, and (2) gain more specificity for the processing of either facial expression or identity as one progresses along each branch towards higher stages. Correspondence analysis between the DNN and monkey visual areas revealed that the amygdala and anterior fundus face patch (AF) matched well with later layers of the DNN's facial expression branch, while the anterior medial face patch (AM) matched well with later layers of the DNN's facial identity branch. Our results highlight the anatomical and functional similarities between macaque visual system and DNN model, suggesting a common mechanism between the two systems.

Analysis of attentional biases in anxiety using 24 facial priming sequences.

The processing of emotional facial expressions helps people to adjust to the physical and social environment. Furthermore, mental disorders such as anxiety have been linked to attentional biases in the processing of this type of information. Nevertheless, there are still contradictory results that might be due to the methodology used and to individual differences in the manifestation of anxiety. Our research goal was to use 24 facial priming sequences to analyse attentional biases in the detection of facial expressions of fear, considering the levels and the ways in which individuals express anxiety. With higher levels of cognitive anxiety and general trait anxiety, those sequences that began in the upper half (vs. lower half) elicited a speedier response in the detection of fear. The results are discussed within the context of other techniques and disorders that prompt a deficit in the processing of facial information.

Event-Related Potentials Index Prediction Error Signalling During Perceptual Processing of Emotional Facial Expressions

Humans use socially relevant stimuli to guide perceptual processing of the surrounding environment, with emotional stimuli receiving preferential attention due to their social importance. Predictive coding theory asserts this cognitive process occurs efficiently by combining predictions about what is to be perceived with incoming sensory information, generating prediction errors that are then used to update future predictions. Recent evidence has identified differing neural activity that demonstrates how spatial and feature-based attention may interact with prediction, yet how emotion-guided attention may influence this relationship remains unknown. In the present study, participants viewed a display of two faces in which attention, prediction, and emotion were manipulated, and responded to a face expressing a specific emotion (anger or happiness). The N170 was found to be enhanced by unpredictable as opposed to predictable stimuli, indicating that it indexes general prediction error signalling processes. The N300 amplitudes were also enhanced by unpredictable stimuli, but they were also affected by the attentional status of angry but not happy faces, suggesting that there are differences in prediction error processes indexed by the N170 and N300. Overall, the findings suggest that the N170 and N300 both index violations of expectation for spatial manipulations of stimuli in accordance with prediction error responding processes.

The widespread action observation/execution matching system for facial expression processing.

Observing and understanding others' emotional facial expressions, possibly through motor synchronization, plays a primary role in face-to-face communication. To understand the underlying neural mechanisms, previous functional magnetic resonance imaging (fMRI) studies investigated brain regions that are involved in both the observation/execution of emotional facial expressions and found that the neocortical motor regions constituting the action observation/execution matching system or mirror neuron system were active. However, it remains unclear (1) whether other brain regions in the limbic, cerebellum, and brainstem regions could be also involved in the observation/execution matching system for processing facial expressions, and (2) if so, whether these regions could constitute a functional network. To investigate these issues, we performed fMRI while participants observed dynamic facial expressions of anger and happiness and while they executed facial muscle activity associated with angry and happy facial expressions. Conjunction analyses revealed that, in addition to neocortical regions (i.e., the right ventral premotor cortex and right supplementary motor area), bilateral amygdala, right basal ganglia, bilateral cerebellum, and right facial nerve nucleus were activated during both the observation/execution tasks. Group independent component analysis revealed that a functional network component involving the aforementioned regions were activated during both observation/execution tasks. The data suggest that the motor synchronization of emotional facial expressions involves a widespread observation/execution matching network encompassing the neocortex, limbic system, basal ganglia, cerebellum, and brainstem.

Neural correlates of emotional valence for faces and words.

Stimuli with negative emotional valence are especially apt to influence perception and action because of their crucial role in survival, a property that may not be precisely mirrored by positive emotional stimuli of equal intensity. The aim of this study was to identify the neural circuits differentially coding for positive and negative valence in the implicit processing of facial expressions and words, which are among the main ways human beings use to express emotions. Thirty-six healthy subjects took part in an event-related fMRI experiment. We used an implicit emotional processing task with the visual presentation of negative, positive, and neutral faces and words, as primary stimuli. Dynamic Causal Modeling (DCM) of the fMRI data was used to test effective brain connectivity within two different anatomo-functional models, for the processing of words and faces, respectively. In our models, the only areas showing a significant differential response to negative and positive valence across both face and word stimuli were early visual cortices, with faces eliciting stronger activations. For faces, DCM revealed that this effect was mediated by a facilitation of activity in the amygdala by positive faces and in the fusiform face area by negative faces; for words, the effect was mainly imputable to a facilitation of activity in the primary visual cortex by positive words. These findings support a role of early sensory cortices in discriminating the emotional valence of both faces and words, where the effect may be mediated chiefly by the subcortical/limbic visual route for faces, and rely more on the direct thalamic pathway to primary visual cortex for words.

Insomnia and impacts on facial expression recognition accuracy, intensity and speed: A meta-analysis

Facial expressions provide nonverbal cues that are important for delivering and interpreting human emotions. Previous studies have shown that the ability to interpret facial emotions correctly could be partially impaired in sleep-deprived people. People with insomnia might also suffer from sleep loss, so we assumed that facial expression recognition ability might also be impaired in people with insomnia. Despite a growing body of research exploring insomnia's potential impacts on facial expression recognition, conflicting results have been reported, and no systematic review of this research body has been conducted. In this study, after screening 1100 records identified through database searches, six articles examining insomnia and facial expression recognition ability were included in a quantitative synthesis. The main outcomes were classification accuracy (ACC), reaction time (RT), and intensity rating—the three most studied facial expression processing variables. Subgroup analysis was performed to identify altered perceptions according to the facial expressions of four emotions—happiness, sadness, fear, and anger—used to examine insomnia and emotion recognition. The pooled standard mean differences (SMDs) and corresponding 95% confidence intervals (CIs) demonstrated that facial expression recognition among people with insomnia was less accurate (SMD = −0.30; 95% CI: −0.46, −0.14) and slower (SMD = 0.67; 95% CI: 0.18, −1.15) compared to good sleepers. The classification ACC of fearful expression was lower in the insomnia group (SMD = −0.66; 95% CI: −1.02, −0.30). This meta-analysis was registered using PROSPERO.

The relationship between alexithymia, interoception, and neural functional connectivity during facial expression processing in autism spectrum disorder

Neural processing differences of emotional facial expressions, while common in autism spectrum disorder (ASD), may be related to co-occurring alexithymia and interoceptive processing differences rather than autism per se. Here, we investigate relationships between alexithymia, interoceptive awareness of emotions, and functional connectivity during observation of facial expressions in youth (aged 8–17) with ASD (n = 28) compared to typically developing peers (TD; n = 37). Behaviorally, we found no significant differences between ASD and TD groups in interoceptive awareness of emotions, though alexithymia severity was significantly higher in the ASD group. In the ASD group, increased alexithymia was significantly correlated with lower interoceptive sensation felt during emotion. Using psycho-physiological interaction (PPI) analysis, the ASD group showed higher functional connectivity between the left ventral anterior insula and the left lateral prefrontal cortex than the TD group when viewing facial expressions. Further, alexithymia was associated with reduced left anterior insula-right precuneus connectivity and reduced right dorsal anterior insula-left ventral anterior insula connectivity when viewing facial expressions. In the ASD group, the degree of interoceptive sensation felt during emotion was positively correlated with left ventral anterior insula-right IFG connectivity when viewing facial expressions. However, across all participants, neither alexithymia nor interoceptive awareness of emotions predicted connectivity between emotion-related brain regions when viewing emotional facial expressions. To summarize, we found that in ASD compared to TD: 1) there is stronger connectivity between the insula and lateral prefrontal cortex; and 2) differences in interhemispheric and within left hemisphere connectivity between the insula and other emotion-related brain regions are related to individual differences in interoceptive processing and alexithymia. These results highlight complex relationships between alexithymia, interoception, and brain processing in ASD.

The other-race effect in facial expression processing: Behavioral and ERP evidence from a balanced cross-cultural study in women.

Although evidence for cultural variants in facial expression decoding is accumulating, the other-race effect in facial expression processing and its neural correlates are still unclear. We investigated this question with a fully balanced design, in which a group of East Asian and a group of European Caucasian women categorized pictures of sad, happy, angry, and neutral facial expressions posed by individuals of their own-race and the other-race. Results revealed a disadvantage in categorizing expressions of anger in other-race faces in both samples, and for sad expressions in the European sample only. Partially consistent, East Asian participants showed longer latency of the N170 component in the event-related potential (ERP) and European Caucasian participants showed larger N170 amplitudes to other-race faces. The late positive complex in the ERP was less distinguishable among other-race facial expressions. Therefore, the present study observed an other-race effect in early and late stages of face processing, reflecting less efficient structural encoding and less elaborate processing for other-race than own-race faces.

Probing the processing of facial expressions in monkeys via time perception and eye tracking.

Accurately recognizing facial expressions is essential for effective social interactions. Non-human primates (NHPs) are widely used in the study of the neural mechanisms underpinning facial expression processing, yet it remains unclear how well monkeys can recognize the facial expressions of other species such as humans. In this study, we systematically investigated how monkeys process the facial expressions of conspecifics and humans using eye-tracking technology and sophisticated behavioral tasks, namely the temporal discrimination task (TDT) and face scan task (FST). We found that monkeys showed prolonged subjective time perception in response to Negative facial expressions in monkeys while showing longer reaction time to Negative facial expressions in humans. Monkey faces also reliably induced divergent pupil contraction in response to different expressions, while human faces and scrambled monkey faces did not. Furthermore, viewing patterns in the FST indicated that monkeys only showed bias toward emotional expressions upon observing monkey faces. Finally, masking the eye region marginally decreased the viewing duration for monkey faces but not for human faces. By probing facial expression processing in monkeys, our study demonstrates that monkeys are more sensitive to the facial expressions of conspecifics than those of humans, thus shedding new light on inter-species communication through facial expressions between NHPs and humans.

ДИНАМІКА ЕЕГ ПІД ЧАС СПРИЙНЯТТЯ ЕМОЦІЙНИХ ВИРАЗІВ ОБЛИЧ ЗА РІЗНОГО ЧАСУ ДЕМОНСТРАЦІЇ: ДОСЛІДЖЕННЯ ПОТЕНЦІАЛІВ, ПОВ'ЯЗАНИХ ІЗ ПОДІЯМИ

The face is one of the most significant elements of non-verbal communication, which allows humans to quickly get various types of information about a specific person. Therefore, understanding facial expressions is crucial for normal social interaction, as it lets us determine the emotional states and intentions of other people. The purpose of this study was to investigate the influence of the time factor on the happy and fearful facial expression processing, reflected in the structure and components of event-related potentials (ERPs). Accordingly, two parallel examinations were conducted with stimulus presentation periods of 500 ms and 100 ms. For the obtained data, two separate analyses (for happy and fearful faces) of the averaged ERP curves corresponding to different stimulus exposure periods were executed. The initial stages of processing (EPN, P3) demonstrated that the difference in the structure and amplitude characteristics of the ERP curve was not that significant, but it demonstrated an increase in attentional resources involvement and a strengthening of the primary emotional analysis. Later ERP components (N400, LPP) showed a more compelling difference, that reflected the complexity of semantic decoding, subsequent conscious evaluation, and processes related to decision-making. Thus, reducing the time of image display from 500 ms to 100 ms created the expected difficulties for the perception and processing of emotional facial expressions and was not sufficient for the normal course of these processes.

Positive and negative facial valence perception are modulated differently by eccentricity in the parafovea

Understanding whether people around us are in a good, bad or neutral mood can be critical to our behavior, both when looking directly at them or when they are in our peripheral visual field. However, facial expressions of emotions are often investigated at central visual field or at locations right or left of fixation. Here we assumed that perception of facial emotional valence (the emotion’s pleasantness) changes with distance from central visual field (eccentricity) and that different emotions may be influenced differently by eccentricity. Participants (n = 58) judged the valence of emotional faces across the parafovea (≤ 4°, positive (happy), negative (fearful), or neutral)) while their eyes were being tracked. As expected, performance decreased with eccentricity. Positive valence perception was least affected by eccentricity (accuracy reduction of 10–19% at 4°) and negative the most (accuracy reduction of 35–38% at 4°), and this was not a result of speed-accuracy trade-off or response biases. Within-valence (but not across-valence) performance was associated across eccentricities suggesting perception of different valences is supported by different mechanisms. While our results may not generalize to all positive and negative emotions, they indicate that beyond-foveal investigations can reveal additional characteristics of the mechanisms that underlie facial expression processing and perception.