Facial Emotion Recognition Research Articles

Differences in emotion recognition between nonimmersive versus immersive virtual reality: preliminary findings in schizophrenia and bipolar disorder.

Deficits in social cognition may impair emotional processing and facial emotional recognition (FER) in patients with bipolar disorder (BD) and schizophrenia. FER is generally explored using photographs or images of static faces that do not fully capture the complexity of real-life facial stimuli. To overcome this limitation, we developed a set of dynamic virtual faces depicting six basic emotions (i.e. happiness, sadness, anger, fear, disgust, and surprise) and a neutral expression suitable for presentation in immersive and nonimmersive virtual realities. This study presents preliminary findings on the differences in FER accuracy from a frontal view between immersive and nonimmersive virtual realities among patients experiencing a relapse of schizophrenia (n = 10), a manic phase of BD (n = 10), and a group of healthy controls (HCs) (n = 10). As a secondary objective, we compare the FER accuracy across these three groups. Patients with schizophrenia and BD showed similar accuracy in recognizing emotions in immersive and nonimmersive virtual reality settings. However, patients with schizophrenia exhibited lower FER accuracy than HCs in both settings. Individuals with BD showed intermediate accuracy between those with schizophrenia and HCs, although these differences were not statistically significant. Notably, recognition of negative emotions was significantly impaired in both groups of patients.

PortraitEmotion3D: A Novel Dataset and 3D Emotion Estimation Method for Artistic Portraiture Analysis

Facial Expression Recognition (FER) has been widely explored in realistic settings; however, its application to artistic portraiture presents unique challenges due to the stylistic interpretations of artists and the complex interplay of emotions conveyed by both the artist and the subject. This study addresses these challenges through three key contributions. First, we introduce the PortraitEmotion3D (PE3D) dataset, designed explicitly for FER tasks in artistic portraits. This dataset provides a robust foundation for advancing emotion recognition in visual art. Second, we propose an innovative 3D emotion estimation method that leverages three-dimensional labeling to capture the nuanced emotional spectrum depicted in artistic works. This approach surpasses traditional two-dimensional methods by enabling a more comprehensive understanding of the subtle and layered emotions often in artistic representations. Third, we enhance the feature learning phase by integrating a self-attention module, significantly improving facial feature representation and emotion recognition accuracy in artistic portraits. This advancement addresses this domain’s stylistic variations and complexity, setting a new benchmark for FER in artistic works. Evaluation of the PE3D dataset demonstrates our method’s high accuracy and robustness compared to existing state-of-the-art FER techniques. The integration of our module yields an average accuracy improvement of over 1% in recent FER systems. Additionally, combining our method with ESR-9 achieves a comparable accuracy of 88.3% on the FER+ dataset, demonstrating its generalizability to other FER benchmarks. This research deepens our understanding of emotional expression in art and facilitates potential applications in diverse fields, including human–computer interaction, security, healthcare diagnostics, and the entertainment industry.

Drug Holidays May Attenuate Beneficial Effects of Treatment on Emotion Regulation and Recognition Among Children with ADHD: A Single-Center, Prospective Study.

In this study, we aimed to evaluate the effects of drug holidays during summer vacations on children with attention deficit/hyperactivity disorder (ADHD) treated with methylphenidate in terms of ADHD symptoms and emotion regulation (ER). This single-center, naturalistic, prospective cohort study included pre-, post-, and post-drug holiday evaluations. All patients diagnosed with ADHD and included in our study were administered the Clinical Global Impression Scale, Affective Reactivity Index-parent and child, reading the mind in the eyes test (RMET), Faces test and Cognitive Emotion Regulation Questionnaire for Children 3 times. Fifty-six patients met the inclusion criteria and ten were lost to follow-up. Thirty-nine participants had complete evaluations at all time points. Both parent and self-report ratings of child irritability were significantly reduced after methylphenidate (MPH) treatment (P=.003 and .002, respectively), although they returned to baseline after drug holidays (P=.618 and .974, respectively). The use of self-blame as a cognitive emotion regulation strategy increased significantly after treatment and remained significantly elevated even after drug holidays (P=.024 and .022, respectively). Children used planning as a cognitive strategy significantly more frequently during MPH treatment (P=.034), although this elevation was temporary and returned to baseline levels after the drug holidays (P=.890). Reading the mind in the eyes test performance was significantly improved after MPH treatment and did not change after drug holidays (P=.009 and .006, respectively), while there was a tendency for facial emotion recognition abilities to improve at the last visit (P=.051). Our results suggest that MPH treatment improves child- and parent-reported irritability, clinician-rated disorder severity, RMET performance, and the use of planning as a cognitive ER strategy, although only improvement in RMET performance continued after drug holidays. Our results may support the importance of continuing treatment over the holidays for most children with ADHD unless necessitated by adverse events, growth problems, or development of tolerance.

Deep-SFER: deep convolutional neural network and MFCC an effective speech and face emotion recognition

There has been a lot of progress in recent years in the fields of expert systems, artificial intelligence (AI) and human machine interface (HMI). The use of voice commands to engage with machinery or instruct it to do a certain task is becoming more common. Numerous consumer electronics have SIRI, Alexa, Cortana, and Google Assistant built in. In the field of human-device interaction, emotion recognition from speech is a complex research subject. We can't imagine modern life without machines, so naturally there's a need to create a more robust framework for human-machine communication. A number of academics are now working on speech emotion recognition (SER) in an effort to improve the interaction between humans and machines. We aimed to identify four fundamental emotions: angry, unhappy, neutral and joyful from speech in our experiment. As you can hear below, we trained and tested our model using audio data of brief Manipuri speeches taken from films. This task makes use of convolutional neural networks (CNNs) to extract functions from speech in order to recognize different moods using the Mel-frequency cepstral coefficient (MFCC).

Music Recommendation System Using Facial Emotion Gestures

Abstract: Music recommendation systems are transformingthe way users interact with music streaming servicesby personalizing experiences. This paper presents a novel approach that leverages facial emotion recognition (FER) to suggest music dynamically based on users' emotional states. Using machinelearning and deep learning models, facial expressions are analyzed in real-time through video input. The system integrates emotion-based recommendations with collaborative and content- based filtering techniques to enhance accuracy. Thisapproach offers a more intuitive and human-centered interaction compared to traditional systems. The proposed solution can be used across streaming platforms, personalized wellness apps, and therapeutic environments.

A 2-Year Longitudinal Neuropsychological Study in Relapsing-Remitting Multiple Sclerosis: A Selective Decline in Social Cognition?

Background/Objectives. Social cognition (SC), which implies the emotional and intellectual understanding of oneself and others, is an important facet of neuropsychological functioning concurrently to academic cognition (AC), which concerns non-social abilities (memory, language…). In relapsing-remitting multiple sclerosis (RRMS), it is not clear whether a cognitive decline occurs in both SC and AC nor whether a link exists between these two cognitive domains. The objective of the present longitudinal study was to conduct an extensive examination of both AC and SC in RRMS to document a 2-year evolution and to look for potential correlations between AC and SC. Methods. The neuropsychological results (AC and SC) of 48 RRMS patients obtained in clinical practice were retrospectively considered; 38 of the patients (30 females) were assessed again about 2 years later. Non-parametric tests were applied to test the intra-group cognitive evolution (Wilcoxon) and the link between AC and SC evolution (Spearman). Results. Whereas AC showed a stability or an improvement of performances during the retest, SC presented the reverse pattern, with a stability or a significant decline in facial emotion (recognition and discrimination) and humor perception. No significant statistical correlation was found between the significant modification of AC and SC during follow-up. Conclusions. The short-term deleterious evolution observed selectively for SC in the present study suggests that SC should be selected as a cognitive marker for RRMS follow-up, and that extensive examination may be preferred to investigate specific SC changes.

Real-Time Facial Emotion Detection Application with Image Processing Based on Convolutional Neural Network (CNN)

Facial Emotion Recognition (FER) is a key technology for identifying emotions based on facial expressions, with applications in human-computer interaction, mental health monitoring, and customer analysis. This study presents the development of a real-time emotion recognition system using Convolutional Neural Networks (CNNs) and OpenCV, addressing challenges such as varying lighting and facial occlusions. The system, trained on the FER2013 dataset, achieved 85% accuracy in emotion classification, demonstrating high performance in detecting happiness, sadness, and surprise. The results highlight the system's effectiveness in real-time applications, offering potential for use in mental health and customer behavior analysis.

Improved facial emotion recognition model based on a novel deep convolutional structure

Facial Emotion Recognition (FER) is a very challenging task due to the varying nature of facial expressions, occlusions, illumination, pose variations, cultural and gender differences, and many other aspects that cause a drastic degradation in quality of facial images. In this paper, an anti-aliased deep convolution network (AA-DCN) model has been developed and proposed to explore how anti-aliasing can increase and improve recognition fidelity of facial emotions. The AA-DCN model detects eight distinct emotions from image data. Furthermore, their features have been extracted using the proposed model and numerous classical deep learning algorithms. The proposed AA-DCN model has been applied to three different datasets to evaluate its performance: The Cohn-Kanade Extending (CK+) database has been utilized, achieving an ultimate accuracy of 99.26% in (5 min, 25 s), the Japanese female facial expressions (JAFFE) obtained 98% accuracy in (8 min, 13 s), and on one of the most challenging FER datasets; the Real-world Affective Face (RAF) dataset; reached 82%, in low training time (12 min, 2s). The experimental results demonstrate that the anti-aliased DCN model is significantly increasing emotion recognition while improving the aliasing artifacts caused by the down-sampling layers.

Eye movement characteristics of emotional face recognizing task in patients with mild to moderate depression.

Depression is a complex affective disorder characterized by high prevalence and severe impact, commonly presenting with cognitive impairment. The objective diagnosis of depression lacks precise standards. This study investigates eye movement characteristics during emotional face recognition task (EFRT) in depressive patients to provide empirical support for objective diagnosis. We recruited 43 patients with depression (Depressive patients, DP) from a psychiatric hospital and 44 healthy participants (Healthy Control, HC) online. All participants completed an EFRT comprising 120 trials. Each trial presented a gray screen for 800 ms followed by a stimulus image for judgment. Emotions were categorized as positive, neutral, or negative. Eye movement trajectories were recorded throughout the task. Latency of First Fixation (LFF), Latency of First Fixation for Eye AOI, and Latency of First Fixation for Mouth AOI were used as representative indicators of early attention, Proportion of Eye AOI, and Proportion of Mouth AOI as measures of intermediate attention, Accuracy (ACC) and Reaction Time (RT) as behavioral indicators of late-stage attention. In this study, these metrics were employed to explore the differences between patients with depression and healthy individuals. Compared to healthy participants, individuals with depression exhibit longer first fixation latencies on the eyes and mouth during the early attention stage of emotional face recognition, indicating an avoidance tendency toward key facial recognition cues. In the mid-to-late attention stages, depressive individuals show an increased fixation ratio on the eyes and a decreased fixation ratio on the mouth, along with lower accuracy and longer response times. These findings suggest that, relative to healthy individuals, individuals with depression have deficits in facial recognition. This study identified distinct attention patterns and cognitive deficits in emotional face recognition among individuals with depression compared to healthy individuals, providing an attention-based approach for exploring potential clinical diagnostic markers for depression.

Age differences in mental state inference of sarcasm: Contributions of facial emotion recognition and cognitive performance.

Older age is associated with poorer ability to accurately infer mental states, but some mental states are more complex than others. Sarcasm is a complex mental state because the literal and intended meaning of a speaker's words are in opposition. Individuals must rely on additional cues (e.g., facial expressions, intonation) for accurate inference. We hypothesized that understanding of sarcastic versus sincere exchanges would be more sensitive to age-related difficulty in mental state understanding. We examined accuracy at identifying sarcasm among 263 adults (ages 18-90 years) using videos of social interactions in The Awareness of Social Inference Test. Hypotheses were tested using a logistic linear mixed effects model predicting correct/incorrect trial-level responses. To characterize why sarcasm differed with age, we measured two abilities commonly implicated in mental state understanding: facial emotion recognition and cognitive performance. Sarcasm understanding declined with age, whereas understanding of sincere exchanges did not. Both better emotion recognition and cognitive performance related to better understanding of sarcastic but not sincere exchanges. Only cognitive performance showed an age-related effect such that the cognitive performance among the oldest participants facilitated their understanding of both sarcastic and sincere exchanges. We showed that individual variation related to age and social and cognitive performance is more pronounced when the use of multiple mental state cues is more (sarcasm) or less (sincerity) necessary for accurate understanding of social interactions. Naturalistic paradigms involving multiple mental state cues can address important questions about how older adults make decisions in the real world.

Facial Emotion Recognition Using Residual Neural Networks

Facial Emotion Recognition Using Residual Neural Networks

Behavioral and neural evidence for difficulty recognizing masked emotional faces.

Facial emotion recognition is vital for human social behavior. During the COVID-19 pandemic, face masks were widely adopted for viral mitigation and remain crucial public health tools. However, questions persist about their impact on emotion recognition and neural processing, especially in children, parents, and young adults. We developed the Masked Affective and Social Cognition task, featuring masked and unmasked faces displaying fear, sadness, and anger. We recruited three racial and ethnically diverse samples: 119 college students, 30 children who entered school age at the beginning of the pandemic, and 31 fathers of the aforementioned children. Of the latter two groups, 41 participants (n = 23 fathers, 18 children) did the Masked Affective and Social Cognition task during a neuroimaging scan, while the remaining 20 participants (n = 8 fathers, 12 children) who were not eligible for scanning completed the task during their lab visit. Behaviorally, we found that participants recognized emotions less accurately when viewing masked faces and also found an interaction of emotion by condition, such that accuracy was particularly compromised by sad masked faces. Neurally, masked faces elicited greater activation in the posterior cingulate, insula, and fusiform gyrus. Anterior insula and inferior frontal gyrus activation were driven by sad, masked faces. These results were consistent across age groups. Among fathers, activation to sad masked faces was associated with stress and depression. Overall, our findings did not depend on previous mask exposure or timing of participation during the pandemic. These results have implications for understanding face emotion recognition, empathy, and socioemotional neurodevelopment. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Impairments in facial expression recognition in patients with euthymic bipolar disorders using the facial emotions recognition test

BackgroundPatients with bipolar disorders (BD) have significant impairments in Facial Expression Recognition (FER), an essential social skill for effective social interactions. While the Facial Emotions Recognition Test, 54 photographs by Gaudelus (TREF-54g) has been used in patients with schizophrenia, no study has evaluated FER using this test in BD patients. The TREF-54g meets the International Society for Bipolar Disorders (ISBD) criteria for assessing FER. The objective of this study was to compare FER in patients with euthymic BD versus healthy controls (HC) using the TREF-54g. MethodsThis study included 60 patients with BD diagnosed according to DSM-5 criteria, 30 with bipolar I disorder (BDI) and 30 with bipolar II disorder (BDII) and 60 HC. All groups were matched for age, gender, and years of education. Overall TREF-54g impairment score was defined as a difference of two SD from the group of HC with mean overall accuracy score expressed as a percentage of correct responses. ResultsOverall TREF-54g impairment scores, as well as sub-scores for anger and contempt recognition, were significantly higher in patients with BD versus those in HC. No significant differences in TREF-54g scores were shown between patients with BDI and those with BDII. LimitationsClinical assessments of this study didn't include neurocognition assessments and classification of contempt emotion as a basic emotion remains controversial. ConclusionsWe found significant impairments in FER in patients with euthymic BD, particularly for anger and contempt using the TREF-54g a test that meets ISBD recommendation criteria.

Disgust-specific impairment of facial emotion recognition in Parkinson's disease patients with mild cognitive impairment.

This study investigated the association between cognitive function and facial emotion recognition (FER) in patients with Parkinson's disease (PD) and mild cognitive impairment (PD-MCI). We enrolled 126 participants from Taiwan, including 63 patients with idiopathic PD and 63 matched healthy controls. The PD group was divided into two groups: those with normal cognitive function (PD-NC) and those with MCI (PD-MCI). Participants underwent a modality emotion recognition test and comprehensive cognitive assessment. Our findings reveal that patients with PD-MCI exhibit significantly impaired FER, especially in recognizing "disgust," compared with patients with PD-NC and healthy adults (P = .001). This deficit correlates with executive function, attention, memory, and visuospatial abilities. Attention mediates the relationship between executive function and "disgust" FER. The findings highlight how patients with PD-MCI are specifically challenged when recognizing "disgust" and suggest that cognitive training focusing on cognitive flexibility and attention may improve their FER abilities. This study contributes to our understanding of the nuanced relationship between cognitive dysfunction and FER in patients with PD-MCI, emphasizing the need for targeted interventions.

The impact of fear of pain on the temporal processing of emotional faces: modulation of attentional resources.

Recent research has focused on the spatial attention bias toward pain-related cues in individuals with fear of pain (FOP), but our understanding of its temporal attention features, particularly regarding emotional faces, is limited. To address this gap, 39 individuals with high fear of pain (H-FOP) and 37 with low fear of pain (L-FOP) completed a rapid serial visual presentation dual task. Participants viewed a series of rapidly presented pictures, first identified neutral building images as the first target (T1) and then emotional faces (painful, neutral, and happy) as the second target (T2). T1 and T2 presented at time intervals of 232ms (lag 2) and 696ms (lag 6), respectively. Results indicate that H-FOP group members were less accurate in recognizing faces at lag 2 shortly after correctly identifying T1 compared to those L-FOP group members. Notably, all participants were more accurate in recognizing painful faces at lag 2 than in identifying neutral and happy faces, suggesting a negative bias toward pain stimuli when attentional resources were limited. In contrast, at lag 6, when attentional resources were more abundant, participants showed greater accuracy in recognizing neutral faces than emotional faces. These findings underscore the influence of fear of pain and available cognitive resources on the attentional processing of emotional faces over time, providing insights into how fear of pain affects the temporal dynamics of emotional face recognition and the role of attentional resources.

An improved student’s facial emotions recognition method using transfer learning

An improved student’s facial emotions recognition method using transfer learning

Facial emotion recognition using deep quantum and advanced transfer learning mechanism.

Facial expressions have become a common way for interaction among humans. People cannot comprehend and predict the emotions or expressions of individuals through simple vision. Thus, in psychology, detecting facial expressions or emotion analysis demands an assessment and evaluation of decisions for identifying the emotions of a person or any group during communication. With the recent evolution of technology, AI (Artificial Intelligence) has gained significant usage, wherein DL (Deep Learning) based algorithms are employed for detecting facial expressions. The study proposes a system design that detects facial expressions by extracting relevant features using a Modified ResNet model. The proposed system stacks building-blocks with residual connections and employs an advanced extraction method with quantum computing, which significantly reduces computation time compared to conventional methods. The backbone stem utilizes a quantum convolutional layer comprised of several parameterized quantum-filters. Additionally, the research integrates residual connections in the ResNet-18 model with the Modified up Sampled Bottle Neck Process (MuS-BNP), retaining computational efficacy while benefiting from residual connections. The proposed model demonstrates superior performance by overcoming the issue of maximum similarity within varied facial expressions. The system's ability to accurately detect and differentiate between expressions is measured using performance metrics such as accuracy, F1-score, recall, and precision. This performance analysis confirms the efficacy of the proposed system, highlighting the advantages of quantum computing in feature extraction and the integration of residual connections. The model achieves quantum superiority, providing faster and more accurate computations compared to existing methodologies. The results suggest that the proposed approach offers a promising solution for facial expression recognition tasks, significantly improving both speed and accuracy.

Enhancing dance education through convolutional neural networks and blended learning.

This article explores the evolving landscape of dance teaching, acknowledging the transformative impact of the internet and technology. With the emergence of online platforms, dance education is no longer confined to physical classrooms but can extend to virtual spaces, facilitating a more flexible and accessible learning experience. Blended learning, integrating traditional offline methods and online resources, offers a versatile approach that transcends geographical and temporal constraints. The article highlights the utilization of the dual-wing harmonium (DWH) multi-view metric learning (MVML) algorithm for facial emotion recognition, enhancing the assessment of students' emotional expression in dance performances. Moreover, the integration of motion capture technology with convolutional neural networks (CNNs) facilitates a precise analysis of students' dance movements, offering detailed feedback and recommendations for improvement. A holistic assessment of students' performance is attained by combining the evaluation of emotional expression with the analysis of dance movements. Experimental findings support the efficacy of this approach, demonstrating high recognition accuracy and offering valuable insights into the effectiveness of dance teaching. By embracing technological advancements, this method introduces novel ideas and methodologies for objective evaluation in dance education, paving the way for enhanced learning outcomes and pedagogical practices in the future.

Impact of Bell's palsy in facial emotion recognition: a prospective observational study of 60 patients.

Patients with Bell's palsy suffer from functional deficits and cannot convey their emotions through the face as well as they used to. According to embodied cognition, automatic mimicry and facial feedback modulate emotion perception. The aim of our study was to determine the impact of Bell's palsy on facial emotion perception. Facial motor skills, the use of botulinum toxin injections, and anxiety were studied. 60 patients completed Emotest-VA, an assessment tool of facial emotion perception, facial motor assessment and affective questionnaires. Facial perception scores of patients were compared to the normative data provided by Emotest-VA. Relationships between facial motor skills, anxiety and perceptual abilities were carried out with Pearson's correlations. Perception accuracy scores were abnormal in 12% of patients. Patients were more ambiguous than normative population to perceive emotion (t(23) = 4.14, p < .001). Happiness arousal was negatively correlated to smile asymmetry (n = 60, r=-.294 p = .022). Patients who received botulinum toxin were more accurate to perceive disgust (Z = 3.60, p < .001). State anxiety decreased when patients could improve their horizontal smile (r = - 440, p < .001). Bell's palsy impaired sensorimotor simulation, thereby reducing emotional contagion. Smile measures could indicate patients' recovery and patients' anxiety. It could also predict patient's perceptual abilities. These results provide new directions in the management of Bell's palsy. Additionally to physical therapy, it is relevant to improve emotional contagion both in its productive and perceptive aspects through facial feedback.

Children with facial differences experience deficits in emotion skills.

Children with facial differences, such as repaired cleft lip and palate (CLP), may present with reduced capacity for sensorimotor simulation, particularly in the form of facial mimicry. This study examined whether facial mimicry, emotion recognition, and empathy skills are reduced in children with CLP when compared with sex/age-matched controls. A case-control design was utilized. Forty-five children between the ages of 8 and 12 with CLP, and 45 age/sex-matched controls were recruited. Participants completed a facial mimicry task, and facial movements were tracked and quantified using OpenFace. Participants also completed picture and context-based emotion recognition tasks. Picture-based assessment involved identifying emotions from the Dynamic FACES database. Context-based assessment consisted of identifying how a child might feel in various situations. Finally, participants and their parents completed the Empathy Questionnaire for Children and Adolescents (EmQue-CA). Children with CLP presented with significantly reduced facial mimicry (p = .017), picture-based (p < .001), and context-based emotion recognition scores (p < .001) when compared with controls. Better facial mimicry was associated with better picture-based emotion recognition scores in the control group only (r = .22, p < .01). Children with CLP also had significantly lower child and parent-proxy EmQue-CA scores (p < .001). Greater facial mimicry significantly predicted better parent-proxy EmQue-CA scores (p = .016) but did not predict child scores. Children with CLP presented with reduced facial mimicry, poorer emotion recognition, and empathy skills. These findings have a broader relevance as they suggest children with facial differences may present with reduced facial mimicry and/or deficits in emotion recognition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).