Facial Emotion Recognition Research Articles

PortraitEmotion3D: A Novel Dataset and 3D Emotion Estimation Method for Artistic Portraiture Analysis

Facial Expression Recognition (FER) has been widely explored in realistic settings; however, its application to artistic portraiture presents unique challenges due to the stylistic interpretations of artists and the complex interplay of emotions conveyed by both the artist and the subject. This study addresses these challenges through three key contributions. First, we introduce the PortraitEmotion3D (PE3D) dataset, designed explicitly for FER tasks in artistic portraits. This dataset provides a robust foundation for advancing emotion recognition in visual art. Second, we propose an innovative 3D emotion estimation method that leverages three-dimensional labeling to capture the nuanced emotional spectrum depicted in artistic works. This approach surpasses traditional two-dimensional methods by enabling a more comprehensive understanding of the subtle and layered emotions often in artistic representations. Third, we enhance the feature learning phase by integrating a self-attention module, significantly improving facial feature representation and emotion recognition accuracy in artistic portraits. This advancement addresses this domain’s stylistic variations and complexity, setting a new benchmark for FER in artistic works. Evaluation of the PE3D dataset demonstrates our method’s high accuracy and robustness compared to existing state-of-the-art FER techniques. The integration of our module yields an average accuracy improvement of over 1% in recent FER systems. Additionally, combining our method with ESR-9 achieves a comparable accuracy of 88.3% on the FER+ dataset, demonstrating its generalizability to other FER benchmarks. This research deepens our understanding of emotional expression in art and facilitates potential applications in diverse fields, including human–computer interaction, security, healthcare diagnostics, and the entertainment industry.

Drug Holidays May Attenuate Beneficial Effects of Treatment on Emotion Regulation and Recognition Among Children with ADHD: A Single-Center, Prospective Study.

In this study, we aimed to evaluate the effects of drug holidays during summer vacations on children with attention deficit/hyperactivity disorder (ADHD) treated with methylphenidate in terms of ADHD symptoms and emotion regulation (ER). This single-center, naturalistic, prospective cohort study included pre-, post-, and post-drug holiday evaluations. All patients diagnosed with ADHD and included in our study were administered the Clinical Global Impression Scale, Affective Reactivity Index-parent and child, reading the mind in the eyes test (RMET), Faces test and Cognitive Emotion Regulation Questionnaire for Children 3 times. Fifty-six patients met the inclusion criteria and ten were lost to follow-up. Thirty-nine participants had complete evaluations at all time points. Both parent and self-report ratings of child irritability were significantly reduced after methylphenidate (MPH) treatment (P=.003 and .002, respectively), although they returned to baseline after drug holidays (P=.618 and .974, respectively). The use of self-blame as a cognitive emotion regulation strategy increased significantly after treatment and remained significantly elevated even after drug holidays (P=.024 and .022, respectively). Children used planning as a cognitive strategy significantly more frequently during MPH treatment (P=.034), although this elevation was temporary and returned to baseline levels after the drug holidays (P=.890). Reading the mind in the eyes test performance was significantly improved after MPH treatment and did not change after drug holidays (P=.009 and .006, respectively), while there was a tendency for facial emotion recognition abilities to improve at the last visit (P=.051). Our results suggest that MPH treatment improves child- and parent-reported irritability, clinician-rated disorder severity, RMET performance, and the use of planning as a cognitive ER strategy, although only improvement in RMET performance continued after drug holidays. Our results may support the importance of continuing treatment over the holidays for most children with ADHD unless necessitated by adverse events, growth problems, or development of tolerance.

Deep-SFER: deep convolutional neural network and MFCC an effective speech and face emotion recognition

There has been a lot of progress in recent years in the fields of expert systems, artificial intelligence (AI) and human machine interface (HMI). The use of voice commands to engage with machinery or instruct it to do a certain task is becoming more common. Numerous consumer electronics have SIRI, Alexa, Cortana, and Google Assistant built in. In the field of human-device interaction, emotion recognition from speech is a complex research subject. We can't imagine modern life without machines, so naturally there's a need to create a more robust framework for human-machine communication. A number of academics are now working on speech emotion recognition (SER) in an effort to improve the interaction between humans and machines. We aimed to identify four fundamental emotions: angry, unhappy, neutral and joyful from speech in our experiment. As you can hear below, we trained and tested our model using audio data of brief Manipuri speeches taken from films. This task makes use of convolutional neural networks (CNNs) to extract functions from speech in order to recognize different moods using the Mel-frequency cepstral coefficient (MFCC).

Music Recommendation System Using Facial Emotion Gestures

Abstract: Music recommendation systems are transformingthe way users interact with music streaming servicesby personalizing experiences. This paper presents a novel approach that leverages facial emotion recognition (FER) to suggest music dynamically based on users' emotional states. Using machinelearning and deep learning models, facial expressions are analyzed in real-time through video input. The system integrates emotion-based recommendations with collaborative and content- based filtering techniques to enhance accuracy. Thisapproach offers a more intuitive and human-centered interaction compared to traditional systems. The proposed solution can be used across streaming platforms, personalized wellness apps, and therapeutic environments.

Real-Time Facial Emotion Detection Application with Image Processing Based on Convolutional Neural Network (CNN)

Facial Emotion Recognition (FER) is a key technology for identifying emotions based on facial expressions, with applications in human-computer interaction, mental health monitoring, and customer analysis. This study presents the development of a real-time emotion recognition system using Convolutional Neural Networks (CNNs) and OpenCV, addressing challenges such as varying lighting and facial occlusions. The system, trained on the FER2013 dataset, achieved 85% accuracy in emotion classification, demonstrating high performance in detecting happiness, sadness, and surprise. The results highlight the system's effectiveness in real-time applications, offering potential for use in mental health and customer behavior analysis.

Improved facial emotion recognition model based on a novel deep convolutional structure

Facial Emotion Recognition (FER) is a very challenging task due to the varying nature of facial expressions, occlusions, illumination, pose variations, cultural and gender differences, and many other aspects that cause a drastic degradation in quality of facial images. In this paper, an anti-aliased deep convolution network (AA-DCN) model has been developed and proposed to explore how anti-aliasing can increase and improve recognition fidelity of facial emotions. The AA-DCN model detects eight distinct emotions from image data. Furthermore, their features have been extracted using the proposed model and numerous classical deep learning algorithms. The proposed AA-DCN model has been applied to three different datasets to evaluate its performance: The Cohn-Kanade Extending (CK+) database has been utilized, achieving an ultimate accuracy of 99.26% in (5 min, 25 s), the Japanese female facial expressions (JAFFE) obtained 98% accuracy in (8 min, 13 s), and on one of the most challenging FER datasets; the Real-world Affective Face (RAF) dataset; reached 82%, in low training time (12 min, 2s). The experimental results demonstrate that the anti-aliased DCN model is significantly increasing emotion recognition while improving the aliasing artifacts caused by the down-sampling layers.

Age differences in mental state inference of sarcasm: Contributions of facial emotion recognition and cognitive performance.

Older age is associated with poorer ability to accurately infer mental states, but some mental states are more complex than others. Sarcasm is a complex mental state because the literal and intended meaning of a speaker's words are in opposition. Individuals must rely on additional cues (e.g., facial expressions, intonation) for accurate inference. We hypothesized that understanding of sarcastic versus sincere exchanges would be more sensitive to age-related difficulty in mental state understanding. We examined accuracy at identifying sarcasm among 263 adults (ages 18-90 years) using videos of social interactions in The Awareness of Social Inference Test. Hypotheses were tested using a logistic linear mixed effects model predicting correct/incorrect trial-level responses. To characterize why sarcasm differed with age, we measured two abilities commonly implicated in mental state understanding: facial emotion recognition and cognitive performance. Sarcasm understanding declined with age, whereas understanding of sincere exchanges did not. Both better emotion recognition and cognitive performance related to better understanding of sarcastic but not sincere exchanges. Only cognitive performance showed an age-related effect such that the cognitive performance among the oldest participants facilitated their understanding of both sarcastic and sincere exchanges. We showed that individual variation related to age and social and cognitive performance is more pronounced when the use of multiple mental state cues is more (sarcasm) or less (sincerity) necessary for accurate understanding of social interactions. Naturalistic paradigms involving multiple mental state cues can address important questions about how older adults make decisions in the real world.

Facial Emotion Recognition Using Residual Neural Networks

Facial Emotion Recognition Using Residual Neural Networks

Behavioral and neural evidence for difficulty recognizing masked emotional faces.

Facial emotion recognition is vital for human social behavior. During the COVID-19 pandemic, face masks were widely adopted for viral mitigation and remain crucial public health tools. However, questions persist about their impact on emotion recognition and neural processing, especially in children, parents, and young adults. We developed the Masked Affective and Social Cognition task, featuring masked and unmasked faces displaying fear, sadness, and anger. We recruited three racial and ethnically diverse samples: 119 college students, 30 children who entered school age at the beginning of the pandemic, and 31 fathers of the aforementioned children. Of the latter two groups, 41 participants (n = 23 fathers, 18 children) did the Masked Affective and Social Cognition task during a neuroimaging scan, while the remaining 20 participants (n = 8 fathers, 12 children) who were not eligible for scanning completed the task during their lab visit. Behaviorally, we found that participants recognized emotions less accurately when viewing masked faces and also found an interaction of emotion by condition, such that accuracy was particularly compromised by sad masked faces. Neurally, masked faces elicited greater activation in the posterior cingulate, insula, and fusiform gyrus. Anterior insula and inferior frontal gyrus activation were driven by sad, masked faces. These results were consistent across age groups. Among fathers, activation to sad masked faces was associated with stress and depression. Overall, our findings did not depend on previous mask exposure or timing of participation during the pandemic. These results have implications for understanding face emotion recognition, empathy, and socioemotional neurodevelopment. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Impairments in facial expression recognition in patients with euthymic bipolar disorders using the facial emotions recognition test

BackgroundPatients with bipolar disorders (BD) have significant impairments in Facial Expression Recognition (FER), an essential social skill for effective social interactions. While the Facial Emotions Recognition Test, 54 photographs by Gaudelus (TREF-54g) has been used in patients with schizophrenia, no study has evaluated FER using this test in BD patients. The TREF-54g meets the International Society for Bipolar Disorders (ISBD) criteria for assessing FER. The objective of this study was to compare FER in patients with euthymic BD versus healthy controls (HC) using the TREF-54g. MethodsThis study included 60 patients with BD diagnosed according to DSM-5 criteria, 30 with bipolar I disorder (BDI) and 30 with bipolar II disorder (BDII) and 60 HC. All groups were matched for age, gender, and years of education. Overall TREF-54g impairment score was defined as a difference of two SD from the group of HC with mean overall accuracy score expressed as a percentage of correct responses. ResultsOverall TREF-54g impairment scores, as well as sub-scores for anger and contempt recognition, were significantly higher in patients with BD versus those in HC. No significant differences in TREF-54g scores were shown between patients with BDI and those with BDII. LimitationsClinical assessments of this study didn't include neurocognition assessments and classification of contempt emotion as a basic emotion remains controversial. ConclusionsWe found significant impairments in FER in patients with euthymic BD, particularly for anger and contempt using the TREF-54g a test that meets ISBD recommendation criteria.

Disgust-specific impairment of facial emotion recognition in Parkinson's disease patients with mild cognitive impairment.

This study investigated the association between cognitive function and facial emotion recognition (FER) in patients with Parkinson's disease (PD) and mild cognitive impairment (PD-MCI). We enrolled 126 participants from Taiwan, including 63 patients with idiopathic PD and 63 matched healthy controls. The PD group was divided into two groups: those with normal cognitive function (PD-NC) and those with MCI (PD-MCI). Participants underwent a modality emotion recognition test and comprehensive cognitive assessment. Our findings reveal that patients with PD-MCI exhibit significantly impaired FER, especially in recognizing "disgust," compared with patients with PD-NC and healthy adults (P = .001). This deficit correlates with executive function, attention, memory, and visuospatial abilities. Attention mediates the relationship between executive function and "disgust" FER. The findings highlight how patients with PD-MCI are specifically challenged when recognizing "disgust" and suggest that cognitive training focusing on cognitive flexibility and attention may improve their FER abilities. This study contributes to our understanding of the nuanced relationship between cognitive dysfunction and FER in patients with PD-MCI, emphasizing the need for targeted interventions.

The impact of fear of pain on the temporal processing of emotional faces: modulation of attentional resources.

Recent research has focused on the spatial attention bias toward pain-related cues in individuals with fear of pain (FOP), but our understanding of its temporal attention features, particularly regarding emotional faces, is limited. To address this gap, 39 individuals with high fear of pain (H-FOP) and 37 with low fear of pain (L-FOP) completed a rapid serial visual presentation dual task. Participants viewed a series of rapidly presented pictures, first identified neutral building images as the first target (T1) and then emotional faces (painful, neutral, and happy) as the second target (T2). T1 and T2 presented at time intervals of 232ms (lag 2) and 696ms (lag 6), respectively. Results indicate that H-FOP group members were less accurate in recognizing faces at lag 2 shortly after correctly identifying T1 compared to those L-FOP group members. Notably, all participants were more accurate in recognizing painful faces at lag 2 than in identifying neutral and happy faces, suggesting a negative bias toward pain stimuli when attentional resources were limited. In contrast, at lag 6, when attentional resources were more abundant, participants showed greater accuracy in recognizing neutral faces than emotional faces. These findings underscore the influence of fear of pain and available cognitive resources on the attentional processing of emotional faces over time, providing insights into how fear of pain affects the temporal dynamics of emotional face recognition and the role of attentional resources.

An improved student’s facial emotions recognition method using transfer learning

An improved student’s facial emotions recognition method using transfer learning

Facial emotion recognition using deep quantum and advanced transfer learning mechanism.

Facial expressions have become a common way for interaction among humans. People cannot comprehend and predict the emotions or expressions of individuals through simple vision. Thus, in psychology, detecting facial expressions or emotion analysis demands an assessment and evaluation of decisions for identifying the emotions of a person or any group during communication. With the recent evolution of technology, AI (Artificial Intelligence) has gained significant usage, wherein DL (Deep Learning) based algorithms are employed for detecting facial expressions. The study proposes a system design that detects facial expressions by extracting relevant features using a Modified ResNet model. The proposed system stacks building-blocks with residual connections and employs an advanced extraction method with quantum computing, which significantly reduces computation time compared to conventional methods. The backbone stem utilizes a quantum convolutional layer comprised of several parameterized quantum-filters. Additionally, the research integrates residual connections in the ResNet-18 model with the Modified up Sampled Bottle Neck Process (MuS-BNP), retaining computational efficacy while benefiting from residual connections. The proposed model demonstrates superior performance by overcoming the issue of maximum similarity within varied facial expressions. The system's ability to accurately detect and differentiate between expressions is measured using performance metrics such as accuracy, F1-score, recall, and precision. This performance analysis confirms the efficacy of the proposed system, highlighting the advantages of quantum computing in feature extraction and the integration of residual connections. The model achieves quantum superiority, providing faster and more accurate computations compared to existing methodologies. The results suggest that the proposed approach offers a promising solution for facial expression recognition tasks, significantly improving both speed and accuracy.

Enhancing dance education through convolutional neural networks and blended learning

This article explores the evolving landscape of dance teaching, acknowledging the transformative impact of the internet and technology. With the emergence of online platforms, dance education is no longer confined to physical classrooms but can extend to virtual spaces, facilitating a more flexible and accessible learning experience. Blended learning, integrating traditional offline methods and online resources, offers a versatile approach that transcends geographical and temporal constraints. The article highlights the utilization of the dual-wing harmonium (DWH) multi-view metric learning (MVML) algorithm for facial emotion recognition, enhancing the assessment of students’ emotional expression in dance performances. Moreover, the integration of motion capture technology with convolutional neural networks (CNNs) facilitates a precise analysis of students’ dance movements, offering detailed feedback and recommendations for improvement. A holistic assessment of students’ performance is attained by combining the evaluation of emotional expression with the analysis of dance movements. Experimental findings support the efficacy of this approach, demonstrating high recognition accuracy and offering valuable insights into the effectiveness of dance teaching. By embracing technological advancements, this method introduces novel ideas and methodologies for objective evaluation in dance education, paving the way for enhanced learning outcomes and pedagogical practices in the future.

Impact of Bell's palsy in facial emotion recognition: a prospective observational study of 60 patients.

Patients with Bell's palsy suffer from functional deficits and cannot convey their emotions through the face as well as they used to. According to embodied cognition, automatic mimicry and facial feedback modulate emotion perception. The aim of our study was to determine the impact of Bell's palsy on facial emotion perception. Facial motor skills, the use of botulinum toxin injections, and anxiety were studied. 60 patients completed Emotest-VA, an assessment tool of facial emotion perception, facial motor assessment and affective questionnaires. Facial perception scores of patients were compared to the normative data provided by Emotest-VA. Relationships between facial motor skills, anxiety and perceptual abilities were carried out with Pearson's correlations. Perception accuracy scores were abnormal in 12% of patients. Patients were more ambiguous than normative population to perceive emotion (t(23) = 4.14, p < .001). Happiness arousal was negatively correlated to smile asymmetry (n = 60, r=-.294 p = .022). Patients who received botulinum toxin were more accurate to perceive disgust (Z = 3.60, p < .001). State anxiety decreased when patients could improve their horizontal smile (r = - 440, p < .001). Bell's palsy impaired sensorimotor simulation, thereby reducing emotional contagion. Smile measures could indicate patients' recovery and patients' anxiety. It could also predict patient's perceptual abilities. These results provide new directions in the management of Bell's palsy. Additionally to physical therapy, it is relevant to improve emotional contagion both in its productive and perceptive aspects through facial feedback.

Children with facial differences experience deficits in emotion skills.

Children with facial differences, such as repaired cleft lip and palate (CLP), may present with reduced capacity for sensorimotor simulation, particularly in the form of facial mimicry. This study examined whether facial mimicry, emotion recognition, and empathy skills are reduced in children with CLP when compared with sex/age-matched controls. A case-control design was utilized. Forty-five children between the ages of 8 and 12 with CLP, and 45 age/sex-matched controls were recruited. Participants completed a facial mimicry task, and facial movements were tracked and quantified using OpenFace. Participants also completed picture and context-based emotion recognition tasks. Picture-based assessment involved identifying emotions from the Dynamic FACES database. Context-based assessment consisted of identifying how a child might feel in various situations. Finally, participants and their parents completed the Empathy Questionnaire for Children and Adolescents (EmQue-CA). Children with CLP presented with significantly reduced facial mimicry (p = .017), picture-based (p < .001), and context-based emotion recognition scores (p < .001) when compared with controls. Better facial mimicry was associated with better picture-based emotion recognition scores in the control group only (r = .22, p < .01). Children with CLP also had significantly lower child and parent-proxy EmQue-CA scores (p < .001). Greater facial mimicry significantly predicted better parent-proxy EmQue-CA scores (p = .016) but did not predict child scores. Children with CLP presented with reduced facial mimicry, poorer emotion recognition, and empathy skills. These findings have a broader relevance as they suggest children with facial differences may present with reduced facial mimicry and/or deficits in emotion recognition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A facial and vocal emotion recognition system based on deep learning

The traditional teaching mode, mainly characterized by "cramming" instruction, often overlooks the absorption of knowledge by students during the teaching process. Even in recent years, the emerging "flipped classroom" model heavily relies on teachers' experience and expertise, making it challenging to quantify students' classroom learning states. Among young teachers lacking teaching experience, the replication of excellent teaching models faces difficulties. With the rapid development of artificial intelligence technology, deep learning and other AI techniques are increasingly applied to people's lives. Additionally, emotion computing technology is also becoming more sophisticated. In the field of education, the use of AI technology holds the potential to help teachers systematically and comprehensively assess teaching quality, promoting the sharing of high-quality educational resources nationally and globally. Existing emotion detection technologies often focus on single techniques such as image recognition or natural language processing, resulting in low accuracy in discerning human emotions and making it difficult to accurately identify emotional changes in complex scenarios such as classrooms. Furthermore, there is a lack of comprehensive emotional statistical analysis methods for audio and visual data. In light of these, we have designed and implemented a multi-modal emotion recognition and analysis system. Through testing in classroom settings, this system has the potential to assist classroom teaching and enhance teaching quality.

Evaluating the Impact of BoNT-A Injections on Facial Expressions: A Deep Learning Analysis.

Botulinum Toxin Type A (BoNT-A) injections are widely used for facial rejuvenation, but their effects on facial expressions remain unclear. This study aims to objectively measure the impact of BoNT-A injections on facial expressions using deep learning techniques. 180 patients aged 25-60 years who underwent BoNT-A application to the upper face were included. Patients were photographed with neutral, happy, surprised, and angry expressions before and 14 days after the procedure. A Convolutional Neural Network (CNN)-based Facial Emotion Recognition (FER) system analyzed 1440 photographs using a hybrid dataset of clinical images and the Karolinska Directed Emotional Faces (KDEF) dataset. The CNN model accurately predicted 90.15% of the test images. Significant decreases in the recognition of angry and surprised expressions were observed post-injection (p<0.05), with no significant changes in happy and neutral expressions (p>0.05). Angry expressions were often misclassified as neutral or happy (p<0.05), and surprised expressions were more likely to be perceived as neutral (p<0.05). Deep learning can effectively assess the impact of BoNT-A injections on facial expressions, providing more standardized data than traditional surveys. BoNT-A may reduce the expression of anger and surprise, potentially leading to a more positive facial appearance and emotional state. Further studies are needed to understand the broader implications of these changes.

Effects of Sleep Quality, Acute Sleep Deprivation, and Napping on Facial Emotion Recognition Accuracy and Speed.

To investigate the effects of sleep quality, sleep deprivation, and napping on facial emotion recognition (FER) accuracy and speed. This research included a cross-sectional study (102 qualified participants) and a randomized controlled study (26 in the napping group and 24 in the control group). The stimuli for the FER task were obtained from the Chinese Facial Affective Picture System (CFAPS). Four facial expressions (fearful, disgusted, sad, and angry) were used. The Pittsburgh Sleep Quality Index (PSQI), Self-Rating Anxiety Scale, and Self-Rating Depression Scale were used to measure participants' sleep quality and psychological conditions. In Study 1, FER ability was compared between good and poor sleepers. In Study 2, all participants were sleep-deprived for one night, and completed the FER task before and after sleep deprivation. After different interventions (ie, napping for one hour, or walking around for ten minutes), the participants completed the third FER task. Study 1: Poor sleepers were able to recognize sad expressions more accurately compared with good sleepers. Study 2: 30-h sleep deprivation had no significant effect on the accuracy (ACC). Napping after sleep deprivation improved the FER ACC of upper-face expressions and marginally significantly improved the FER ACC of disgusted expressions. Better sleep quality was linked to lower FER accuracy, particularly in recognizing sad expressions, while no significant differences in recognition speed were observed. Additionally, 30hours of sleep deprivation did not affect FER accuracy, but napping after sleep deprivation improved accuracy for upper-face and marginally for disgusted expressions.