Facial Emotion Recognition Research Articles

RF‐GCN: Residual fused‐graph convolutional network using multimodalities for facial emotion recognition

AbstractBackgroundThe emotional state of individuals is difficult to identify and it is developing now a days because of vast interest in recognition. Many technologies have been developed to identify this emotional expression based on facial expressions, vocal expressions, physiological signals, and body expressions. Among these, facial emotion is very expressive for recognition using multimodalities. Understanding facial emotions has applications in mental well‐being, decision‐making, and even social change, as emotions play a crucial role in our lives. This recognition is complicated by the high dimensionality of data and non‐linear interactions across modalities. Moreover, the way emotion is expressed by people varies and these feature identification remains challenging, where these limitations are overcome by Deep learning models.MethodsThis research work aims at facial emotion recognition through the utilization of a deep learning model, named the proposed Residual Fused‐Graph Convolution Network (RF‐GCN). Here, multimodal data included is video as well as an Electroencephalogram (EEG) signal. Also, the Non‐Local Means (NLM) filter is used for pre‐processing input video frames. Here, the feature selection process is carried out using chi‐square, after feature extraction, which is done in both pre‐processed video frames and input EEG signals. Finally, facial emotion recognition and its types are determined by RF‐GCN, which is a combination of both the Deep Residual Network (DRN) and Graph Convolutional Network (GCN).ResultsFurther, RF‐GCN is evaluated for performance by metrics such as accuracy, recall, and precision, with superior values of 91.6%, 96.5%, and 94.7%.ConclusionsRF‐GCN captures the nuanced relationships between different emotional states and improves recognition accuracy. The model is trained and evaluated on the dataset and reflects real‐world conditions.

Digital assessment of nonverbal behaviors forecasts first onset of depression.

Adolescence is marked by a sharp increase in the incidence of depression, especially in females. Identification of risk for depressive disorders (DD) in this key developmental stage can help prevention efforts, mitigating the clinical and public burden of DD. While frequently used in diagnosis, nonverbal behaviors are relatively understudied as risk markers for DD. Digital technology, such as facial recognition, may provide objective, fast, efficient, and cost-effective means of measuring nonverbal behavior. Here, we analyzed video-recorded clinical interviews of 359 never-depressed adolescents females via commercially available facial emotion recognition software. We found that average head and facial movements forecast future first onset of depression (AUC = 0.70) beyond the effects of other established self-report and physiological markers of DD risk. Overall, these findings suggest that digital assessment of nonverbal behaviors may provide a promising risk marker for DD, which could aid in early identification and intervention efforts.

The effect of aging on the dual-route model of emotion processing applied to memory recognition.

Emotional faces are automatically processed in the human brain through a cortical route (conscious processing based on high spatial frequencies, HSF) and a subcortical route (subliminal processing based on low spatial frequencies, LSF). How each route contributes to emotional face recognition is still debated, and little is known about this process in aging. Here, 147 younger adults (YA) and 137 older adults (OA) were passively presented with neutral, happy, and angry faces, shown as (a) unfiltered, (b) filtered at LSF, and (c) hybrid (emotional LSF superimposed to the neutral HSF of the same face). In a succeeding recognition phase, the same faces and new faces were shown as unfiltered, and participants were asked whether each face had been already presented in the encoding phase. Despite the better performance by YA compared with OA for neutral faces presented as unfiltered (cortical route), the performance of OA was better than that of YA for angry faces presented as hybrid and for happy faces presented at LSF and as hybrid. We conclude that the activity of the subcortical route during the encoding phase facilitates emotional recognition in aging. Results are discussed in accordance with the dual-route model. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Accuracy is not enough: a heterogeneous ensemble model versus FGSM attack

AbstractIn this paper, based on facial landmark approaches, the possible vulnerability of ensemble algorithms to the FGSM attack has been assessed using three commonly used models: convolutional neural network-based antialiasing (A_CNN), Xc_Deep2-based DeepLab v2, and SqueezeNet (Squ_Net)-based Fire modules. Firstly, the three individual deep learning classifier-based Facial Emotion Recognition (FER) classifications have been developed; the predictions from all three classifiers are then merged using majority voting to develop the HEM_Net-based ensemble model. Following that, an in-depth investigation of their performance in the case of attack-free has been carried out in terms of the Jaccard coefficient, accuracy, precision, recall, F1 score, and specificity. When applied to three benchmark datasets, the ensemble-based method (HEM_Net) significantly outperforms in terms of precision and reliability while also decreasing the dimensionality of the input data, with an accuracy of 99.3%, 87%, and 99% for the Extended Cohn-Kanade (CK+), Real-world Affective Face (RafD), and Japanese female facial expressions (Jaffee) data, respectively. Further, a comprehensive analysis of the drop in performance of every model affected by the FGSM attack is carried out over a range of epsilon values (the perturbation parameter). The results from the experiments show that the advised HEM_Net model accuracy declined drastically by 59.72% for CK + data, 42.53% for RafD images, and 48.49% for the Jaffee dataset when the perturbation increased from A to E (attack levels). This demonstrated that a successful Fast Gradient Sign Method (FGSM) can significantly reduce the prediction performance of all individual classifiers with an increase in attack levels. However, due to the majority voting, the proposed HEM_Net model could improve its robustness against FGSM attacks, indicating that the ensemble can lessen deception by FGSM adversarial instances. This generally holds even as the perturbation level of the FGSM attack increases.

Individual differences in interoception and autistic traits share altered facial emotion perception, but not recognition per se

While alterations in both physiological responses to others’ emotions as well as interoceptive abilities have been identified in autism, their relevance in altered emotion recognition is largely unknown. We here examined the role of interoceptive ability, facial mimicry, and autistic traits in facial emotion processing in non-autistic individuals. In an online Experiment 1, participants (N = 99) performed a facial emotion recognition task, including ratings of perceived emotional intensity and confidence in emotion recognition, and reported on trait interoceptive accuracy, interoceptive sensibility and autistic traits. In a follow-up lab Experiment 2 involving 100 participants, we replicated the online experiment and additionally investigated the relationship between facial mimicry (measured through electromyography), cardiac interoceptive accuracy (evaluated using a heartbeat discrimination task), and autistic traits in relation to emotion processing. Across experiments, neither interoception measures nor facial mimicry accounted for a reduced recognition of specific expressions with higher autistic traits. Higher trait interoceptive accuracy was rather associated with more confidence in correct recognition of some expressions, as well as with higher ratings of their perceived emotional intensity. Exploratory analyses indicated that those higher intensity ratings might result from a stronger integration of instant facial muscle activations, which seem to be less integrated in intensity ratings with higher autistic traits. Future studies should test whether facial muscle activity, and physiological signals in general, are correspondingly less predictive of perceiving emotionality in others in individuals on the autism spectrum, and whether training interoceptive abilities might facilitate the interpretation of emotional expressions.

Social Cognition in Parkinson's Disease after Focused Ultrasound Subthalamotomy: A Controlled Study.

Social cognition (SC) encompasses a set of cognitive functions that enable individuals to understand and respond appropriately to social interactions. Although focused ultrasound subthalamotomy (FUS-STN) effectively treats Parkinson's disease (PD) clinical motor features, its impact and safety on cognitive-behavioral interactions/interpersonal awareness are unknown. This study investigated the effects of unilateral FUS-STN on facial emotion recognition (FER) and affective and cognitive theory of mind (ToM) in PD patients from a randomized sham-controlled trial (NCT03454425). Subjects performed SC evaluation before and 4 months after the procedure while still under blind assessment conditions. The SC assessment included the Karolinska Directed Emotional Faces task for FER, the Reading the Mind in the Eyes (RME) test for affective ToM, and The Theory of Mind Picture Stories Task (ToM PST) (order, questions, and total score) for cognitive ToM. The active treatment group showed anecdotal-to-moderate evidence of no worsening in SC after FUS-STN. Anecdotal evidence for an improvement was recognized in the SC score changes, from baseline to post-treatment, for the active treatment group compared with sham for the RME, ToM PST order, ToM PST total, FER total, and recognition of fear, disgust, and anger. This study provides the first evidence that unilateral FUS-STN does not impair social cognitive abilities, indicating that it can be considered a safe treatment approach for this domain in PD patients. Furthermore, the results suggest FUS-STN may even lead to some improvement in social cognitive outcomes, which should be considered as a preliminary finding requiring further investigation with larger samples sizes. © 2024 International Parkinson and Movement Disorder Society.

Exploring adversarial deep learning for fusion in multi-color channel skin detection applications

Deep learning, a robust framework for complex learning, outperforms previous machine learning approaches and finds widespread use. However, security vulnerabilities, especially in fusion in multi-color channel skin detection applications using adversarial machine learning (AML) and generative adversarial networks (GANs), lead to misclassifications. Researchers are actively exploring AML's and GANs' impact on misclassification, focusing on vulnerabilities in lighting conditions, skin-like patches in lesion segmentation, and insufficient data in facial emotion recognition. Yet, these areas only scratch the surface of potential AML vulnerabilities and GANs. To comprehensively address challenges, an in-depth investigation into AML and GANs components is crucial to uncover underlying reasons for misclassifying skin detection. This study addresses challenges of fusion in multi-color channel skin detection by creating a diverse dataset with 17M patches for enhanced feature fusion/training and meeting dataset criteria, investigating misclassifications using various deep learning models belonging to AML and GANs and color spaces (e.g., RGB, YCbCr, HSV, YUV), and exploring binary and multiclass scenarios. Notably, YCbCr outperformed RGB, achieving 98 % for binary skin classification, 84 % and 69 % for multiclass four and five-class scenarios. Binary classification for skin tones and their skin-like counterparts (e.g., black skin tone and black skin-like) yielded 97 %, 81 %, 60 %, and 51 % for black, brown, medium, and fair, respectively. Exploration of darker skin tones showed improved accuracy. Benchmarking with a CNN and RNN hybrid achieved 99 % accuracy, surpassing the initial 91 %, while SAE reached 97 %. The study explores implications of overlapping between skin and skin-tone recognition, offering insights for developing a generalized skin detector. The investigation demonstrates that improper color space selection can make lighting conditions exploitable in AML attacks and GANs, emphasizing the crucial role of color space choice in mitigating vulnerabilities.

RS-Xception: A Lightweight Network for Facial Expression Recognition

Facial expression recognition (FER) utilizes artificial intelligence for the detection and analysis of human faces, with significant applications across various scenarios. Our objective is to deploy the facial emotion recognition network on mobile devices and extend its application to diverse areas, including classroom effect monitoring, human–computer interaction, specialized training for athletes (such as in figure skating and rhythmic gymnastics), and actor emotion training. Recent studies have employed advanced deep learning models to address this task, though these models often encounter challenges like subpar performance and an excessive number of parameters that do not align with the requirements of FER for embedded devices. To tackle this issue, we have devised a lightweight network structure named RS-Xception, which is straightforward yet highly effective. Drawing on the strengths of ResNet and SENet, this network integrates elements from the Xception architecture. Our models have been trained on FER2013 datasets and demonstrate superior efficiency compared to conventional network models. Furthermore, we have assessed the model’s performance on the CK+, FER2013, and Bigfer2013 datasets, achieving accuracy rates of 97.13%, 69.02%, and 72.06%, respectively. Evaluation on the complex RAF-DB dataset yielded an accuracy rate of 82.98%. The incorporation of transfer learning notably enhanced the model’s accuracy, with a performance of 75.38% on the Bigfer2013 dataset, underscoring its significance in our research. In conclusion, our proposed model proves to be a viable solution for precise sentiment detection and estimation. In the future, our lightweight model may be deployed on embedded devices for research purposes.

Dimensions of Alexithymia and Identification of Emotions in Masked and Unmasked Faces.

Alexithymia, a multifaceted personality construct, is known to be related to difficulties in the decoding of emotional facial expressions, especially in case of suboptimal stimuli. The present study investigated whether and which facets of alexithymia are related to impairments in the recognition of emotions in faces with face masks. Accuracy and speed of emotion recognition were examined in a block of faces with and a block of faces without face masks in a sample of 102 healthy individuals. The order of blocks varied between participants. Emotions were recognized better and faster in unmasked than in masked faces. Recognition performance was worst and slowest for participants starting the task with masked faces. In the whole sample, there were no correlations of alexithymia facets with accuracy and speed of emotion recognition for masked and unmasked faces. In participants starting the task with masked faces, the facet externally oriented thinking was positively correlated with reaction latencies of correct responses for masked faces. Our findings indicate that an externally oriented thinking style could be linked to a less efficient identification of emotions from faces wearing masks when task difficulty is high and support the utility of a facet approach in alexithymia research.

Facial and Emotion Recognition Deficits in Myasthenia Gravis.

Myasthenia gravis (MG) is a neuromuscular disease of autoimmune etiology and chronic evolution. In addition to the muscle weakness and fatigue that characterize MG, in some studies patients show an inferior performance in cognitive tasks and difficulties in recognizing basic emotions from facial expressions. However, it remains unclear if these difficulties are due to anxious-depressive symptoms that these patients present or related to cognitive abilities, such as facial recognition. This study had a descriptive cross-sectional design with a sample of 92 participants, 52 patients with MG and 40 healthy controls. The data collection protocol included measures to assess recognition of facial expressions (BRFT), facial emotional expression (FEEL), and levels of anxiety and depression (HADS). The MG group had worse performance than the control group in recognizing "fear" (p = 0.001; r = 0.344), "happiness" (p = 0.000; r = 0.580), "disgust" (p = 0.000; r = 0.399), "surprise" (p = 0.000; r = 0.602), and "anger" (p = 0.007; r = 0.284). Likewise, the MG group also underperformed in facial recognition (p = 0.001; r = 0.338). These difficulties were not related to their levels of anxiety and depression. Alterations were observed both in the recognition of facial emotions and in facial recognition, without being mediated by emotional variables. These difficulties can influence the interpersonal interaction of patients with MG.

Facial Emotion Recognition with Deep Neural Network: A Study of Visual Geometry Group-16 (VGG16) Technique with Data Augmentation for Improved Precision

Emotions play a significant role in both verbal and nonverbal communication. Facial emotion recognition has applications in various sectors where we can get real-time feedback about student activeness by detecting their expression. In this paper, we aim to provide an improved deep-learning technique to detect emotions by using publicly available datasets to perform this detection. To get more data for the well-being of the Machine Learning Model, we have used data augmentation using the TensorFlow framework. Visual Geometry Group-16 (VGG16) is a convolutional neural network of 16 layers deep. There has been an alteration to the default VGG16 structure to get better classification results. Various optimization algorithms and loss functions increase the model’s accuracy. We have used many evaluation parameters from the technical side, like precision, accuracy, recall, Area Under the Receiver Operating Characteristic Curve (AUC), and F1 Score. The proposed model has an accuracy of 89% while having a precision of 81 percent for classification. We have achieved an F1 Score of 0.42 and an area under the ROC curve (AUC) of 0.734. Overall, it would be beneficial for analyzing and categorizing positive and negative emotions, which would aid in detecting signs of stress, anxiety, and burnout, as well as taking preventive actions to enhance well-being.

Advancing Virtual Interviews: AI-Driven Facial Emotion Recognition for Better Recruitment

Behavior analysis involves the detailed process of identifying, modeling, and comprehending the various nuances and patterns of emotional expressions exhibited by individuals. It poses a significant challenge to accurately detect and predict facial emotions, especially in contexts like remote interviews, which have become increasingly prevalent. Notably, many participants struggle to convey their thoughts to interviewers with a happy expression and good posture, which may unfairly diminish their chances of employment, despite their qualifications. To address this challenge, artificial intelligence techniques such as image classification offer promising solutions. By leveraging AI models, behavior analysis can be applied to perceive and interpret facial reactions, thereby paving the way to anticipate future behaviors based on learned patterns to the participants. Despite existing works on facial emotion recognition (FER) using image classification, there is limited research focused on platforms like remote interviews and online courses. In this paper, our primary focus lies on emotions such as happiness, sadness, anger, surprise, eye contact, neutrality, smile, confusion, and stooped posture. We have curated our dataset, comprising a diverse range of sample interviews captured through participants' video recordings and other images documenting facial expressions and speech during interviews. Additionally, we have integrated existing datasets such as FER 2013 and the Celebrity Emotions dataset. Through our investigation, we explore a variety of AI and deep learning methodologies, including VGG19, ResNet50V2, ResNet152V2, Inception-ResNetV2, Xception, EfficientNet B0, and YOLO V8 to analyze facial patterns and predict emotions. Our results demonstrate an accuracy of 73% using the YOLO v8 model. However, we discovered that the categories of happy and smile, as well as surprised and confused, are not disjoint, leading to potential inaccuracies in classification. Furthermore, we considered stooped posture as a non-essential class since the interviews are conducted via webcam, which does not allow for the observation of posture. By removing these overlapping categories, we achieved a remarkable accuracy increase to around 76.88% using the YOLO v8 model.

Relationship between severity of hypomimia and basic emotion recognition in Parkinson's disease

Parkinson's disease is characterised by the presence of motor symptoms including hypomimia, and by non-motor symptoms including alterations in facial recognition of basic emotions. Few studies have investigated this alteration and its relationship to the severity of hypomimia. The objective is to study the relationship between hypomimia and the facial recognition of basic emotions in subjects with Parkinson's disease. Twenty-three patients and 29 controls were evaluated with the test battery for basic emotion facial recognition. The patients were divided into two subgroups according to the intensity of their hypomimia. The comparison in battery test performance between the minimal/mild hypomimia and moderate/severe hypomimia groups was statistically significant in favour of the former group. This finding shows a close relationship between expression and facial recognition of emotions, which could be explained through the mechanism of motor simulation.

Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition.

Demographic biases in source datasets have been shown as one of the causes of unfairness and discrimination in the predictions of Machine Learning models. One of the most prominent types of demographic bias are statistical imbalances in the representation of demographic groups in the datasets. In this article, we study the measurement of these biases by reviewing the existing metrics, including those that can be borrowed from other disciplines. We develop a taxonomy for the classification of these metrics, providing a practical guide for the selection of appropriate metrics. To illustrate the utility of our framework, and to further understand the practical characteristics of the metrics, we conduct a case study of 20 datasets used in Facial Emotion Recognition (FER), analyzing the biases present in them. Our experimental results show that many metrics are redundant and that a reduced subset of metrics may be sufficient to measure the amount of demographic bias. The article provides valuable insights for researchers in AI and related fields to mitigate dataset bias and improve the fairness and accuracy of AI models.

A Comparison of CNN-based 2D Static Facial Emotion Recognition Techniques

A Comparison of CNN-based 2D Static Facial Emotion Recognition Techniques

Facial emotion recognition in sexual offenders

Facial emotion recognition in sexual offenders

Recognition of Facial Expression with the Help of IoT, AI and Robotics

The emerging field of "Smart Face Recognition" utilizes IoT and machine learning to accurately identify individuals based on their facial characteristics. Various industries such as security, retail, and healthcare are leveraging this technology to enhance customer satisfaction and increase productivity. By combining IoT and machine learning, large amounts of data can be collected from multiple sources, such as cameras and sensors, and used to train algorithms for real-time, precise identification of individuals. This technology is gaining popularity due to its accuracy, speed, and scalability, making it essential for applications like security and access control. Recognizing human facial emotions is a key focus in today's technological landscape, with robotic applications across various sectors highlighting the importance of emotion recognition for effective human-robot interaction. This project aims to develop and implement a new automated system for emotion detection and facial recognition using Artificial Intelligence (AI) and the Internet of Things (IoT).

Advanced Techniques for Real-time Facial Expression Recognition Using Deep Learning

Abstract—Developing systems that can automatically recognize and interpret human emotions from facial expressions is the aim of the quickly expanding field of facial emotion identification and detection research. This technology finds applications in a wide range of areas, including as healthcare, marketing, security, and human-computer interface. Using computer vision and machine learning algorithms, facial emotion recognition systems analyze a face’s features and classify it into numerous emotional categories, including joyful, sorrowful, angry, fearful, and surprised.The three steps in the multi-step process that goes into identifying facial emotions are face detection, facial feature extraction, and emotion categorization. Thanks to recent advances in deep learn- ing, facial emotion detection systems can now identify emotions with high resilience and precision. Further, the development of real-time face expression recognition systems has opened up new avenues for applications such as sentiment analysis, emotional intelligence, and affective computing. This technology could fundamentally alter human-machine interactions and open the way to more compassionate and personalized relationships.A multitude of applications, including virtual assistants, mental health aids, and human-centered technology, will be greatly impacted by the development of systems for identifying and de- tecting facial expressions. Artificial intelligence (AI) technologies that recognize emotions allow people to interact with the digital world in more intelligent and flexible ways. But the complexity of emotion identification lies in the fact that it requires context and geometric elements in addition to facial expressions.

Deficits of Facial Emotion Recognition in Elderly Individuals with Mild Cognitive Impairment.

The study of facial emotion recognition is under-explored in subjects with mild cognitive impairment (MCI). We investigated whether deficits in facial emotion recognition are present in patients with MCI. We also analyzed the relationship between facial emotion recognition and different domains of cognitive function. This study included 300 participants aged 60 years or older with cognitive decline. We evaluated 181 MCI and 119 non-MCI subjects using the Seoul Neuropsychological Screening Battery-Core (SNSB-C) and facial emotion recognition task using six facial expressions (anger, disgust, fear, happiness, sadness and surprise). A Generalized Linear Model (GLM) was used to assess the association between cognitive performance and accuracy of facial emotion recognition and to compare facial emotion recognition in the MCI group based on the impairment of five different domains of cognitive function. The model was adjusted for age, sex, years of education, and depressive symptoms. Patients with MCI had a lower score for accurately recognizing total facial emotion (0.48 vs. 0.53; ρ = 0.0003) and surprise (0.73 vs. 0.81; ρ = 0.0215) when compared to cognitively healthy subjects. We also discovered that frontal/executive function domain (Digit Symbol Coding [DSC, 0.38 vs. 0.49; p < 0.0001], Controlled Oral Word Association Test [COWAT, 0.42 vs. 0.49; p = 0.0001], Korean-Trail Making Test [K-TMT, 0.37 vs. 0.48; p = 0.0073], Korean-Color Word Stroop Test [K-CWST, 0.43 vs. 0.49; p = 0.0219]) and language domain (Korean-Boston Naming Test [S-K-BNT, 0.46 vs. 0.47; p = 0.003]) were statistically associated with the deficits of facial emotion recognition in patients with MCI. We observed a significant association between deficits in facial emotion recognition and cognitive impairment in elderly individuals.

HTNet for micro-expression recognition

Facial expression is related to facial muscle contractions and different muscle movements correspond to different emotional states. For micro-expression recognition, the muscle movements are usually subtle, which has a negative impact on the performance of current facial emotion recognition algorithms. Most existing methods use self-attention mechanisms to capture relationships between tokens in a sequence, but they do not take into account the inherent spatial relationships between facial landmarks. This can result in sub-optimal performance on micro-expression recognition tasks. Therefore, learning to recognize facial muscle movements is a key challenge in the area of micro-expression recognition. In this paper, we propose a Hierarchical Transformer Network (HTNet) to identify critical areas of facial muscle movement. HTNet includes two major components: a transformer layer that leverages the local temporal features and an aggregation layer that extracts local and global semantical facial features. Specifically, HTNet divides the face into four different facial areas: left lip area, left eye area, right eye area and right lip area. The transformer layer is used to focus on representing local minor muscle movement with local self-attention in each area. The aggregation layer is used to learn the interactions between eye areas and lip areas. The experiments on four publicly available micro-expression datasets show that the proposed approach outperforms previous methods by a large margin. The codes and models are available at: https://github.com/wangzhifengharrison/HTNet.