Facial Emotion Recognition Research Articles

Facial emotion recognition: A comprehensive review

AbstractFacial emotion recognition (FER) represents a significant outcome of the rapid advancements in artificial intelligence (AI) technology. In today's digital era, the ability to decipher emotions from facial expressions has evolved into a fundamental mode of human interaction and communication. As a result, FER has penetrated diverse domains, including but not limited to medical diagnosis, customer feedback analysis, the automation of automobile driver systems, and the evaluation of student comprehension. Furthermore, it has matured into a captivating and dynamic research field, capturing the attention and curiosity of contemporary scholars and scientists. The primary objective of this paper is to provide an exhaustive review of FER systems. Its significance goes beyond offering a comprehensive resource; it also serves as a valuable guide for emerging researchers in the FER domain. Through a meticulous examination of existing FER systems and methodologies, this review equips them with essential insights and guidance for their future research pursuits. Moreover, this comprehensive review contributes to the expansion of their knowledge base, facilitating a profound understanding of this rapidly evolving field. In a world increasingly dependent on technology for communication and interaction, the study of FER holds a pivotal role in human‐computer interaction (HCI). It not only provides valuable insights but also unlocks a multitude of possibilities for future innovations and applications. As we continue to integrate AI and facial emotion recognition into our daily lives, the importance of comprehending and enhancing FER systems becomes increasingly evident. This paper serves as a stepping stone for researchers, nurturing their involvement in this exciting and ever‐evolving field.

Facial Emotion Recognition System Using CNN for Song Mapping

This project introduces a novel system that integrates Facial Emotion Recognition (FER) with music mapping to enhance human-computer interaction. Leveraging a custom Convolutional Neural Network (CNN) algorithm, developed using the FER2013 dataset, we classify emotions into four categories- happy, sad, neutral, and angry- employing Haar Cascade for precise face detection and grayscale conversion for optimal CNN input. Our custom CNN demonstrates superior performance, achieving a testing accuracy of 77.23%, notably surpassing established models like VGG16 and ResNet50, which achieved 55.87% and 62.76% respectively. This system swiftly identifies emotions and recommends songs from the "278k Emotion Labeled Spotify Songs" playlist, aiming to boost user satisfaction. Through nuanced comparisons with VGG16 and ResNet50, our approach underscores its inherent strengths, suggesting promising advancements in Facial Emotion Recognition (FER) and music recommendation systems. With a focus on precision in emotion detection, our system subtly elevates user experience, contributing meaningfully to ongoing research in the field.

Application of entertainment e-learning mode based on genetic algorithm and facial emotion recognition in environmental art and design courses

This article is based on facial recognition and further designs an entertainment electronic learning mode. Facial emotion recognition may be affected by light, angle, and expression. In order to improve recognition accuracy and stability, distortion adjustment techniques were used to process facial images, ensuring that the model can accurately capture and recognize the features of facial expressions. The study applies online facial emotion recognition to entertainment electronic learning modes, where learners interact with the system. The system can detect and recognize learners’ facial expressions in real-time, and provide corresponding feedback and learning resources based on different expressions. By collecting and analyzing experimental data, evaluate the practicality of the model and the level of acceptance and satisfaction of learners towards the model. The entertainment electronic learning mode based on facial recognition provides an innovative learning approach by constructing a pattern architecture, distortion adjustment, and applying online facial emotion recognition. By optimizing the positioning and integrating the entertainment electronic learning mode with environmental art and design courses, we aim to enhance students’ learning motivation and interest. Develop optimization strategies to enhance students’ comprehensive abilities in the field of environmental art and design.

Image-based facial emotion recognition using convolutional neural network on emognition dataset

Detecting emotions from facial images is difficult because facial expressions can vary significantly. Previous research on using deep learning models to classify emotions from facial images has been carried out on various datasets that contain a limited range of expressions. This study expands the use of deep learning for facial emotion recognition (FER) based on Emognition dataset that includes ten target emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, sadness, and neutral. A series of data preprocessing was carried out to convert video data into images and augment the data. This study proposes Convolutional Neural Network (CNN) models built through two approaches, which are transfer learning (fine-tuned) with pre-trained models of Inception-V3 and MobileNet-V2 and building from scratch using the Taguchi method to find robust combination of hyperparameters setting. The proposed model demonstrated favorable performance over a series of experimental processes with an accuracy and an average F1-score of 96% and 0.95, respectively, on the test data.

Masked emotions: does children's affective state influence emotion recognition?

Facial emotion recognition abilities of children have been the focus of attention across various fields, with implications for communication, social interaction, and human behavior. In response to the COVID-19 pandemic, wearing a face mask in public became mandatory in many countries, hindering social information perception and emotion recognition. Given the importance of visual communication for children's social-emotional development, concerns have been raised on whether face masks could impair their ability to recognize emotions and thereby possibly impact their social-emotional development. To this extent, a quasiexperimental study was designed with a two-fold objective: firstly, to identify children's accuracy in recognizing basic emotions (anger, happiness, fear, disgust, sadness) and emotional neutrality when presented with faces under two conditions: one with no-masks and another with faces partially covered by various types of masks (medical, nonmedical, surgical, or cloth); secondly, to explore any correlation between children's emotion recognition accuracy and their affective state. Sixty-nine (69) elementary school students aged 6-7 years old from Greece were recruited for this purpose. Following specific requirements of the second phase of the experiment students were assigned to one of three (3) distinct affective condition groups: Group A-Happiness, Group B-Sadness, and Group C-Emotional Neutrality. Image stimuli were drawn from the FACES Dataset, and students' affective state was registered using the self-reporting emotions-registration tool, AffectLecture app. The study's findings indicate that children can accurately recognize emotions even with masks, although recognizing disgust is more challenging. Additionally, following both positive and negative affective state priming promoted systematic inaccuracies in emotion recognition. Most significantly, results showed a negative bias for children in negative affective state and a positive bias for those in positive affective state. Children's affective state significantly influenced their emotion recognition abilities; sad affective states led to lower recognition overall and a bias toward recognizing sad expressions, while happy affective states resulted in a positive bias, improving recognition of happiness, and affecting how emotional neutrality and sadness were actually perceived. In conclusion, this study sheds light on the intriguing dynamics of how face masks affect children's emotion recognition, but also underlines the profound influence of their affective state.

Facial emotion recognition in children of parents with a mental illness

ObjectiveFacial emotion recognition (FER) is a fundamental social skill essential for adaptive social behaviors, emotional development, and overall well-being. FER impairments have been linked to various mental disorders, making it a critical transdiagnostic mechanism influencing the development and trajectory of mental disorders. FER has also been found to play a role in the transgenerational transmission of mental disorders, with the majority of research suggesting FER impairments in children of parents with a mental illness (COPMI). Previous research primarily concentrated on COPMI of parents with internalizing disorders, which does not cover the full spectrum of outpatient mental health service populations. Furthermore, research focuses on varying components of FER by using different assessment paradigms, making it challenging to compare study results. To address these gaps, we comprehensively investigated FER abilities in COPMI using multiple tasks varying in task characteristics.MethodsWe included 189 children, 77 COPMI and 112 children of parents without a diagnosed mental illness (COPWMI), aged 6 to 16 years. We assessed FER using three tasks with varying task demands: an emotional Go/NoGo task, a morphing task, and a task presenting short video sequences depicting different emotions. We fitted separate two-level hierarchical Bayesian models (to account for sibling pairs in our sample) for reaction times and accuracy rates for each task. Good model fit was assured by comparing models using varying priors.ResultsContrary to our expectations, our results revealed no general FER deficit in COPMI compared to COPWMI. The Bayesian models fitted for accuracy in the morphing task and Go/NoGo task yielded small yet significant effects. However, Bayes factors fitted for the models suggested that these effects could be due to random variations or noise in the data.ConclusionsOur study does not support FER impairments as a general feature of COPMI. Instead, individual factors, such as the type of parental disorder and the timing of its onset, may play a crucial role in influencing FER development. Future research should consider these factors, taking into account the diverse landscape of parental mental disorders.

The neural substrates of social cognition deficits in newly diagnosed multiple sclerosis patients.

Cognitive and affective symptoms in multiple sclerosis (MS) can be independently impaired and have different pathways of progression. Cognitive alterations have been described since the earliest MS stages; by contrast, the social cognition (SC) domain has never been investigated in the first year from MS diagnosis. We aimed to evaluate SC and unravel its neural bases in newly diagnosed MS patients. Seventy MS patients underwent at diagnosis a 3 T-MRI and a neuropsychological/SC assessment (median time between diagnosis and MRI/cognitive evaluation = 0 months). We tested two matched reference samples: 31 relapsing-remitting MS patients with longer course (mean ± SD disease duration = 7.0 ± 4.5 years) and 38 healthy controls (HCs). Cortical thicknesses (CTh) and volumes of brain regions were calculated. Newly diagnosed MS patients performed significantly lower than HCs in facial emotion recognition (global: p < 0.001; happiness: p = 0.041, anger: p = 0.007; fear: p < 0.001; disgust: p = 0.004) and theory of mind (p = 0.005), while no difference was found between newly diagnosed and longer MS patients. Compared to lower performers, higher performers in facial emotion recognition showed greater volume of amygdala (p = 0.032) and caudate (p = 0.036); higher performers in theory of mind showed greater CTh in lingual gyrus (p = 0.006), cuneus (p = 0.024), isthmus cingulate (p = 0.038), greater volumes of putamen (p = 0.016), pallidum (p = 0.029), and amygdala (p = 0.032); patients with higher empathy showed lower cuneus CTh (p = 0.042) and putamen volume (p = 0.007). SC deficits are present in MS patients since the time of diagnosis and remain persistent along the disease course. Specific basal, limbic, and occipital areas play a significant role in the pathogenesis of these alterations.

Research Review: Social cognition and everyday social skills in children and adolescents with attention-deficit/hyperactivity disorder: ameta-analysis of case-control studies.

Previous studies that have assessed social cognition in Attention-Deficit/Hyperactivity Disorder (ADHD) have produced inconsistent findings. To summarize these data and shed light upon moderators that may explain observed inconsistencies, we conducted a systematic review and meta-analysis exploring social cognition (Theory of Mind (ToM), Empathy, Facial and Non-Facial Emotion Recognition) and Everyday Social Skills in children and adolescents with ADHD. The current meta-analysis involved 142 studies including 652 effect sizes. These studies compared children and adolescents with ADHD (n = 8,300) and with typical development (n = 7,983). Participants with ADHD exhibited moderate to very large deficits in ToM (SMD = 0.84, 95% CI = 0.68-0.99), Facial Emotion Recognition (SMD = 0.63, 95% CI = 0.46-0.81), and Everyday Social Skills (SMD = 1.23, 95% CI = 1.08-1.37). The magnitude of these impairments was similar when considering effect sizes adjusted for some covariates and the methodological quality of the studies. Few studies have investigated Empathy and Non-Facial Emotion Recognition, which precludes definitive conclusions. Children and adolescents with ADHD experience robust impairments in ToM, Facial Emotion Recognition and Everyday Social Skills. Future studies should explore whether these deficits are a consequence of difficulties in other areas of cognition (e.g., executive functioning). We have made all our raw data open access to facilitate the use of the present work by the community (e.g., clinicians looking for tools, assessing social impairments, or researchers designing new studies).

Disentangling the relationship between social cognition, executive functions and behaviour changes in amyotrophic lateral sclerosis

BackgroundSocial cognition (SC) deficits are included in the amyotrophic lateral sclerosis-frontotemporal spectrum disorder (ALS-FTDS) revised diagnostic criteria. However, the impact of SC assessment on cognitive classification and the cognitive–behavioural correlates...

Simple Histogram Equalization Technique Improves Performance of VGG Models on Facial Emotion Recognition Datasets

Facial emotion recognition (FER) is crucial across psychology, neuroscience, computer vision, and machine learning due to the diversified and subjective nature of emotions, varying considerably across individuals, cultures, and contexts. This study explored FER through convolutional neural networks (CNNs) and Histogram Equalization techniques. It investigated the impact of histogram equalization, data augmentation, and various model optimization strategies on FER accuracy across different datasets like KDEF, CK+, and FER2013. Using pre-trained VGG architectures, such as VGG19 and VGG16, this study also examined the effectiveness of fine-tuning hyperparameters and implementing different learning rate schedulers. The evaluation encompassed diverse metrics including accuracy, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), Area Under the Precision–Recall Curve (AUC-PRC), and Weighted F1 score. Notably, the fine-tuned VGG architecture demonstrated a state-of-the-art performance compared to conventional transfer learning models and achieved 100%, 95.92%, and 69.65% on the CK+, KDEF, and FER2013 datasets, respectively.

Effectiveness of Group-Play Training on Emotion Recognition for Young Children Who Are Deaf or Hard of Hearing

Abstract: Deaf and hard of hearing (DHH) children are developmentally delayed in facial emotion recognition (FER). This study aimed to explore the possibility of enhancing the FER ability of DHH preschoolers through a group-play intervention. Nineteen children with a cochlear implant or hearing aid were enrolled in a 4-week intervention; six DHH children were assigned to a control group. The training program included a learning procedure for four basic emotions: happiness, sadness, anger, and fear. A pretest/posttest design was used to measure the DHH children's FER performance. The results indicated that although the two groups performed comparably on the FER task before the training (pretest), the performance of the intervention group was significantly better than that of the control group in the posttest. Moreover, the DHH children in the intervention group showed the greatest improvement in FER of happiness.

Low-resolution facial emotion recognition on low-cost devices

The low-resolution input image is a crucial challenge for applying facial emotion recognition in real-world scenarios. The critical problem is that valuable object features are relatively lost in the extraction process due to their small size. On the other hand, this vision system is required by a machine to run smoothly on low-cost devices. Facial emotion recognition using a lightweight feature extractor is proposed in this study to effectively capture crucial facial components in a low-resolution image. To compromise the running speed, this work offers an efficient feature convolution to discriminate specific facial features. In addition, the system is embedded with an attentive module to capture important features and correlate them. Our model performance is evaluated on low-resolution public datasets achieving the accuracy of 97.34\%, 81.10\%, and 80.12\% on KDEF, RFDB, and FER-plus, respectively. The practical application demands that the deep learning model can operate fast on inexpensive devices. Consequently, the model achieved a speed of 290 FPS on a CPU device.

Transfer Learning for Facial Emotion Recognition on Small Datasets

In the context of human interactions, complexity science (CS) provides a way to understand the dynamics that arise from the interplay of different individuals. Recently, the possibility of applying the theory of CS to computer science, has shifted the focus to the research of machine learning methods for studying human behaviors and relational dynamics. Among the various existing AI techniques, facial emotion recognition (FER) has proven to be the best-performing and easiest to use human- computer interaction (HCI) tool for emotion detection. Despite the numerous existing approaches, the task of FER is not trivial, mainly due to the absence of large enough datasets to train deep learning (DL) models. A widely used solution is transfer learning (TL), which allows a model, pre-trained on enough data, to be used for a specific task where there is much less data available. The aim of our work is to test the effectiveness of TL for FER on an extremely small dataset, to understand which parameters need to be optimised to obtain the best outcomes. The results showed an overall accuracy of 85.54% for our model and revealed the concrete possibility of applying computer science to complex systems typical of the human psyche.

An Approach for Evaluation and Recognition of Facial Emotions Using EMG Signal

Background: Facial electromyography (fEMG) records muscular activities from the facial muscles, which provides details regarding facial muscle stimulation patterns in experimentation. Objectives: The Principal Component Analysis (PCA) is mostly implemented, whereas the actual or unprocessed initial fEMG data are rendered into low-spatial units with minimizing the level of data repetition. Methods: Facial EMG signal was acquired by using the instrument BIOPAC MP150. Four electrodes were fixed on the face of each participant for capturing the four different emotions like happiness, anger, sad and fear. Two electrodes were placed on arm for grounding purposes. Results: The aim of this research paper is to propagate the functioning of PCA in synchrony with the subjective fEMG analysis and to give a thorough apprehension of the advanced PCA in the areas of machine learning. It describes its arithmetical characteristics, while PCA is estimated by implying the covariance matrix. Datasets which are larger in size are progressively universal and their interpretation often becomes complex or tough. So, it is necessary to minimize the number of variables and elucidate linear compositions of the data to explicate it on a huge number of variables with a relevant approach. Therefore, Principal Component Analysis (PCA) is applied because it is an unsupervised training method that utilizes advanced statistical concept to minimize the dimensionality of huge datasets. Conclusion: This work is furthermore inclined toward the analysis of fEMG signals acquired for four different facial expressions using Analysis of Variance (ANOVA) to provide clarity on the variation of features.

Facial emotion recognition and judgment of affective scenes in Parkinson's disease

Emotional dysfunctions in Parkinson's disease (PD) remain a controversial issue. While previous investigations showed compromised recognition of expressive faces in PD, no studies evaluated potential deficits in recognizing the emotional valence of affective scenes. This study aimed to investigate both facial emotion recognition performance and the ability to judge affective scenes in PD patients.Forty PD patients (mean age ± SD: 64.50 ± 8.19 years; 27 men) and forty healthy subjects (64.95 ± 8.25 years; 27 men) were included. Exclusion criteria were previous psychiatric disorders, previous Deep Brain Stimulation, and cognitive impairment. Participants were evaluated through the Ekman 60-Faces test and the International Affective Picture System. The accuracy in recognizing the emotional valence of facial expressions and affective scenes was compared between groups using linear mixed models. Pearson's correlation was performed to test the association between accuracy measures.The groups did not differ in sex, age, education, and Mini-Mental State Examination scores. Patients showed a lower recognition accuracy of facial expressions (68.54 % ± 15.83 %) than healthy participants (78.67 % ± 12.04 %; p < 0.001). Specifically, the PD group was characterized by lower recognition of faces expressing fear, sadness, and anger than the control group (all p < 0.020). No difference was detected for faces expressing disgust, surprise, and happiness (all p ≥ 0.25). Furthermore, patients showed lower accuracy in recognizing the emotional valence of affective scenes (66.75 % ± 14.59 %) than healthy subjects (74.83 % ± 12.65 %; p = 0.010). Pearson's correlations indicated that higher accuracy in recognizing the emotional facial expressions was associated with higher accuracy in classifying the valence of affective scenes in patients (r = 0.57, p < 0.001) and control participants (r = 0.57, p < 0.001).Our study suggested maladaptive affective processing in PD, leading patients to misinterpret both facial expressions and the emotional valence of complex evocative scenes.

Deciphering Crime Suspects' Emotion for Investigative Precision using Deep Learning

Using Convolutional Neural Networks (CNNs) for Facial Emotion Recognition (FER) in criminal investigations is the main goal of the proposed study. Using cutting-edge CNN architectures trained on a variety of datasets to precisely identify emotions including joy, anger, fear, surprise, disgust, and sadness is one of the main components. The research seeks to improve the effectiveness of investigations by giving law enforcement a tool to interpret the subtle emotional cues in the facial expressions of potential suspects in crimes. The technique has potential uses in human-computer interface, security, and surveillance in addition to criminal investigations. The larger significance is in helping to create more precise and effective investigative procedures, which will ultimately promote a society that is safer and more secure. The study focuses on providing a brief overview, with a suggested method utilizing CNNs to trace crime suspects.

Transfer Learning Based Face Emotion Recognition Using Meshed Faces and Oval Cropping: A Novel Approach

Transfer Learning Based Face Emotion Recognition Using Meshed Faces and Oval Cropping: A Novel Approach

Mist–fog-assisted real-time emotion recognition using deep transfer learning framework for smart city 4.0

Face emotion recognition (FER) has found several applications across numerous industries. One such application area lies in the field of human–computer interaction, where it can improve user experiences by allowing systems to adjust based on users’ emotional states by leveraging the capabilities of the Internet of Things (IoT) to facilitate more responsive interactions. The increasing demand for precise and immediate emotion analysis across various applications drives the need for FER utilizing deep transfer learning (DTL). In this study, a novel Mist-Fog framework for real-time FER is introduced, which employs a DTL algorithm called Mobile-Net for quick and easy image classification on mobile and embedded devices. It addresses the challenges of operating deep neural networks (DNN) under limits on power and processing resources. By utilizing depth-wise separable convolutions, the MobileNet algorithm reduces computational complexity while maintaining reasonable accuracy on the FER-2013 dataset. A Fog-based architecture, in which a mist layer comprising edge devices like Raspberry-Pi (RPi), an ARM-based single-board computer, was considered for a smart city scenario. Further, the M/M/1/K queuing system was considered, which follows the First-Come-First-Serve (FCFS) scheduling criteria and a Markov Modulated Poisson Process (MMPP) arrival rate to evaluate the system’s performance. The effectiveness of this novel approach towards industrial computer vision paradigms is assessed pertaining to different performance criteria.

Facial recognition, laterality judgement, alexithymia and resulting central nervous system adaptations in chronic primary headache and facial pain-A systematic review and meta-analysis.

Patients with chronic headaches and chronic oro-facial pain commonly present psychosocial issues that can affect social interactions. A possible reason could be that patients with these disorders might present impairments in facial recognition, laterality judgement and also alexithymia. However, a systematic review summarizing the effects of facial emotion recognition, laterality judgement and alexithymia in individuals with headaches and oro-facial pain is still not available. The main objective of this systematic review (SR) and meta-analysis (MA) was to compile and synthesize the evidence on the occurrence of alexithymia, deficits in laterality or left-right (LR) recognition and/or facial emotion recognition (FER) in patients with chronic headache and facial pain. Electronic searches were conducted in five databases (up to September 2023) and a manual search to identify relevant studies. The outcomes of interest were alexithymia scores, speed and accuracy in LR and/or FER, or any other quantitative data assessing body image distortions. The screening process, data extraction, risk of bias and data analysis were performed by two independent assessors following standards for systematic reviews. From 1395 manuscripts found, only 34 studies met the criteria. The overall quality/certainty of the evidence was very low. Although the results should be interpreted carefully, individuals with chronic headaches showed significantly higher levels of alexithymia when compared to healthy individuals. No conclusive results were found for the other variables of interest. Although the overall evidence from this review is very low, people with chronic primary headaches and oro-facial pain could be regularly screened for alexithymia to guarantee appropriate management.

Facial Emotion Recognition for Visually Impaired People using Transfer Learning

Individuals with visual impairment often face chal- lenges in social interactions, specifically at recognizing emotional cues. The proposed framework tackles this issue head-on by de- vising a Facial Emotion Recognition(FER) system, by employing an advanced Transfer Learning approach within Convolutional Neural Networks (CNNs). By leveraging the dataset FER-2013 [13], the proposed system aims to transcend the limitationsof traditional emotion recognition methods. Transfer learningallows the model to benefit from pre-trained knowledge on vast datasets, making it more efficient and effective in capturing complex facial features associated with different emotions. This approach is designed to offer better accuracy and generalization capabilities than other conventional methods. During training, the system will be designed to comprehensively capture the intricacies of facial expressions, enabling it to not only identify individuals but also interpret subtle changes in their emotional states throughout conversations. An innovative audio output system will be integrated into the FER system to provide a smoothand accessible experience for visually impaired users, allowing for a better understanding of social dynamics. By emphasizing transfer learning, this framework is designed to be efficient and robust, potentially revolutionizing emotional understanding for visually impaired individuals and setting a new standard in the field by showcasing the superior performance achievable throughadvanced machine learning techniques. Ultimately, this research aims to bridge the social gap for the visually impaired by fosteringinclusivity, independence, and safety in their daily life.