Facial Emotion Recognition Research Articles

Neural correlates ofsuccessful emotion recognition inhealthy elderly: a multimodal imaging study.

The ageing process is associated with reduced emotional recognition (ER) performance. The ER ability is an essential part of non-verbal communication, and its role is crucial for proper social functioning. Here, using the 'Cambridge Centre for Ageing and Neuroscience cohort sample', we investigated when ER, measured using a facial emotion recognition test, begins to consistently decrease along the lifespan. Moreover, using structural and functional MRI data, we identified the neural correlates associated with ER maintenance in the age groups showing early signs of ER decline (N = 283; age range: 58-89 years). The ER performance was positively correlated with greater volume in the superior parietal lobule, higher white matter integrity in the corpus callosum and greater functional connectivity in the mid-cingulate area. Our results suggest that higher ER accuracy in older people is associated with preserved gray and white matter volumes in cognitive or interconnecting areas, subserving brain regions directly involved in emotional processing.

Facing Change: Using Automated Facial Expression Analysis to Examine Emotional Flexibility in the Treatment of Depression.

Depression involves deficits in emotional flexibility. To date, the varied and dynamic nature of emotional processes during therapy has mostly been measured at discrete time intervals using clients' subjective reports. Because emotions tend to fluctuate and change from moment to moment, the understanding of emotional processes in the treatment of depression depends to a great extent on the existence of sensitive, continuous, and objectively codified measures of emotional expression. In this observational study, we used computerized measures to analyze high-resolution time-series facial expression data as well as self-reports to examine the association between emotional flexibility and depressive symptoms at the client as well as at the session levels. Video recordings from 283 therapy sessions of 58 clients who underwent 16 sessions of manualized psychodynamic psychotherapy for depression were analyzed. Data was collected as part of routine practice in a university clinic that provides treatments to the community. Emotional flexibility was measured in each session using an automated facial expression emotion recognition system. The clients' depression level was assessed at the beginning of each session using the Beck Depression Inventory-II (Beck et al., 1996). Higher emotional flexibility was associated with lower depressive symptoms at the treatment as well as at the session levels. These findings highlight the centrality of emotional flexibility both as a trait-like as well as a state-like characteristic of depression. The results also demonstrate the usefulness of computerized measures to capture key emotional processes in the treatment of depression at a high scale and specificity.

Algorithms used for facial emotion recognition: a systematic review of the literature

INTRODUCTION: We currently live in a society that is constantly changing and technology has developed algorithms that allow facial emotion recognition, because facial expression transmits people's mood, feelings and state of soul. However, it is required that future research can improve the quality of emotion detection by improving the quality of the data set and the model used, for this reason, it is necessary to investigate other machine learning algorithms in the recognition of facial emotions, as they exist. identification deficiencies that limit the discrimination of extracted structural features.OBJECTIVE: The purpose of the article was to analyze the most used algorithms for facial emotion recognition, through a systematic literature review, according to the PRISMA method.METHOD: A search for information was carried out in articles published in open access such as: Scopus, Web of Science (WOS) and Association for Computing Machiner (ACM) in the period 2022 and 2023, totaling 38 selected articles.RESULTS: The results obtained indicate that the algorithms most used by the authors are SVM and SoftMax with a total of 17.65% each.CONCLUSION: It is concluded that the SVM and SoftMax algorithms are the most predominant, playing a crucial role in achieving optimal levels of precision in the training of the models. These algorithms, with their robustness and ability to deal with complex data, have proven to be fundamental pillars in the field of facial emotion recognition.

Facial emotion recognition based on improved ResNet

A common and difficult task in computer vision is the identification of facial expressions of emotion. Facial emotion recognition has shown great promise in many areas including healthcare, robotic communication and customer service. However, the variability of human appearance and muscle movement leads to difficulties in facial emotion recognition. Therefore, deeper convolutional neural networks are introduced to recognize facial emotions. The residual network (ResNet) can be built in a deep architecture that can solve the degradation problem when the depth of the network increases. In this paper, an improved ResNet50 is proposed to implement facial emotion recognition. Specifically, the proposed model appends two blocks consisting of fully connected layers to the ResnNet50. The layers are stacked with shortcut connections to solve the degradation problem and to make the training process smoother. The accuracy achieved by the improved model on the Facial Emotion Recognition 2013 dataset (FER-2013) is 13.31% higher than that of the ResNet50. Experimental data indicate that the improved model performs efficiently in facial emotion recognition due to shortcut connectivity and the addition of fully connected layers. Meanwhile, the degradation and gradient disappearance problems are improved.

Facial emotion detection based on improved VGG-16

In real life, facial emotion recognition is very important because it can convey information, build relationships, and facilitate communication. Therefore, emotion recognition technology is used in medicine, education, entertainment, security, and other fields. In the emotion detection field, the Facial Emotion Recognition 2013 Dataset (FER-2013) is a dataset that has been used in many places, that contains images of seven emotional expressions. In the area of detecting emotions from facial expressions, the deep learning structures, especially the convolutional neural networks (CNNs), have demonstrated significant potential since they have the ability to extract features and their computational efficiency. In this paper, the author constructs a model named Improved VGG-16 based on Visual Geometry Group Network of 16 weight layers (VGG-16). To be specific, first, the author adds two dense layers to improve the complexity and expressiveness; second, two dropout layers are used in order to reduce overfitting. An accuracy of 68.0% is achieved by this model on the test dataset of FER-2013. The result is better than some previous methods and shows that the improved VGG-16 model can recognize facial expressions effectively. In conclusion, this work aims to increase the accuracy and reliability of facial emotion recognition, providing support for research and application in related fields.

Convolutional neural network combined with the attention mechanism for facial emotion recognition

Facial Emotion Recognition (FER) holds great importance in the fields of computer vision and machine learning. In this study, the aim is to improve the accuracy of facial expression recognition by incorporating attention mechanisms into Convolutional Neural Networks (CNN) with FER2013 dataset, which consists of grayscale images categorized into seven expressions. The combination of proposed CNN architecture and attention mechanisms is thoroughly elucidated, emphasizing the operations and interactions of their components. Additionally, the effectiveness of the new model is evaluated through experiments, comparing its performance with existing approaches in terms of accuracy. Besides, the results demonstrate that the CNN architecture with attention mechanisms outperforms the original CNN by achieving an improved accuracy rate of 69.07%, which is higher than 68.04% accuracy rate of original CNN. Moreover, the study further discusses the confusion matrix analysis, revealing the challenges faced in recognizing specific emotions due to limited training data and vague facial features. In the future, this study suggests addressing these limitations through data augmentation and to reduce the gap between training and testing accuracy. Overall, this research highlights the potential of attention mechanisms in enhancing facial expression recognition systems, paving the way for advanced applications in various domains.

A critical examination of deficiencies in the convolutional neural network model for facial emotion recognition

The challenge of addressing the issue of low accuracy in specific scenarios encountered during the implementation of facial emotion recognition systems arises due to the wide array of environments and varying conditions. In this study, the Facial Expression Recognition-2013 (FER-2013) dataset sourced from the Kaggle serves as the basis for training the models, with subsequent analysis conducted on the experimental outcomes. The dataset comprises a training set and a testing set, each annotated with labels representing seven distinct emotions, ranging from "angry" to "surprise". The models developed for facial emotion classification, tasked with automatically recognizing emotions based on provided images, consist of a MobileNet-based model and a self-built model employing convolutional neural networks. Both models exhibit an accuracy of approximately 60%, yet demonstrate deficiencies in predicting the "neutral" label. Additionally, the utilization of techniques such as confusion matrix and saliency map enable the comparative evaluation of model performance across different emotion labels and facilitates an analysis of their corresponding dominant facial regions. Based on a comparison of results obtained from representative cases, two potential factors contributing to these limitations are identified: a paucity of training data and the presence of ambiguous features. The findings of this study are expected to inform future directions for improvement and modification of facial emotion recognition models in order to enhance their applicability in diverse scenarios.

Face-emotion classification guided by deep convolutional neural network

With the rapid development of computer vision and convolutional neural networks, the task of automatic face emotion classification has become a reality. The aim of this study is to improve the underlying neural network model to achieve effective face emotion classification. By presenting a simplified network to generate the recognition model, the author enhances the underlying neural network architecture. The model, in particular, augments the underlying neural network with a convolutional layer, a maximum pooling layer, and a discard layer, and increases the number of neurons in the dense layer from 25 to 128. The convolutional layer allows for the automatic extraction of sentiment features. To decrease the parameters in the feature maps, the maximum pooling layer is applied. The experiments are constructed on the Facial Emotion Recognition 2013 dataset (FER-2013). The streamlined network model improves performance by 6% to 56.32% as compared with the basic neural network model. Numerous experiments show that the proposed streamlined network model can effectively recognize facial emotions. In addition, the author analysis the confusion matrix and finds that the model has weak feedback for aversive emotions. Future research will focus on improving the representation of unclear features such as aversive emotions to enhance model generalization.

Recognition and Perception of Emotions in Juvenile Myoclonic Epilepsy.

Perception and recognition of emotions are fundamental prerequisites of human life. Patients with juvenile myoclonic epilepsy (JME) may have emotional and behavioral impairments which might influence socially desirable interactions. We aimed to investigate perception and recognition of emotions in patients with JME by means of neuropsychological tests and functional MRI (fMRI). Sixty-five patients with JME (median age 27 years, IQR 23-34) were prospectively recruited at the Department of Neurology, Christian Doppler University Hospital, Paracelsus Medical University, Salzburg, Austria. Patients were compared to 68 healthy controls (median age 24 years, IQR 21-31), matched for sex, age and education. All study participants underwent the Networks of Emotion Processing test battery (NEmo), an fMRI paradigm of "dynamic fearful faces", structured interview for psychiatric and personality disorders and comprehensive neuropsychological testing. JME patients versus healthy controls, demonstrated significant deficits in emotion recognition in facial and verbal tasks of all emotions, especially fear. fMRI revealed decreased amygdala activation in JME patients as compared to healthy controls. Patients were at a higher risk of experiencing psychiatric disorders as compared to healthy controls. Cognitive evaluation revealed impaired attentional and executive functioning, namely psychomotor speed, tonic alertness, divided attention, mental flexibility, and inhibition of automated reactions. Duration of epilepsy correlated negatively with parallel prosodic and facial emotion recognition in NEmo. Deficits in emotion recognition were not associated with psychiatric comorbidities, impaired attention and executive functions, types of seizures and treatment. This prospective study demonstrated that as compared to healthy subjects, patients with JME had significant deficits in recognition and perception of emotions as shown by neuropsychological tests and fMRI. The results of this study may have importance for psychological/psychotherapeutical interventions in the management of patients with JME.

Hearts and Minds: Emotion Recognition and Mentalizing in Parkinson's Disease and Progressive Supranuclear Palsy.

There are scarce data comparing Parkinson's disease (PD) and Progressive Supranuclear Palsy (PSP) in social cognition (SC). We aimed to compare patients with PSP and PD in SC. We included three groups: PD (n = 18), PSP (n = 20) and controls (n = 23). Participants underwent neuropsychological exams, including the mini-version of the Social and Emotional Assessment, which is composed of the facial emotion recognition test (FERT) and the modified faux-pas (mFP) test, which assesses Theory of Mind (ToM). Patients with PD scored lower than controls in the FERT, but not in the mFP test. Patients with PSP performed worse than controls in both the mFP and FERT. PD and PSP groups did not differ in the FERT, but PSP performed worse than PD in the mFP test. The mFP test distinguished PSP from PD with 89% accuracy. The assessment of ToM may contribute to the differentiation between PD and PSP.

Facial Emotion Recognition and Classification Using the Convolutional Neural Network-10 (CNN-10)

The importance of facial expressions in nonverbal communication is significant because they help better represent the inner emotions of individuals. Emotions can depict the state of health and internal wellbeing of individuals. Facial expression detection has been a hot research topic in the last couple of years. The motivation for applying the convolutional neural network-10 (CNN-10) model for facial expression recognition stems from its ability to detect spatial features, manage translation invariance, understand expressive feature representations, gather global context, and achieve scalability, adaptability, and interoperability with transfer learning methods. This model offers a powerful instrument for reliably detecting and comprehending facial expressions, supporting usage in recognition of emotions, interaction between humans and computers, cognitive computing, and other areas. Earlier studies have developed different deep learning architectures to offer solutions to the challenge of facial expression recognition. Many of these studies have good performance on datasets of images taken under controlled conditions, but they fall short on more difficult datasets with more image diversity and incomplete faces. This paper applied CNN-10 and ViT models for facial emotion classification. The performance of the proposed models was compared with that of VGG19 and INCEPTIONV3. The CNN-10 outperformed the other models on the CK + dataset with a 99.9% accuracy score, FER-2013 with an accuracy of 84.3%, and JAFFE with an accuracy of 95.4%.

Pseudo-labeling of transfer learning convolutional neural network data for human facial emotion recognition

The relevance of solving the problem of facial emotion recognition on human images in the creation of modern intelligent systems of computer vision and human-machine interaction, online learning and emotional marketing, health care and forensics, machine graphics and game intelligence is shown. Successful examples of technological solutions to the problem of facial emotion recognition using transfer learning of deep convolutional neural networks are shown. But the use of such popular datasets as DISFA, CelebA, AffectNet, for deep learning of convolutional neuralnetworks does not give good results in terms of the accuracy of emotion recognition, because almost all training sets have fundamental flaws related to errors in their creation, such as the lack of data of a certain class, imbalance of classes, subjectivity and ambiguity of labeling, insufficient amount of data for deep learning, etc. It is proposed to overcome the noted shortcomings of popular datasets for emotion recognition by adding to the training sample additional pseudo-labeled images with human emotions, on which recognition occurs with high accuracy. The aim of the research is to increase the accuracy of facial emotion recognitionon the image of a human by developing a pseudo-labeling method for transfer learning of a deep neural network. To achieve the aim, the following tasks were solved: a convolutional neural network model, previously trained on the ImageNet set using the transfer learning method, was adjusted on the RAF-DB data set to solve emotion recognition tasks; a pseudo-labeling method of the RAF−DB set data was developed for semi-supervised learning of a convolutional neural network model for the task of facial emotion recognition; the accuracy of facial emotion recognition was analyzed based on the developed convolutional neural network model and the method of pseudo-labeling of RAF-DB set data for its correction. It is shown that the use of the developed method of pseudo-labeling data and transfer learning of the MobileNet V1 convolutional neural network model allowed to increase the accuracy of facial emotion recognitionon the images of the RAF-DB dataset by 2 percent (from 76 to 78%) according to the F1 estimate. Atthe same time, taking into account the significant imbalance of the classes, for the 7 main emotions in the trainingset, we have a significant increase in the accuracy of recognizing a few representatives of such emotions as surprise (from 71 to 77%), fearful(from 64 to 69%), sad (from 72 to 76%), angrywith (from 64 to 74%), neutral(from 66 to 71%). The accuracy of recognizing the emotion of happy, which is the most common, decreased (from 91 to 86 %) Thus, it can be concluded that the use of the developed pseudo-labeling method gives good results in overcoming such shortcomings of datasets for deep learning of convolutional neural networks such as lack of data of a certain type, imbalance of classes, insufficient amount of data for deep learning, etc.

Facial expression recognition in virtual reality environments: challenges and opportunities.

This study delved into the realm of facial emotion recognition within virtual reality (VR) environments. Using a novel system with MobileNet V2, a lightweight convolutional neural network, we tested emotion detection on 15 university students. High recognition rates were observed for emotions like "Neutral", "Happiness", "Sadness", and "Surprise". However, the model struggled with 'Anger' and 'Fear', often confusing them with "neutral". These discrepancies might be attributed to overlapping facial indicators, limited training samples, and the precision of the devices used. Nonetheless, our research underscores the viability of using facial emotion recognition technology in VR and recommends model improvements, the adoption of advanced devices, and a more holistic approach to foster the future development of VR emotion recognition.

Research on face emotion recognition algorithm based on deep learning neural network

Abstract This paper explores face emotion recognition based on deep learning neural networks in order to make the computer recognize facial emotion more accurately. Firstly, we introduce the convolutional neural network, which consists of a convolutional layer, pooling layer and fully connected layer in the deep learning neural network, and propose the variational self-encoder face emotion recognition algorithm based on the convolutional neural network in combination with the facial emotion recognition process. The correct recognition rates achieved by EVA using 32×32 pixels, LBP features and 50-dimensional Gabor wavelet features are 95.13%, 96.74% and 98.84%, respectively. In terms of emotion recognition ability, the recognition accuracy percentages were around 70% for large emotion fluctuations and around 30% for small emotion fluctuations. Since the neutral facial emotion is not well distinguished, the recognition degree of neutral facial emotion is only 80%, and the recognition degree using the self-encoder-based face emotion recognition algorithm in several facial features extraction is above 80%, and the recognition degree with neutral emotion removed is up to about 90%. Therefore, the algorithm in this paper has good classification performance and can recognize and analyze facial emotion features accurately.

A Computational Tool for Recoloring Based on User Emotions

This work describes a system to recolor a user’s painting based on the perceived emotional state of the viewer. An automatic palette selection algorithm is used to generate color palettes for a set of emotions. A user can create a painting using one of the generated palettes. To notify the end of the painting, the user clicks on the DONE button. Once the button is pressed, the colors of the user's painting change as the facial expression of the user changes. Facial emotion recognition is used in this process to classify the emotional status of the user’s face.

Learning emotional dialects: A British population study of cross-cultural communication.

The aim of the current research was to explore whether we can improve the recognition of cross-cultural freely-expressed emotional faces in British participants. We tested several methods for improving the recognition of freely-expressed emotional faces, such as different methods for presenting other-culture expressions of emotion from individuals from Chile, New Zealand and Singapore in two experimental stages. In the first experimental stage, in phase one, participants were asked to identify the emotion of cross-cultural freely-expressed faces. In the second phase, different cohorts were presented with interactive side-by-side, back-to-back and dynamic morphing of cross-cultural freely-expressed emotional faces, and control conditions. In the final phase, we repeated phase one using novel stimuli. We found that all non-control conditions led to recognition improvements. Morphing was the most effective condition for improving the recognition of cross-cultural emotional faces. In the second experimental stage, we presented morphing to different cohorts including own-to-other and other-to-own freely-expressed cross-cultural emotional faces and neutral-to-emotional and emotional-to-neutral other-culture freely-expressed emotional faces. All conditions led to recognition improvements and the presentation of freely-expressed own-to-other cultural-emotional faces provided the most effective learning. These findings suggest that training can improve the recognition of cross-cultural freely-expressed emotional expressions.

Ameliorate grasshopper optimization algorithm based long short term memory classification for face emotion recognition system

Ameliorate grasshopper optimization algorithm based long short term memory classification for face emotion recognition system

Alibaba and forty thieves algorithm and novel Prioritized Prewitt Pattern(PPP)-based convolutional neural network (CNN) using hyperspherically compressed weights for facial emotion recognition

The visual representations created using the self-distillation paradigm of Bootstrap Your Emotion Latent (BYEL) are empirically found to be less evenly distributed than those created using proposed technique. This proposed work promotes the compression of weights on a hypersphere by minimizing the hyperspherical energy of network weights using a novel method of optimizing manifolds through Riemannian metrics and the Conjugate gradient technique. The proposed work demonstrates how regularising the networks of the BYEL architecture reduces the hyperspherical energy of neurons by directly optimising a measure of uniformity alongside the standard loss. This leads to more uniformly distributed representation and better performance for downstream tasks. The Alibaba and Forty Thieves Algorithm-based Optimization (AFTAO) methodology is used to select the most precise collection of hyperparameters for a novel Prioritized Prewitt Pattern (PPP)-based Convolutional Neural Network (CNN) that results in a higher accuracy for all the six datasets used for facial emotion recognition.

Sex Differences in Recognition of Face Expressions

Abstract: Emotional facial expressions are a crucial non-verbal communication skill for humans’ interactions. The current study assessed the impact of sex on emotional face recognition. A total of 125 individuals per-formed an online emotional face recognition task. The stimuli were created recutting male and female Saudi volunteers. Results showed that, except of the male participants having faster responses compared to females in general, no main significant differences between the sex group in accuracy nor any significant interaction between participant sex and the sex of the faces. Our findings suggests that the effect of sex on emotional face recognition needs further investigation with well calibrated stimuli. Limitations and the direction of future research in this area were discussed.

Internet-based identification of anxiety in university students using text and facial emotion analysis

BackgroundAnxiety in university students can lead to poor academic performance and even dropout. The Adult Manifest Anxiety Scale (AMAS-C) is a validated measure designed to assess the level and nature of anxiety in college students. ObjectiveThe aim of this study is to provide internet-based alternatives to the AMAS-C in the automated identification and prediction of anxiety in young university students. Two anxiety prediction methods, one based on facial emotion recognition and the other on text emotion recognition, are described and validated using the AMAS-C Test Anxiety, Lie and Total Anxiety scales as ground truth data. MethodsThe first method analyses facial expressions, identifying the six basic emotions (anger, disgust, fear, happiness, sadness, surprise) and the neutral expression, while the students complete a technical skills test. The second method examines emotions in posts classified as positive, negative and neutral in the students' profile on the social network Facebook. Both approaches aim to predict the presence of anxiety. ResultsBoth methods achieved a high level of precision in predicting anxiety and proved to be effective in identifying anxiety disorders in relation to the AMAS-C validation tool. Text analysis-based prediction showed a slight advantage in terms of precision (86.84 %) in predicting anxiety compared to face analysis-based prediction (84.21 %). ConclusionsThe applications developed can help educators, psychologists or relevant institutions to identify at an early stage those students who are likely to fail academically at university due to an anxiety disorder.