Facial Emotion Recognition Research Articles

Impact of psychosocial stress on facial emotion recognition in schizophrenia and controls: an experimental study in a forensic sample.

Psychotic disorders have been associated with dysregulated stress reactions and adaptation. Little is known about the neuroendocrine responses to psychosocial stress in justice-involved individuals with schizophrenia. Using an experimental research design, the present study aims to examine differences in the subjective and neuroendocrine responses to psychosocial stress and its impact on facial emotion recognition (FER) and performance on an arithmetic task in chronically ill justice-involved individuals with schizophrenia (PAT) and a healthy control group. PAT undergoing treatment in forensic psychiatric inpatient wards (n = 17) and a healthy control group (n = 17) were assessed regarding sociodemographic and clinical characteristics. Additionally, salivary cortisol levels, measured before and after performing a psychosocial stress task [Montreal Imaging Stress Task (MIST)], and performance on an arithmetic problem-solving task and two FER tasks were recorded. Two participants dropped out, one from each group. Therefore, the final sample consisted of 32 individuals. Significant group differences in FER were recorded. There was a significant rise in subjective perception of momentary strain relating to the induction of psychosocial stress in both groups. Notably, the pre-stress level of subjective strain was higher in the PAT group than controls. Acute psychosocial stress induced an increase in FER performance in a sub-task related to naming emotions in individuals with schizophrenia spectrum disorder. The results underline the importance of psychosocial and therapeutic interventions aimed at strengthening stress resilience in individuals with schizophrenia spectrum disorders.

Medical image intelligent diagnosis system based on facial emotion recognition and convolutional neural network

The facial emotional information of patients is one of the important indicators of their health status. By combining facial emotion recognition technology, medical imaging intelligent diagnostic systems can more comprehensively understand the psychological and physiological status of patients. This study first selected medical imaging datasets from the publicly available Medical Machine Learning Image Standard database and performed necessary preprocessing, including data cleaning, enhancement, and standardization. Next, a CNN model is constructed to extract key features from medical images and facial expression images through structures such as convolutional layers, activation functions, pooling layers, and fully connected layers. In addition, the system also integrates facial emotion recognition module and medical image analysis module, improving the comprehensiveness of diagnosis through feature fusion. Finally, the diagnostic results are displayed through a user interaction interface. Through comparative experiments, this study verified that the proposed CNN model outperforms traditional LSTM and SVM models in terms of accuracy, recall, and precision. The CNN model has shown significant advantages in medical imaging intelligent diagnosis systems, especially achieving a recall rate of 99%, demonstrating extremely high disease recognition ability. The results of this study not only provide new research directions for the field of intelligent diagnosis in medical imaging, but also lay a solid foundation for future clinical applications.

Medical image intelligent diagnosis system based on facial emotion recognition and convolutional neural network

The facial emotional information of patients is one of the important indicators of their health status. By combining facial emotion recognition technology, medical imaging intelligent diagnostic systems can more comprehensively understand the psychological and physiological status of patients. This study first selected medical imaging datasets from the publicly available Medical Machine Learning Image Standard database and performed necessary preprocessing, including data cleaning, enhancement, and standardization. Next, a CNN model is constructed to extract key features from medical images and facial expression images through structures such as convolutional layers, activation functions, pooling layers, and fully connected layers. In addition, the system also integrates facial emotion recognition module and medical image analysis module, improving the comprehensiveness of diagnosis through feature fusion. Finally, the diagnostic results are displayed through a user interaction interface. Through comparative experiments, this study verified that the proposed CNN model outperforms traditional LSTM and SVM models in terms of accuracy, recall, and precision. The CNN model has shown significant advantages in medical imaging intelligent diagnosis systems, especially achieving a recall rate of 99%, demonstrating extremely high disease recognition ability. The results of this study not only provide new research directions for the field of intelligent diagnosis in medical imaging, but also lay a solid foundation for future clinical applications.

Examining learners’ cognitive processes and learning emotions in digital game-based vocabulary learning: evidence from eye-tracking technology and facial emotion recognition

Purpose This research explored the intersection of cognitive processes, emotions and their impacts on digital game-based vocabulary learning (DGVL) among university students. Recognizing the scant research in this area, especially with integrating innovative technologies, this study aims to understand the influence of these elements using advanced monitoring tools. Design/methodology/approach This inquiry was carried out as an observational study involving 44 university students segmented into three English language proficiency levels: high, intermediate and low based on their English course scores. The methodological tools included a portable eye tracker to observe visual behaviors and deep learning technology to identify and analyze the participants’ emotional responses and engagement with the DGVL during the learning process. Findings The results showed that distinct fixation sequences and variations in visual attention during DGVL were correlated with different levels of competency, suggesting a direct correlation between visual engagement and language competence. In addition, emotional transitions, predominantly from engagement (“flow”) to challenge (“frustration”), were common among participants, reflecting the emotional dynamics of learning. Furthermore, all participants consistently focused on the English vocabulary definitions, indicative of their targeted approach to understanding and test preparation. These findings highlighted the intricate dynamics between emotions and cognitive processes in learning environments. Originality/value Contribution of this study shows the interplay of cognitive engagement and emotional experiences in the context of DGVL. It underscored the complex nature of these factors and their collective influence on learners’ visual and emotional engagement, offering valuable implications for educational strategies and technological applications in language learning.

Confrontation with others' emotions changes bodily resonance differently in those with low and high levels of depersonalization.

We present novel research on the cortical dynamics of atypical perceptual and emotional processing in people with symptoms of depersonalization-derealization disorder (DP-DR). We used electroencephalography (EEG)/event-related potentials (ERPs) to delineate the early perceptual mechanisms underlying emotional face recognition and mirror touch in adults with low and high levels of DP-DR symptoms (low-DP and high-DP groups). Face-sensitive visual N170 showed markedly less differentiation for emotional versus neutral face-voice stimuli in the high- than in the low-DP group. This effect was related to self-reported bodily symptoms like disembodiment. Emotional face-voice primes altered mirror touch at somatosensory cortical components P45 and P100 differently in the two groups. In the high-DP group, mirror touch occurred only when seeing touch after being confronted with angry face-voice primes. Mirror touch in the low-DP group, however, was unaffected by preceding emotions. Modulation of mirror touch following angry others was related to symptoms of self-other confusion. Results suggest that others' negative emotions affect somatosensory processes in those with an altered sense of bodily self. Our findings are in line with the idea that disconnecting from one's body and self (core symptom of DP-DR) may be a defence mechanism to protect from the threat of negative feelings, which may be exacerbated through self-other confusion. This article is part of the theme issue 'Sensing and feeling: an integrative approach to sensory processing and emotional experience'.

Psilocybin for the treatment of Alzheimer's disease.

Alzheimer's disease (AD) stands as a formidable neurodegenerative ailment and a prominent contributor to dementia. The scarcity of available therapies for AD accentuates the exigency for innovative treatment modalities. Psilocybin, a psychoactive alkaloid intrinsic to hallucinogenic mushrooms, has garnered attention within the neuropsychiatric realm due to its established safety and efficacy in treating depression. Nonetheless, its potential as a therapeutic avenue for AD remains largely uncharted. This comprehensive review endeavors to encapsulate the pharmacological effects of psilocybin while elucidating the existing evidence concerning its potential mechanisms contributing to a positive impact on AD. Specifically, the active metabolite of psilocybin, psilocin, elicits its effects through the modulation of the 5-hydroxytryptamine 2A receptor (5-HT2A receptor). This modulation causes heightened neural plasticity, diminished inflammation, and improvements in cognitive functions such as creativity, cognitive flexibility, and emotional facial recognition. Noteworthy is psilocybin's promising role in mitigating anxiety and depression symptoms in AD patients. Acknowledging the attendant adverse reactions, we proffer strategies aimed at tempering or mitigating its hallucinogenic effects. Moreover, we broach the ethical and legal dimensions inherent in psilocybin's exploration for AD treatment. By traversing these avenues, We propose therapeutic potential of psilocybin in the nuanced management of Alzheimer's disease.

An efficient facial emotion recognition using convolutional neural network with local sorting binary pattern and whale optimization algorithm

An efficient facial emotion recognition using convolutional neural network with local sorting binary pattern and whale optimization algorithm

Expression perceptive fields explain individual differences in the recognition of facial emotions

Humans can use the facial expressions of another to infer their emotional state, although it remains unknown how this process occurs. Here we suppose the presence of perceptive fields within expression space, analogous to feature-tuned receptive-fields of early visual cortex. We developed genetic algorithms to explore a multidimensional space of possible expressions and identify those that individuals associated with different emotions. We next defined perceptive fields as probabilistic maps within expression space, and found that they could predict the emotions that individuals infer from expressions presented in a separate task. We found profound individual variability in their size, location, and specificity, and that individuals with more similar perceptive fields had similar interpretations of the emotion communicated by an expression, providing possible channels for social communication. Modelling perceptive fields therefore provides a predictive framework in which to understand how individuals infer emotions from facial expressions.

Banning biometric mind reading: the case for criminalising mind probing

ABSTRACT Several emerging technologies afford a specific type of interference with privacy which has not yet been conceptualised: inferring information about mental properties of persons on the basis of physiological or biometric signals, in short: mind probing. This paper conceptualises it, analyses several forms and proposes concrete norms that ban and criminalise it. Facial emotion recognition, neuroimaging, or wearable consumer devices such as smartwatches process biometric data to draw inferences about minds; the prevalence and sophistication of such mind probes will soon increase through AI data analysis. This paper explores how human rights law, the EU General Data Protection Regulation and the EU AI Act address mind probes and shows why the current level of protection appears insufficient. This motivates novel norms banning and criminalising non-consensual biometric-based forms of mind reading, ensuring that at least some parts of the mind remain in principle free from non-consensual mind probing.

Functional connectivity during facial and vocal emotion recognition: Preliminary evidence for dissociations in developmental change by nonverbal modality

The developmental trajectory of emotion recognition (ER) skills is thought to vary by nonverbal modality, with vocal ER becoming mature later than facial ER. To investigate potential neural mechanisms contributing to this dissociation at a behavioural level, the current study examined whether youth's neural functional connectivity during vocal and facial ER tasks showed differential developmental change across time. Youth ages 8–19 (n = 41) completed facial and vocal ER tasks while undergoing functional magnetic resonance imaging, at two timepoints (1 year apart; n = 36 for behavioural data, n = 28 for neural data). Partial least squares analyses revealed that functional connectivity during ER is both distinguishable by modality (with different patterns of connectivity for facial vs. vocal ER) and across time—with changes in connectivity being particularly pronounced for vocal ER. ER accuracy was greater for faces than voices, and positively associated with age; although task performance did not change appreciably across a 1-year period, changes in latent functional connectivity patterns across time predicted participants' ER accuracy at Time 2. Taken together, these results suggest that vocal and facial ER are supported by distinguishable neural correlates that may undergo different developmental trajectories. Our findings are also preliminary evidence that changes in network integration may support the development of ER skills in childhood and adolescence.

No Pain, No Gain—Giving Real-Time Emotional Feedback in a Virtual Mirror Improves Collaboration in Virtual Teamwork

This study investigates the impact of real-time emotional feedback on the quality of teamwork conducted over videoconferencing. We developed a framework that provides real-time feedback through a virtual mirror based on facial and voice emotion recognition. In an experiment with 28 teams (84 participants), teams collaborated over Zoom (version 5.16.6) to set up a virtual Mars station using custom simulation software (Mars Star City, version 4.0). Participants were divided into 14 experimental teams, which were shown the virtual mirror, and 14 control teams without it. Team performance was measured by the improvement in the Mars simulation output quality. Our analysis using correlation, multi-level regression, and machine learning revealed that fewer interruptions but an increasing number over time correlated with higher performance. Higher vocal arousal and happiness also enhanced performance. We confirmed that female presence in teams boosts performance. SHAP values indicated that high variability in happiness, head movement, and positive facial valence—an “emotional rollercoaster”—positively predicted team performance. The experimental group outperformed the control group, suggesting that virtual mirroring improves virtual teamwork and that interrupting each other more while speaking less, leads to better results.

A Randomized Trial Utilizing EEG Brain Computer Interface to Improve Facial Emotion Recognition in Autistic Adults

A Randomized Trial Utilizing EEG Brain Computer Interface to Improve Facial Emotion Recognition in Autistic Adults

Facial emotion recognition: A comprehensive review

AbstractFacial emotion recognition (FER) represents a significant outcome of the rapid advancements in artificial intelligence (AI) technology. In today's digital era, the ability to decipher emotions from facial expressions has evolved into a fundamental mode of human interaction and communication. As a result, FER has penetrated diverse domains, including but not limited to medical diagnosis, customer feedback analysis, the automation of automobile driver systems, and the evaluation of student comprehension. Furthermore, it has matured into a captivating and dynamic research field, capturing the attention and curiosity of contemporary scholars and scientists. The primary objective of this paper is to provide an exhaustive review of FER systems. Its significance goes beyond offering a comprehensive resource; it also serves as a valuable guide for emerging researchers in the FER domain. Through a meticulous examination of existing FER systems and methodologies, this review equips them with essential insights and guidance for their future research pursuits. Moreover, this comprehensive review contributes to the expansion of their knowledge base, facilitating a profound understanding of this rapidly evolving field. In a world increasingly dependent on technology for communication and interaction, the study of FER holds a pivotal role in human‐computer interaction (HCI). It not only provides valuable insights but also unlocks a multitude of possibilities for future innovations and applications. As we continue to integrate AI and facial emotion recognition into our daily lives, the importance of comprehending and enhancing FER systems becomes increasingly evident. This paper serves as a stepping stone for researchers, nurturing their involvement in this exciting and ever‐evolving field.

Facial Emotion Recognition System Using CNN for Song Mapping

This project introduces a novel system that integrates Facial Emotion Recognition (FER) with music mapping to enhance human-computer interaction. Leveraging a custom Convolutional Neural Network (CNN) algorithm, developed using the FER2013 dataset, we classify emotions into four categories- happy, sad, neutral, and angry- employing Haar Cascade for precise face detection and grayscale conversion for optimal CNN input. Our custom CNN demonstrates superior performance, achieving a testing accuracy of 77.23%, notably surpassing established models like VGG16 and ResNet50, which achieved 55.87% and 62.76% respectively. This system swiftly identifies emotions and recommends songs from the "278k Emotion Labeled Spotify Songs" playlist, aiming to boost user satisfaction. Through nuanced comparisons with VGG16 and ResNet50, our approach underscores its inherent strengths, suggesting promising advancements in Facial Emotion Recognition (FER) and music recommendation systems. With a focus on precision in emotion detection, our system subtly elevates user experience, contributing meaningfully to ongoing research in the field.

Application of entertainment e-learning mode based on genetic algorithm and facial emotion recognition in environmental art and design courses

This article is based on facial recognition and further designs an entertainment electronic learning mode. Facial emotion recognition may be affected by light, angle, and expression. In order to improve recognition accuracy and stability, distortion adjustment techniques were used to process facial images, ensuring that the model can accurately capture and recognize the features of facial expressions. The study applies online facial emotion recognition to entertainment electronic learning modes, where learners interact with the system. The system can detect and recognize learners’ facial expressions in real-time, and provide corresponding feedback and learning resources based on different expressions. By collecting and analyzing experimental data, evaluate the practicality of the model and the level of acceptance and satisfaction of learners towards the model. The entertainment electronic learning mode based on facial recognition provides an innovative learning approach by constructing a pattern architecture, distortion adjustment, and applying online facial emotion recognition. By optimizing the positioning and integrating the entertainment electronic learning mode with environmental art and design courses, we aim to enhance students’ learning motivation and interest. Develop optimization strategies to enhance students’ comprehensive abilities in the field of environmental art and design.

Image-based facial emotion recognition using convolutional neural network on emognition dataset

Detecting emotions from facial images is difficult because facial expressions can vary significantly. Previous research on using deep learning models to classify emotions from facial images has been carried out on various datasets that contain a limited range of expressions. This study expands the use of deep learning for facial emotion recognition (FER) based on Emognition dataset that includes ten target emotions: amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, sadness, and neutral. A series of data preprocessing was carried out to convert video data into images and augment the data. This study proposes Convolutional Neural Network (CNN) models built through two approaches, which are transfer learning (fine-tuned) with pre-trained models of Inception-V3 and MobileNet-V2 and building from scratch using the Taguchi method to find robust combination of hyperparameters setting. The proposed model demonstrated favorable performance over a series of experimental processes with an accuracy and an average F1-score of 96% and 0.95, respectively, on the test data.

Masked emotions: does children's affective state influence emotion recognition?

Facial emotion recognition abilities of children have been the focus of attention across various fields, with implications for communication, social interaction, and human behavior. In response to the COVID-19 pandemic, wearing a face mask in public became mandatory in many countries, hindering social information perception and emotion recognition. Given the importance of visual communication for children's social-emotional development, concerns have been raised on whether face masks could impair their ability to recognize emotions and thereby possibly impact their social-emotional development. To this extent, a quasiexperimental study was designed with a two-fold objective: firstly, to identify children's accuracy in recognizing basic emotions (anger, happiness, fear, disgust, sadness) and emotional neutrality when presented with faces under two conditions: one with no-masks and another with faces partially covered by various types of masks (medical, nonmedical, surgical, or cloth); secondly, to explore any correlation between children's emotion recognition accuracy and their affective state. Sixty-nine (69) elementary school students aged 6-7 years old from Greece were recruited for this purpose. Following specific requirements of the second phase of the experiment students were assigned to one of three (3) distinct affective condition groups: Group A-Happiness, Group B-Sadness, and Group C-Emotional Neutrality. Image stimuli were drawn from the FACES Dataset, and students' affective state was registered using the self-reporting emotions-registration tool, AffectLecture app. The study's findings indicate that children can accurately recognize emotions even with masks, although recognizing disgust is more challenging. Additionally, following both positive and negative affective state priming promoted systematic inaccuracies in emotion recognition. Most significantly, results showed a negative bias for children in negative affective state and a positive bias for those in positive affective state. Children's affective state significantly influenced their emotion recognition abilities; sad affective states led to lower recognition overall and a bias toward recognizing sad expressions, while happy affective states resulted in a positive bias, improving recognition of happiness, and affecting how emotional neutrality and sadness were actually perceived. In conclusion, this study sheds light on the intriguing dynamics of how face masks affect children's emotion recognition, but also underlines the profound influence of their affective state.

Facial emotion recognition in children of parents with a mental illness

ObjectiveFacial emotion recognition (FER) is a fundamental social skill essential for adaptive social behaviors, emotional development, and overall well-being. FER impairments have been linked to various mental disorders, making it a critical transdiagnostic mechanism influencing the development and trajectory of mental disorders. FER has also been found to play a role in the transgenerational transmission of mental disorders, with the majority of research suggesting FER impairments in children of parents with a mental illness (COPMI). Previous research primarily concentrated on COPMI of parents with internalizing disorders, which does not cover the full spectrum of outpatient mental health service populations. Furthermore, research focuses on varying components of FER by using different assessment paradigms, making it challenging to compare study results. To address these gaps, we comprehensively investigated FER abilities in COPMI using multiple tasks varying in task characteristics.MethodsWe included 189 children, 77 COPMI and 112 children of parents without a diagnosed mental illness (COPWMI), aged 6 to 16 years. We assessed FER using three tasks with varying task demands: an emotional Go/NoGo task, a morphing task, and a task presenting short video sequences depicting different emotions. We fitted separate two-level hierarchical Bayesian models (to account for sibling pairs in our sample) for reaction times and accuracy rates for each task. Good model fit was assured by comparing models using varying priors.ResultsContrary to our expectations, our results revealed no general FER deficit in COPMI compared to COPWMI. The Bayesian models fitted for accuracy in the morphing task and Go/NoGo task yielded small yet significant effects. However, Bayes factors fitted for the models suggested that these effects could be due to random variations or noise in the data.ConclusionsOur study does not support FER impairments as a general feature of COPMI. Instead, individual factors, such as the type of parental disorder and the timing of its onset, may play a crucial role in influencing FER development. Future research should consider these factors, taking into account the diverse landscape of parental mental disorders.

The neural substrates of social cognition deficits in newly diagnosed multiple sclerosis patients.

Cognitive and affective symptoms in multiple sclerosis (MS) can be independently impaired and have different pathways of progression. Cognitive alterations have been described since the earliest MS stages; by contrast, the social cognition (SC) domain has never been investigated in the first year from MS diagnosis. We aimed to evaluate SC and unravel its neural bases in newly diagnosed MS patients. Seventy MS patients underwent at diagnosis a 3 T-MRI and a neuropsychological/SC assessment (median time between diagnosis and MRI/cognitive evaluation = 0 months). We tested two matched reference samples: 31 relapsing-remitting MS patients with longer course (mean ± SD disease duration = 7.0 ± 4.5 years) and 38 healthy controls (HCs). Cortical thicknesses (CTh) and volumes of brain regions were calculated. Newly diagnosed MS patients performed significantly lower than HCs in facial emotion recognition (global: p < 0.001; happiness: p = 0.041, anger: p = 0.007; fear: p < 0.001; disgust: p = 0.004) and theory of mind (p = 0.005), while no difference was found between newly diagnosed and longer MS patients. Compared to lower performers, higher performers in facial emotion recognition showed greater volume of amygdala (p = 0.032) and caudate (p = 0.036); higher performers in theory of mind showed greater CTh in lingual gyrus (p = 0.006), cuneus (p = 0.024), isthmus cingulate (p = 0.038), greater volumes of putamen (p = 0.016), pallidum (p = 0.029), and amygdala (p = 0.032); patients with higher empathy showed lower cuneus CTh (p = 0.042) and putamen volume (p = 0.007). SC deficits are present in MS patients since the time of diagnosis and remain persistent along the disease course. Specific basal, limbic, and occipital areas play a significant role in the pathogenesis of these alterations.

Research Review: Social cognition and everyday social skills in children and adolescents with attention-deficit/hyperactivity disorder: ameta-analysis of case-control studies.

Previous studies that have assessed social cognition in Attention-Deficit/Hyperactivity Disorder (ADHD) have produced inconsistent findings. To summarize these data and shed light upon moderators that may explain observed inconsistencies, we conducted a systematic review and meta-analysis exploring social cognition (Theory of Mind (ToM), Empathy, Facial and Non-Facial Emotion Recognition) and Everyday Social Skills in children and adolescents with ADHD. The current meta-analysis involved 142 studies including 652 effect sizes. These studies compared children and adolescents with ADHD (n = 8,300) and with typical development (n = 7,983). Participants with ADHD exhibited moderate to very large deficits in ToM (SMD = 0.84, 95% CI = 0.68-0.99), Facial Emotion Recognition (SMD = 0.63, 95% CI = 0.46-0.81), and Everyday Social Skills (SMD = 1.23, 95% CI = 1.08-1.37). The magnitude of these impairments was similar when considering effect sizes adjusted for some covariates and the methodological quality of the studies. Few studies have investigated Empathy and Non-Facial Emotion Recognition, which precludes definitive conclusions. Children and adolescents with ADHD experience robust impairments in ToM, Facial Emotion Recognition and Everyday Social Skills. Future studies should explore whether these deficits are a consequence of difficulties in other areas of cognition (e.g., executive functioning). We have made all our raw data open access to facilitate the use of the present work by the community (e.g., clinicians looking for tools, assessing social impairments, or researchers designing new studies).