Emotion Recognition Research Articles

MHealth Service Effects for Negative Emotions Among Parents of Preterm Infants: A Systematic Review and Meta-Analysis.

This review aims to ascertain the effects of mHealth on negative emotions among parents of preterm infants and analyze the current state of mHealth services in this context. Caring for preterm infants often engenders negative emotions in parents, such as anxiety, depression, and stress. Mobile Health (mHealth) services offer potential benefits for preterm infant care. However, there are several uncertainties regarding the impact of mHealth services on parental emotions. Randomized controlled trial (RCT) and quasi-RCT were published in English and Chinese until 2023 and are available in the electronic databases of MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINHAHL), Chinese Biomedical Literature Database, China National Knowledge Infrastructure, Wanfang Database, and China Science and Technology Journal Database were reviewed. Review Manager 5.4 was employed for random-effects meta-analysis. Seven RCTs and 6 quasi-RCTs with high risks were included, most of them were based on small sample size and single-centered. Meta-analysis demonstrated that mHealth interventions mitigated negative emotions, including anxiety (standard mean difference [SMD]=-1.07; 95% confidence interval [CI], -1.47 to -0.66; P<0.0001), depression (SMD=-1.94; 95% CI, -3.01 to -0.86; P<0.0001), and stress (SMD=-1.24; 95% CI, -2.07 to -0.40; P<0.0001), among parents of preterm infants. It is recommended to develop independent specialized software for mHealth, enhance theoretical underpinnings, and standardize practical protocols. Incorporating early recognition and screening of negative emotions within mHealth could offer comprehensive support for the mental health of parents with preterm infants. The mHealth-based interventions are needed to focus on the parents' emotional needs and minimize the impacts associated with mental stress.

"Numbers call for action, personalized narratives provide support and recognition": a qualitative assessment of cancer patients' perspectives on patient-reported outcome measures (PROMs) feedback with narratives.

Patient-reported outcome measures (PROMs) are questionnaires completed by patients to gain insight in their health-related quality of life. However, patients often find the interpretation of PROMS challenging. A personalized narrative, i.e., a story with patients' experiences tailored to the reader, could help explain PROMs and might be appreciated alongside numerical outcomes. We studied how cancer patients perceive PROMs feedback presented in a regular numerical and a novel narrative format. Cancer patients who completed PROMs in routine clinical practice were recruited. All participants received numerical feedback and a personalized narrative. Semi-structured interviews were conducted to uncover perceptions of both formats. Interviews were analyzed with an inductive reflexive approach to thematic analysis. Twenty-nine patients with breast cancer, melanoma, and bladder cancer participated. Thematic analysis identified six themes: "Understanding: I get the gist of it!"; "Usefulness: Tell me why I should complete PROMs"; "Format preferences: Numbers are cold, narratives are warm"; "Taking action: Can I do something about my score?"; "Personal relevance: Personalized narratives show me what life has in store for me"; and "Personal relevance: That's (not) me!" Numbers seemed to help participants act, whereas narratives may provide emotional support and recognition. Participants identified with the content of the narrative yet differed in how they related to the main character. Personalized narratives could be a useful addition to PROMs feedback. The studied formats seem to serve different purposes; numbers help to facilitate action, personalized narratives provide recognition. Personalized narratives may be a useful new way to communicate about quality of life to cancer survivors and help them to envision what the impact of cancer can be.

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition

Affective brain-computer interface is an important part of realizing emotional human–computer interaction. However, existing objective individual differences among subjects significantly hinder the application of electroencephalography (EEG) emotion recognition. Existing methods still lack the complete extraction of subject-invariant representations for EEG and the ability to fuse valuable information from multiple subjects to facilitate the emotion recognition of the target subject. To address the above challenges, we propose a Multi-source Selective Graph Domain Adaptation Network (MSGDAN), which can better utilize data from different source subjects and perform more robust emotion recognition on the target subject. The proposed network extracts and selects the individual information specific to each subject, where public information refers to subject-invariant components from multi-source subjects. Moreover, the graph domain adaptation network captures both functional connectivity and regional states of the brain via a dynamic graph network and then integrates graph domain adaptation to ensure the invariance of both functional connectivity and regional states. To evaluate our method, we conduct cross-subject emotion recognition experiments on the SEED, SEED-IV, and DEAP datasets. The results demonstrate that the MSGDAN has superior classification performance.

The Application of Emotion Valence Ratios in Facial Emotion Recognition for Detecting Depression Among Older Adults in Institutional Settings.

Developing evidence-based parameters to enhance the reliability of face emotion recognition (FER) systems in detecting depression among the elderly is essential. This study aims to elucidate the relationship between the ratio of each emotion valence captured by the FER system and heart rate variability (HRV) while participants watch a video in relation to their depression scores. YOLO, an open-source data analysis toolkit, was used to extract three facial emotion valence features (neutral, positive, and negative) and determine the ratio of each emotion valence over time during video viewing. Additionally, HRV was assessed, and the Geriatric Depression Scale was administered to understand the correlation between FER parameters and depression scores.

Exploiting Optimal Self-Attention Deep Learning-based Recognition of Textual Emotions for Disabled Persons

Exploiting Optimal Self-Attention Deep Learning-based Recognition of Textual Emotions for Disabled Persons

MFGCN: Multimodal fusion graph convolutional network for speech emotion recognition

Speech emotion recognition (SER) is challenging owing to the complexity of emotional representation. Hence, this article focuses on multimodal speech emotion recognition that analyzes the speaker’s sentiment state via audio signals and textual content. Existing multimodal approaches utilize sequential networks to capture the temporal dependency in various feature sequences, ignoring the underlying relations in acoustic and textual modalities. Moreover, current feature-level and decision-level fusion methods have unresolved limitations. Therefore, this paper develops a novel multimodal fusion graph convolutional network that comprehensively executes information interactions within and between the two modalities. Specifically, we construct the intra-modal relations to excavate exclusive intrinsic characteristics in each modality. For the inter-modal fusion, a multi-perspective fusion mechanism is devised to integrate the complementary information between the two modalities. Substantial experiments on the IEMOCAP and RAVDESS datasets and experimental results demonstrate that our approach achieves superior performance.

Comparative study of emotional facial expression recognition among Prader-Willi syndrome subtypes.

Prader-Willi syndrome (PWS) is a congenital disease caused by a rare and generally non-inherited genetic disorder. The inability to recognise facial expressions of emotion is an apparent social cognition deficit in people diagnosed with PWS. The main objective of the present study is to compare the ability to recognise emotional facial expression, in both non-contextualised and contextualised scenarios, among the main subtypes of PWS and a control group. The sample consisted of 46 children divided into three groups: deletion (n=10), maternal uniparental disomy (mUPD) (n=13) and control (n=23). The protocol included the Facially Expressed Emotion Labeling and the Deusto-e-Motion 1.0. The control group recognised facial emotions more accurately and quickly in both non-contextualised and contextualised scenarios than children with PWS, regardless of genetic subtype. Despite no differences being detected between PWS subtypes when non-contextualised scenarios were analysed, in contextualised situations, a longer reaction time was observed in children with the mUPD subtype. This is the first study to assess the ability to recognise emotional facial expressions in contextualised situations among PWS subtypes and a control group. The findings suggest that some of the social cognitive deficits evidenced in children with mUPD PWS may be similar to those in autism spectrum disorder.

Impaired facial emotion recognition in individuals with bipolar disorder

BackgroundIndividuals with bipolar disorder (BD) often struggle with emotional regulation and social interactions, partly due to difficulties in accurately recognizing facial emotions. MethodsFrom September 2021 to February 2023, 69 BD individuals-comprising 23 with bipolar manic/hypomanic episode (BME), 23 with bipolar depressive episode (BDE), 23 with bipolar euthymic (EUT)-and 23 healthy controls (HCs) were enrolled. Diagnosis adhered to DSM-IV criteria using M.I.N.I 5.0, alongside assessments via Hamilton Depression Scale 17 and Young Manic Rating Scale. Recognition tasks involving 84 facial expression images across six categories. The Wilcoxon rank-sum test compares two groups, while the Kruskal-Wallis test compares multiple groups with subsequent adjusted pairwise comparisons. ResultsThe overall correct recognition rate of facial expressions in the BD group (79 %) was significantly lower than that of the HC group (83 %) (P=0.004). Primary differences were noted in neutral (93 % vs. 100 %, P=0.012) and fear (79 % vs. 86 %, P=0.023) expressions. Within the BD group, correct recognition rates were 71 % for BME, 80 % for BDE, and 80 % for EUT, all lower than in the HC group. Significant differences in correct recognition rates of neutral, fear, and joy expressions were observed among the four groups (P<0.05), with the BME group exhibiting the lowest rate. Misidentification of facial expressions was more frequent in the BD group compared to the HC group, particularly among negative expressions. ConclusionPatients with BD demonstrate lower correct recognition and higher misidentification rates of facial expressions, with those experiencing manic episodes showing impaired recognition of neutral, joy, and fear expressions.

Multimodal Emotion Recognition Based on Facial Expressions, Speech, and Body Gestures

The research of multimodal emotion recognition based on facial expressions, speech, and body gestures is crucial for oncoming intelligent human–computer interfaces. However, it is a very difficult task and has seldom been researched in this combination in the past years. Based on the GEMEP and Polish databases, this contribution focuses on trimodal emotion recognition from facial expressions, speech, and body gestures, including feature extraction, feature fusion, and multimodal classification of the three modalities. In particular, for feature fusion, two novel algorithms including supervised least squares multiset kernel canonical correlation analysis (SLSMKCCA) and sparse supervised least squares multiset kernel canonical correlation analysis (SSLSMKCCA) are presented, respectively, to carry out efficient facial expression, speech, and body gesture feature fusion. Different from the traditional multiset kernel canonical correlation analysis (MKCCA) algorithms, our SLSKMCCA algorithm is a supervised version and is based on the least squares form. The SSLSKMCCA algorithm is implemented by the combination of SLSMKCCA and a sparse item (L1 Norm). Moreover, two effective solving algorithms for SLSMKCCA and SSLSMKCCA are presented in addition, which use the alternated least squares and augmented Lagrangian multiplier methods, respectively. The extensive experimental results on the popular public GEMEP and Polish databases show that the recognition rate of multimodal emotion recognition is superior to bimodal and monomodal emotion recognition on average, and our presented SLSMKCCA and SSLSMKCCA fusion methods both obtain very high recognition rates, especially for the SSLSMKCCA fusion method.

Interactive multi-agent convolutional broad learning system for EEG emotion recognition

Electroencephalogram (EEG) emotion recognition is gaining significance in intelligent human–computer interaction. Multi-agent learning can capture more complete and reliable features, compensating for the lack of a single agent’s knowledge. However, previous EEG emotion recognition only focused on a single agent. Therefore, to extract effective features from high-dimensional EEG data, a novel interactive multi-agent (IMA) learning framework is proposed, and introduced into convolutional broad learning system (CNNBLS), then the interactive multi-agent convolutional broad learning system (IMA-CNNBLS) is proposed for EEG emotion recognition. It can effectively model high-dimensional EEG data, automatically extract EEG features related to emotions through convolutional neural network (CNN), and then extend the above features to a vast space to quickly extract generalized features through broad learning system (BLS), consider a CNNBLS as an agent, and multi-agent interact in the IMA part. As the theoretical basis of IMA-CNNBLS, the consistency is mathematically proved through the stationary distribution theory of Markov processes. To demonstrate the superiority of our proposed model, sufficient experiments are conducted on the DEAP, DREAMER and SEED datasets. The experimental results show the model can effectively improve the EEG emotion recognition accuracy, the best recognition performance is achieved on all datasets. In addition, our proposed model also shows the multi-agent interaction significantly affects EEG emotion recognition.

Distinct neural correlates of accuracy and bias in the perception of facial emotion expressions

ABSTRACT We investigated neural correlates of Emotion Recognition Accuracy (ERA) using the Assessment of Contextualized Emotions (ACE). ACE infuses context by presenting emotion expressions in a naturalistic group setting and distinguishes between accurately perceiving intended emotions (signal), and bias due to perceiving additional, secondary emotions (noise). This social perception process is argued to induce perspective taking in addition to pattern matching in ERA. Thirty participants were presented with an fMRI-compatible adaptation of the ACE consisting of blocks of neutral and emotional faces in single and group-embedded settings. Participants rated the central character’s expressions categorically or using scalar scales in consequent fMRI scans. Distinct brain activations were associated with the perception of emotional vs. neutral faces in the four conditions. Moreover, accuracy and bias scores from the original ACE task performed on another day were associated with brain activation during the scalar (vs. categorical) condition for emotional (vs. neutral) faces embedded in group. These findings suggest distinct cognitive mechanisms linked to each type of emotional rating and highlight the importance of considering cognitive bias in the assessment of social emotion perception.

Electroencephalography Emotion Recognition Based on Rhythm Information Entropy Extraction

Electroencephalography (EEG) is a physiological signal directly generated by the central nervous system. Brain rhythm is closely related to a person’s emotional state and is widely used for EEG emotion recognition. In previous studies, the rhythm specificity between different brain channels was seldom explored. In this paper, the rhythm specificity of brain channels is studied to improve the accuracy of EEG emotion recognition. Variational mode decomposition is used to decompose rhythm signals and enhance features, and two kinds of information entropy, i.e., differential entropy (DE) and dispersion entropy (DispEn) are extracted. The rhythm being used to get the best result of single channel emotion recognition is selected as the representative rhythm, and the remove one method is employed to obtain rhythm information entropy feature. In the experiment, the DEAP database was used for EEG emotion recognition in valence-arousal space. The results showed that the best result of rhythm DE feature classification in the valence dimension is 77.04%, and the best result of rhythm DispEn feature classification in the arousal dimension is 79.25%.

Improved ShuffleNet V2 network with attention for speech emotion recognition

This paper enhanced human-robot interaction (HRI) towards interacting with humans through a speech-emotion recognition algorithm based on the improved ShuffleNet V2 network as the means of communication. When recognizing speech emotion, previous studies typically analyze speech signals solely from the time or frequency domain, resulting in the loss of detailed information. Our algorithm initially converts speech data into an acoustic spectrogram to address this limitation, preserving rich original information across both time and frequency domains. This approach optimizes feature extraction by leveraging the capabilities of deep convolutional networks to capture local changes in images. Furthermore, we integrated the attention mechanism from ECA-Net into the ShuffleNet V2 model. This incorporation enhances the extraction of significant features, amplifying the expression of features strongly correlated with the emotional state conveyed in speech while suppressing irrelevant ones. In addition to these enhancements, we replace the ReLU activation function with Hardswish to deepen the neural network, thereby improving feature information without compromising model accuracy. Finally, the training stage incorporates the hot restart Cosine Annealing Learning Rate to optimize the network further. Simulation experiments are conducted on the enterface'05 dataset to evaluate the accuracy of the proposed algorithm. The experimental results show that the recognition rate of the method is higher than that of the existing algorithm in terms of the effectiveness of HRI.

Optimizing Interactive Online Education Through Advanced Multimodal Emotion Recognition Techniques

Driven by the global pandemic, the demand for online education has significantly increased, making it crucial to enhance the interactive experience of online teaching to improve student learning outcomes and engagement. In this context, we have designed a model based on Particle Swarm Optimization (PSO) that combines the strengths of LSTNet in handling long-term dependencies and the capabilities of the Prophet model in trend and seasonality modeling. Our PSO-optimized LSTNet-Prophet model outperformed baseline models by 10% in accuracy and 12% in F-1 score, as shown by experiments conducted on the CMU-MOSEI dataset. Additionally, we have implemented a real-time emotion monitoring system capable of analyzing students' emotional states in real-time and providing feedback to teachers, aiding them in promptly adjusting teaching strategies, thereby improving teaching quality and interaction effectiveness. Our method achieved an emotion recognition accuracy of 82.42% and an F-1 score of 82.31%, demonstrating its effectiveness and robustness.

Multiplex graph aggregation and feature refinement for unsupervised incomplete multimodal emotion recognition

Multimodal Emotion Recognition (MER) involves integrating information of various modalities, including audio, visual, text and physiological signals, to comprehensively grasp human sentiments, which has emerged as a vibrant area within human–computer interaction. Researchers have developed many methods for this task, but many of these methods rely on labeled supervised learning and struggle to address the issue of missing some modalities of data. To address these issues, we propose a Multiplex Graph Aggregation and Feature Refinement framework for unsupervised incomplete MER, comprising four modules: Completion, Aggregation, Refinement, and Embedding. Specifically, we first capture the correlation information between samples using the graph structures, which aids in the completion of missing data and the multiplex aggregation of multimodal data. Then, we perform refinement operations on the aggregated features as well as alignment and enhancement operations on the embedding features to obtain the fused feature representations, which are consistent, highly separable and conducive to emotion recognition. Experimental results on multimodal emotion recognition datasets demonstrate that our method achieves state-of-the-art performance among unsupervised methods, validating its effectiveness.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.

Machine learning-enabled mental health risk prediction for youths with stressful life events: A modelling study

BackgroundYouths face significant mental health challenges exacerbated by stressful life events, particularly in the context of the COVID-19 pandemic. Immature coping strategies can worsen mental health outcomes. MethodsThis study utilised a two-wave cross-sectional survey design with data collected from Chinese youth aged 14–25 years. Wave 1 (N = 3038) and Wave 2 (N = 539) datasets were used for model development and external validation, respectively. Twenty-five features, encompassing dimensions related to demographic information, stressful life events, social support, coping strategies, and emotional intelligence, were input into the model to predict the mental health status of youth, which was considered their coping outcome. Shapley additive explanation (SHAP) was used to determine the importance of each risk factor in the feature selection. The intersection of top 10 features identified by random forest and XGBoost were considered the most influential predictors of mental health during the feature selection process, and was then taken as the final set of features for model development. Machine learning models, including logistic regression, AdaBoost, and a backpropagation neural network (BPNN), were trained to predict the outcomes. The optimum model was selected according to the performance in both internal and external validation. ResultsThis study identified six key features that were significantly associated with mental health outcomes: punishment, adaptation issues, self-regulation of emotions, learning pressure, use of social support, and recognition of others' emotions. The BPNN model, optimized through feature selection methods like SHAP, demonstrated superior performance in internal validation (C-index [95 % CI] = 0.9120 [0.9111, 0.9129], F-score [95 % CI] = 0.8861 [0.8853, 0.8869]). Additionally, external validation showed the model had strong discrimination (C-index = 0.9749, F-score = 0.8442) and calibration (Brier score = 0.029) capabilities. LimitationsAlthough the clinical prediction model performed well, the study it still limited by self-reported data and representativeness of samples. Causal relationships need to be established to interpret the coping mechanism from multiple perspectives. Also, the limited data on minority groups may lead to algorithmic unfairness. ConclusionsMachine learning models effectively identified and predicted mental health outcomes among youths, with the SHAP+BPNN model showing promising clinical applicability. These findings emphasise the importance and effectiveness of targeted interventions with the help of clinical prediction model.

How does emotional content influence visual word recognition? A meta-analysis of valence effects

AbstractResearch on the interplay between language and emotion has shown evidence that the affective content of words influences their recognition. However, the direction of the effects is not clear, as there are mixed findings regarding the role of positive and, especially, negative valence. We conducted a Bayesian multi-level meta-analysis to examine the role of valence in visual word recognition, focusing on the lexical decision task. The results revealed a facilitative effect of positive valence on lexical decision times. That is, positive words led to faster responses than both negative and neutral words. In contrast, negative valence did not have any effect, although the analysis of several moderator variables suggested that there might be a facilitative effect in some cases, specifically, when negative words elicit very strong and intense emotions. These results shed light on the complexities of emotional word processing. They also point to the need for psycholinguistic models to take affective information into account, and thus provide a complete view of visual word recognition.

EEG emotion recognition approach using multi-scale convolution and feature fusion

EEG emotion recognition approach using multi-scale convolution and feature fusion

Facial emotion recognition in children with attention deficit hyperactivity disorder

Objective Attention Deficit Hyperactivity Disorder (ADHD) is defined as a persistent pattern of inattention and hyperactivity-impulsivity that interferes with functioning anofd development. Increased emotional reactivity and impaired emotion regulation are established findings in children with ADHD. Impairments in executive functions such as impulse control and working memory, in turn, have also been suggested to have a negative effect on emotion recognition. However, studies exploring suspected deficits in the ability to recognise facial emotions in ADHD have to date yielded controversial results. We sought to clarify the mechanism of possible emotion recognition dysfunction in children with ADHD. Methods Sixty-one children diagnosed with ADHD (aged 10.36 ± 1.89 years) and a control group (N = 78; aged 9.6 ± 1.8 years) were evaluated with questionnaires and computerized tests for cognitive and facial emotion recognition capacity. Results The ADHD group displayed more behavioural issues and performed worse in cognitive tests compared to the control group. Group status (i.e. ADHD vs. control group) did not predict facial emotion recognition when controlled for age, IQ and sex in linear regression models. Performance in Divided Attention predicted facial emotion recognition in linear regression in the ADHD group. Conclusions Individuals with ADHD showed facial emotion recognition capacity similar to a typically developing control group. Good performance in a cognitive test assessing divided attention predicted capacity for facial emotion recognition, but only in the ADHD group.