Emotion Recognition Research Articles

Audiovisual emotion recognition based on bi-layer LSTM and multi-head attention mechanism on RAVDESS dataset

Audiovisual emotion recognition based on bi-layer LSTM and multi-head attention mechanism on RAVDESS dataset

Association of neuroimaging measures with facial emotional processing in healthy adults: a task fMRI study.

Investigating the neural processing of emotion-related neural circuits underlying emotional facial processing may help in understanding mental disorders. We used two subscales of the Toronto Alexithymia Scale to assess the emotional cognitive of 25 healthy participants. A higher score indicates greater difficulty in emotional perception. In addition, participants completed a n-back task during functional magnetic resonance imaging. Psychophysiological interaction analysis was used to explore the functional connectivity (FC) of neural circuits. Next, we used elastic-net regression analysis for feature selection and conducted correlation analysis between the neuroimaging measures and questionnaire scores. Following a 3-fold cross-validation, 5 neuroimaging measures emerged as significant features. Results of correlation analysis demonstrated that participants with higher TAS scores exhibited increased FC between the amygdala and occipital face area (OFA) during facial stimulus processing, but decreased connectivity during emotional processing. These findings suggested that individuals with poor emotional recognition exhibited increased connectivity among face-related brain regions during facial processing. However, during emotional processing, decreasing neural synchronization among neural circuits involved in emotional processing affects facial expression processing. These findings suggest potential neural marker related to subjective emotional perception, which may contribute to the diagnosis and treatment of emotional dysregulation in individuals with psychiatric conditions.

Introducing a new social cognition online therapy: SoCoBo

ABSTRACT Integrating technology-based therapies into existing treatment approaches has proven useful in the endeavour to improve impaired social cognition in patients with neurological diseases. To this end, we present a novel online-based therapy programme for the treatment of impairments of social cognition (“SoCoBo”), particularly tailored for patients with acquired brain injuries (ABIs). This study was designed to investigate the online therapy's feasibility including processing time and acceptability in a healthy older sample as, according to previous studies, older individuals show subtle impairments of social cognition. Between 50 and 52 participants (depending on outcome measures) underwent pre–post assessments, completed one out of three therapy modules (emotion recognition, perspective taking, social problem-solving) over a period of four weeks (four sessions per week) and evaluated the therapy concerning feasibility, acceptability and content aspects with a newly developed questionnaire. All modules showed comparable ratings on a low to moderate level in terms of feasibility, acceptability and content aspects, which seems plausible due to the sample under investigation. Processing time of the three modules was comparable for psychoeducation (mean 17.45 min) and training sessions (mean 21.91 min). The results provided important indications for necessary adjustments regarding a subsequently completed randomized controlled trial involving patients with ABIs.

Research on Speech Emotion Recognition Method Based on ResSE_CNN1D

This article examines a speech emotion recognition (SER) technique based on the enhanced one-dimensional convolution neural network ResSE_CNN1D. With the artificial intelligence developing rapidly, SER has a profound impact in many areas. The model of this article is used to extract the characteristics of the input data through opensmile, and is sent into the ResSE_CNN1D model, which is eventually classified by the softmax activation function and obtains the final results. The key to this model is efficient learning of decimal sets and the rapid deployment in resource-constrained environments. The ResSE_CNN1D model improves the performance of the model by adding the residual connection and the SE module on the basis of cnn1d. This increasing the accuracy of the recognition and preventing the fitting problem. After the model was created, the study adopted the audio sampling and training of the casia data concentration. The final accuracy was 0.900, which increased the accuracy of 2.9 percent compared to the cnn1d method. And by the analysis of the relationship diagram of the confusion matrix and the accuracy and loss rate relative to the number of training, the model has a high robustness and effectively prevents the appearance of the fitting problem. And it also can achieve high precision and achieve a lightweight goal relative to less training.

Cross-Modal Harmony: Low-Rank Fusion for Enhanced Artificial Emotion Recognition

Emotion recognition has become a subject of considerable interest in recent times, owing to its diverse and far-reaching applications in various fields. These applications span from enhancing human-computer interactions to assessing mental health and improving entertainment systems. The proposed study presents a novel approach for emotion recognition by fusing audio and video modalities using low-rank fusion techniques. The proposed methodology leverages the complementary nature of audio and video data in capturing emotional cues. Audio data often encapsulates tone, speech patterns, and vocal nuances, while video data captures facial expressions, body language, and gestures. However, the challenge lies in effectively integrating these two modalities to enhance recognition accuracy. To address the challenge, it employs low-rank fusion, a dimensionality reduction technique that extracts the most informative features from both modalities while minimizing redundancy. Furthermore, it presents the implementation of the chosen low-rank fusion algorithm in a real-world emotion recognition system. The results can contribute to advancing the field of emotion recognition by providing a practical and efficient solution for combining audio and video data to achieve more robust and accurate emotion classification. Keywords: Deep Learning; Emotion Recognition; Human-Computer Interaction; Low-Rank Fusion; Multimodal Fusion.

Multi-source domain transfer network based on subdomain adaptation and minimum class confusion for EEG emotion recognition

Electroencephalogram (EEG) signals, which objectively reflect the state of the brain, are widely favored in emotion recognition research. However, the presence of cross-session and cross-subject variation in EEG signals has hindered the practical implementation of EEG-based emotion recognition technologies. In this article, we propose a multi-source domain transfer method based on subdomain adaptation and minimum class confusion (MS-SAMCC) in response to the addressed issue. First, we introduce the mix-up data augmentation technique to generate augmented samples. Next, we propose a minimum class confusion subdomain adaptation method (MCCSA) as a sub-module of the multi-source domain adaptation module. This approach enables global alignment between each source domain and the target domain, while also achieving alignment among individual subdomains within them. Additionally, we employ minimum class confusion (MCC) as a regularizer for this sub-module. We performed experiments on SEED, SEED IV, and FACED datasets. In the cross-subject experiments, our method achieved mean classification accuracies of 87.14% on SEED, 63.24% on SEED IV, and 42.07% on FACED. In the cross-session experiments, our approach obtained average classification accuracies of 94.20% on SEED and 71.66% on SEED IV. These results demonstrate that the MS-SAMCC approach proposed in this study can effectively address EEG-based emotion recognition tasks.

THE ROLE OF EMOTIONS IN CONSUMER DECISION-MAKING: ANALYZE HOW EMOTIONAL APPEAL IN MARKETING AFFECTS CONSUMER BEHAVIOR

Emotions play a pivotal role in consumer decision-making, influencing how individuals perceive, evaluate, and choose products or services. Emotional appeal in marketing taps into consumers' feelings, often bypassing rational thought to create strong associations with a brand or product. This paper analyzes how emotional appeal shapes consumer behavior by building brand loyalty, enhancing engagement, and driving impulse purchases. Positive emotions such as happiness, excitement, and belonging can foster long-term consumer relationships, while negative emotions like fear or anxiety can prompt immediate action. Through emotional resonance, brands not only differentiate themselves in a crowded marketplace but also create memorable experiences that impact purchasing decisions. The findings suggest that successful emotional marketing strategies are those that align with consumers’ values, desires, and emotional states, highlighting the importance of understanding both rational and emotional drivers in shaping consumer behavior. KEYWORDS Emotional Appeal, Consumer Behavior, Brand Loyalty, Impulse Buying, Positive Emotions, Negative Emotions, Engagement, Emotional Marketing, Brand Differentiation, Cultural Sensitivity, Neuromarketing, AI and Emotion Recognition, Personalization, Social Media, Sustainability, Behavioral Economics.

Intelligent Borders: Exploring the Suitability of Artificial Intelligence Systems in Refugee Status Determination Under International Law

Abstract This article assesses the potential use of Artificial Intelligence (AI) in Refugee Status Determination, exploring how AI systems, including biometrics, predictive analytics, and emotion recognition, could support or replace human decision-making in determining refugee status. While AI could speed up and improve the objectivity of Refugee Status Determination procedures, the article raises concerns about its limitations in assessing the subjective fear of persecution, a critical element of refugee claims. The inability of AI systems to fully account for emotional and personal nuances leads to the possibility of discriminatory practices, particularly when AI is used without human oversight. The study highlights how AI could exacerbate power imbalances between asylum-seekers and States, transforming these technologies into tools for anti-immigration policies. It therefore calls for caution in the use of AI in Refugee Status Determination procedures and stresses that human interaction remains essential for the fair assessment of refugee claims. This article concludes that while AI can assist in certain aspects of the Refugee Status Determination procedures, its full implementation without careful legal and human rights considerations poses significant risks.

Emotion Classification from Speech Waveform Using Machine Learning and Deep Learning Techniques

Emotions play a key role in determining the human mental state and indirectly express an individual’s well- being. A speech emotion recognition system can extract a person’s emotions from his/her speech inputs. There are some universal emotions such as anger, disgust, fear, happiness, pleasantness, sadness and neutral. These emotions are of significance especially in a situation like the Covid pandemic, when the aged or sick are vulnerable to depression. In the current paper, we examined various classification models with finite computational strength and resources in order to determine the emotion of a person from his/her speech. Speech prosodic features like pitch, loudness, and tone of speech, and work spectral features such as Mel Frequency Capstral Coefficients (MFCCs) of the voice were used to analyze the emotions of a person. Although sequence to sequence state of the art models for speech detection that offer high levels of accuracy and precision are currently in use, the computational needs of such approaches are high and inefficient. Therefore, in this work, we emphasised analysis and comparison of different classification algorithms such as multi layer perceptron, decision tree, support vector machine, and deep neural networks such as convolutional neural network and long short term memory. Given an audio file, the emotions that were exhibited by the speaker were recognized using machine learning and deep learning techniques. A comparative study was performed to identify the most appropriate algorithms that could be used to recognize emotions. Based on the experiment results, the MLP classifier and convolutional neural network model offered better accuracy with smaller variations when compared with other models used for the study.

Visual Attention and Cardiac Response to Faces in Relation to Emotion Recognition

Abstract: Rapid and accurate emotion recognition is a crucial skill for social interactions, and visual attention to informative social cues in the environment can facilitate emotion recognition. Studies have found that emotion recognition is also associated with underlying cardiac autonomic responses. The current study examined (1) visual attention and cardiac response (change in heart rate) to emotionally-expressive faces and houses (as a nonsocial control image), and (2) associations between these responses and emotion recognition on the Reading the Mind in the Eyes Test (RMET) in college students. Results showed increased overall attention, but shorter first fixation durations, to faces as compared to houses. Across faces, attention was greater to eyes than mouth, but the magnitude of this difference was dependent on emotion. Analysis of cardiac responses revealed greater heart rate deceleration to happy and fearful faces as compared to neutral faces and houses. Better emotion recognition accuracy on the RMET was related to greater attention to faces, and eyes specifically, as well as greater heart rate deceleration to faces relative to houses. No relations between visual attention and cardiac response were found. The current work points to variations in visual attention and cardiac responses while viewing emotional faces that can be markers of emotion processing ability.

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition.

EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

Disturbances of Interoception in Schizophrenia – Focus on Neuroimaging

Abstract: The current paper presents the most recent theoretical understanding of interoception (INTC) in light of the predictive processing framework and the Embodied Predictive Interoceptive Coding model and reviews recent experimental data on different interoceptive disturbances in schizophrenia (SCZ) such as lower interoceptive accuracy, nociceptive dysfunction reflected by reduced pain sensitivity, disturbed thermoregulation, impaired gastrointestinal regulation, cardiovascular and respiratory dysfunction, and immune disturbances. The immense clinical implications of these disturbances are discussed in the context of the high rates of morbidity and mortality in SCZ related to suicide and cardiac death. Next, the potential of INTC-related treatments is reviewed, and conclusions are made on the evidence for the use of vagus nerve stimulation, mindfulness-based interventions, and exercise therapy. The neuroimaging perspective is given in a brief account of the INTC-related findings in healthy individuals suggesting the right anterior insula and anterior cingulate cortex as key brain areas for interoceptive awareness, accuracy, and sensibility, by supporting higher-order representation and integration of visceral, and somatosensory interoception with exteroceptive information. In SCZ, on the other hand, neuroimaging research on specific interoceptive abilities is lacking but indirect evidence points to disrupted activation of major INTC network hubs. There is converging evidence of changes in these hubs in terms of both structure and function (volume reductions, aberrant activation, and connectivity). This narrative review identifies several gaps in contemporary INTC-related research in SCZ that need to be addressed in future experimental and clinical studies to elucidate the exact nature of the disturbances of interoceptive abilities in modalities other than cardioception; the interactions between different interoceptive modalities; the association between interoceptive abilities and other mental functions (such as emotion recognition and regulation, decision making, learning, memory, etc.); the efficacy of nonpharmacological and pharmacological INTC-based interventions; and the relation of disturbed interoceptive abilities with brain structure and function.

Exploring the application of synchronous analysis of physiological signals and facial expressions in emotion recognition

In today's increasingly frequent human-computer interaction, the accurate recognition and response of intelligent systems to human emotions has become the key to improving the user experience. This paper aims to explore the application of physiological signals and facial expression synchronization analysis in emotion recognition. This paper adopts the method of literature review, and summarizes the acquisition techniques of physiological signals and facial expressions, synchronous analysis methods, and their application cases in emotion recognition. At the same time, the limitations and challenges of the existing technology are discussed. The review shows that the synchronous analysis of physiological signals and facial expressions has important application value in the field of emotion recognition, especially in mental health monitoring, educational feedback, and customer service automation. However, the practical application of the technology still faces many challenges, including the real-time nature of data collection, the adaptability of individual differences, and other identification problems. Therefore, future research needs to further optimize data acquisition technology and develop more accurate and personalized analysis algorithms.

A study of long-term supraphysiologic-dose anabolic-androgenic steroid use on cognitive function in middle-aged men

ABSTRACT Background: Long-term use of supraphysiologic doses of anabolic-androgenic steroids (AAS) has been associated with impaired visuospatial memory in young men but little is known about its cognitive effects in middle-aged men. Objectives: We compared cognition in middle-aged men with histories of long-term AAS use and age-matched non-users. Methods: We administered cognitive tests from the CANTAB battery to 76 weightlifters aged 37–60 years (mean [SD] 48.5 [6.5] years), of whom 51 reported at least 2 years of cumulative AAS use and 25 reported no AAS exposure. Results: We found no significant AAS user versus non-user group differences on visuospatial, verbal memory, emotional recognition, or executive function tasks (corrected p’s ≥ .00089; effect sizes ≤ .5). Conclusions: Our null visuospatial task findings contrast with our prior younger cohort study (mean age 37.1 [7.1] years), in which we found impaired visuospatial task performance in people who use AAS, and with other reports of cognitive impairments in younger men use AAS. Men who use AAS may develop early visuospatial memory deficits that stabilize by middle age while middle-aged non-users’ performance may “catch up” due to normal age-related visuospatial declines. Similar effects could contribute to our null findings on other tasks. Between-study cohort substance use differences or environmental factor differences that modify cognition, such as study geographical location and time of year, also could contribute to our discordant findings. Since young adult male AAS users experience increased mortality from unnatural causes, improving our understanding of AAS cognitive effects in this age group is important.

Emotion-specific recognition biases and how they relate to emotion-specific recognition accuracy, family and child demographic factors, and social behaviour

ABSTRACT The errors young children make when recognising others’ emotions may be systematic over-identification biases and may partially explain the challenges some have socially. These biases and associations may be differential by emotion. In a sample of 871 ethnically and racially diverse preschool-aged children (i.e. 33–68 months; 49% Hispanic/Latine, 52% Children of Colour), emotion recognition was assessed, and scores for accuracy and bias were calculated by emotion (i.e. anger, sad, happy, calm, and fear). Child and family characteristics and teacher-reported social behaviour were also collected. Multilevel structural equation modelling revealed emotion-specific recognition accuracies varied between 36 and 65% whereas biases varied between 4 and 13%. Anger was the strongest bias followed by sad, happy, fear, and calm, in contrast to the pattern for accuracy – happy, sad, angry, fear, and calm. More variance was explained in emotion-specific recognition accuracies by child and family characteristics – 7–38% – than biases – 3–7%. Negatively-valanced emotion recognition biases associated with positively-valanced accuracies, and positively-valued emotion recognition biases associated with negatively-valued accuracies. Biases did not have meaningful associations with social behaviour. This study highlights that children’s emotion recognition errors may partially be systematic, but future studies are needed to understand the underlying cognitive mechanisms.

Exploring college students' risk perception and acceptance intention of facial recognition technology in China

In response to China’s ‘smart campus’ trend, facial recognition technology (FRT) in higher education has evolved from identification to emotion recognition, raising various risks and controversies. Understanding college students’ perceptions is key to addressing these concerns and facilitating AI integration. This study extends the Unified Theory of Acceptance and Use of Technology (UTAUT) model to gauge students’ attitudes towards FRT, introducing three new constructs: ‘personal innovativeness,’ ‘trust in the system,’ and ‘perceived risks.’ Results show that students’ personal innovativeness and trust in the system positively relate to their acceptance intention. Trust in FRT is hindered by perceived privacy, psychological, and performance risks, while perceived social risks are insignificant. Higher education shows less resistance to FRT due to normalized campus surveillance. Compared to Western societies, Chinese students’ Confucian cultural stance of ‘harmony but not uniformity’ reduces concerns about racial or religious discrimination from FRT. These findings are crucial for assessing FRT’s necessity and legality in higher education and devising risk mitigation strategies.

Children with facial differences experience deficits in emotion skills.

Children with facial differences, such as repaired cleft lip and palate (CLP), may present with reduced capacity for sensorimotor simulation, particularly in the form of facial mimicry. This study examined whether facial mimicry, emotion recognition, and empathy skills are reduced in children with CLP when compared with sex/age-matched controls. A case-control design was utilized. Forty-five children between the ages of 8 and 12 with CLP, and 45 age/sex-matched controls were recruited. Participants completed a facial mimicry task, and facial movements were tracked and quantified using OpenFace. Participants also completed picture and context-based emotion recognition tasks. Picture-based assessment involved identifying emotions from the Dynamic FACES database. Context-based assessment consisted of identifying how a child might feel in various situations. Finally, participants and their parents completed the Empathy Questionnaire for Children and Adolescents (EmQue-CA). Children with CLP presented with significantly reduced facial mimicry (p = .017), picture-based (p < .001), and context-based emotion recognition scores (p < .001) when compared with controls. Better facial mimicry was associated with better picture-based emotion recognition scores in the control group only (r = .22, p < .01). Children with CLP also had significantly lower child and parent-proxy EmQue-CA scores (p < .001). Greater facial mimicry significantly predicted better parent-proxy EmQue-CA scores (p = .016) but did not predict child scores. Children with CLP presented with reduced facial mimicry, poorer emotion recognition, and empathy skills. These findings have a broader relevance as they suggest children with facial differences may present with reduced facial mimicry and/or deficits in emotion recognition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A facial and vocal emotion recognition system based on deep learning

The traditional teaching mode, mainly characterized by "cramming" instruction, often overlooks the absorption of knowledge by students during the teaching process. Even in recent years, the emerging "flipped classroom" model heavily relies on teachers' experience and expertise, making it challenging to quantify students' classroom learning states. Among young teachers lacking teaching experience, the replication of excellent teaching models faces difficulties. With the rapid development of artificial intelligence technology, deep learning and other AI techniques are increasingly applied to people's lives. Additionally, emotion computing technology is also becoming more sophisticated. In the field of education, the use of AI technology holds the potential to help teachers systematically and comprehensively assess teaching quality, promoting the sharing of high-quality educational resources nationally and globally. Existing emotion detection technologies often focus on single techniques such as image recognition or natural language processing, resulting in low accuracy in discerning human emotions and making it difficult to accurately identify emotional changes in complex scenarios such as classrooms. Furthermore, there is a lack of comprehensive emotional statistical analysis methods for audio and visual data. In light of these, we have designed and implemented a multi-modal emotion recognition and analysis system. Through testing in classroom settings, this system has the potential to assist classroom teaching and enhance teaching quality.

Multimodal Information Fusion and Data Generation for Evaluation of Second Language Emotional Expression

This study aims to develop an emotion evaluation method for second language learners, utilizing multimodal information to comprehensively evaluate students’ emotional expressions. Addressing the limitations of existing emotion evaluation methods, which primarily focus on the acoustic features of speech (e.g., pronunciation, frequency, and rhythm) and often neglect the emotional expressions conveyed through voice and facial videos, this paper proposes an emotion evaluation method based on multimodal information. The method includes the following three main parts: (1) generating virtual data using a Large Language Model (LLM) and audio-driven facial video synthesis, as well as integrating the IEMOCAP dataset with self-recorded student videos and audios containing teacher ratings to construct a multimodal emotion evaluation dataset; (2) a graph convolution-based emotion feature encoding network to extract emotion features from multimodal information; and (3) an emotion evaluation network based on Kolmogorov–Arnold Networks (KAN) to compare students’ emotion features with standard synthetic data for precise evaluation. The emotion recognition method achieves an unweighted accuracy (UA) of 68.02% and an F1 score of 67.11% in experiments with the IEMOCAP dataset and TTS data. The emotion evaluation model, using the KAN network, outperforms the MLP network, with a mean squared error (MSE) of 0.811 compared to 0.943, providing a reliable tool for evaluating language learners’ emotional expressions.

ESERNet: Learning spectrogram structure relationship for effective speech emotion recognition with swin transformer in classroom discourse analysis

Speech emotion recognition (SER) has received increased attention due to its extensive applications in many fields, especially in the analysis of teacher-student dialogue in classroom environment. It can help teachers to better learn about students’ emotions and thereby adjust teaching activities. However, SER has faced several challenges, such as the intrinsic ambiguity of emotions and the complex task of interpreting emotions from speech in noisy environments. These issues can result in reduced recognition accuracy due to a focus on less relevant or insignificant features. To address these challenges, this paper presents ESERNet, a Transformer-based model designed to effectively extract crucial clues from speech data by capturing both pivotal cues and long-range relationships in speech signal. The major contribution of our approach is a two-pathway SER framework. By leveraging the Transformer architecture, ESERNet captures long-range dependencies within speech mel-spectrograms, enabling a refined understanding of the emotional cues embedded in speech signals. Extensive experiments were conducted on the IEMOCAP and EmoDB datasets, the results show that ESERNet achieves state-of-the-art performance in SER and outperforms existing methods by effectively leveraging critical clues and capturing long-range dependencies in speech data. These results highlight the effectiveness of the model in addressing the complex challenges associated with SER tasks.