Emotion Recognition System Research Articles

An emotion recognition method based on EWT-3D–CNN–BiLSTM-GRU-AT model

This has become a significant study area in recent years because of its use in brain-machine interaction (BMI). The robustness problem of emotion classification is one of the most basic approaches for improving the quality of emotion recognition systems. One of the two main branches of these approaches deals with the problem by extracting the features using manual engineering and the other is the famous artificial intelligence approach, which infers features of EEG data. This study proposes a novel method that considers the characteristic behavior of EEG recordings and based on the artificial intelligence method. The EEG signal is a noisy signal with a non-stationary and non-linear form. Using the Empirical Wavelet Transform (EWT) signal decomposition method, the signal's frequency components are obtained. Then, frequency-based features, linear and non-linear features are extracted. The resulting frequency-based, linear, and nonlinear features are mapped to the 2-D axis according to the positions of the EEG electrodes. By merging this 2-D images, 3-D images are constructed. In this way, the multichannel brain frequency of EEG recordings, spatial and temporal relationship are combined. Lastly, 3-D deep learning framework was constructed, which was combined with convolutional neural network (CNN), bidirectional long-short term memory (BiLSTM) and gated recurrent unit (GRU) with self-attention (AT). This model is named EWT-3D–CNN–BiLSTM-GRU-AT. As a result, we have created framework comprising handcrafted features generated and cascaded from state-of-the-art deep learning models. The framework is evaluated on the DEAP recordings based on the person-independent approach. The experimental findings demonstrate that the developed model can achieve classification accuracies of 90.57 % and 90.59 % for valence and arousal axes, respectively, for the DEAP database. Compared with existing cutting-edge emotion classification models, the proposed framework exhibits superior results for classifying human emotions.

An Efficient Graph Learning System for Emotion Recognition Inspired by the Cognitive Prior Graph of EEG Brain Network.

Benefiting from the high-temporal resolution of electroencephalogram (EEG), EEG-based emotion recognition has become one of the hotspots of affective computing. For EEG-based emotion recognition systems, it is crucial to utilize state-of-the-art learning strategies to automatically learn emotion-related brain cognitive patterns from emotional EEG signals, and the learned stable cognitive patterns effectively ensure the robustness of the emotion recognition system. In this work, to realize the efficient decoding of emotional EEG, we propose a graph learning system Graph Convolutional Network framework with Brain network initial inspiration and Fused attention mechanism (BF-GCN) inspired by the brain cognitive mechanism to automatically learn graph patterns from emotional EEG and improve the performance of EEG emotion recognition. In the proposed BF-GCN, three graph branches, i.e., cognition-inspired functional graph branch, data-driven graph branch, and fused common graph branch, are first elaborately designed to automatically learn emotional cognitive graph patterns from emotional EEG signals. And then, the attention mechanism is adopted to further capture the brain activation graph patterns that are related to emotion cognition to achieve an efficient representation of emotional EEG signals. Essentially, the proposed BF-CGN model is a cognition-inspired graph learning neural network model, which utilizes the spectral graph filtering theory in the automatic learning and extracting of emotional EEG graph patterns. To evaluate the performance of the BF-GCN graph learning system, we conducted subject-dependent and subject-independent experiments on two public datasets, i.e., SEED and SEED-IV. The proposed BF-GCN graph learning system has achieved 97.44% (SEED) and 89.55% (SEED-IV) in subject-dependent experiments, and the results in subject-independent experiments have achieved 92.72% (SEED) and 82.03% (SEED-IV), respectively. The state-of-the-art performance indicates that the proposed BF-GCN graph learning system has a robust performance in EEG-based emotion recognition, which provides a promising direction for affective computing.

Selective Acoustic Feature Enhancement for Speech Emotion Recognition With Noisy Speech.

A speech emotion recognition (SER) system deployed on a real-world application can encounter speech contaminated with unconstrained background noise. To deal with this issue, a speech enhancement (SE) module can be attached to the SER system to compensate for the environmental difference of an input. Although the SE module can improve the quality and intelligibility of a given speech, there is a risk of affecting discriminative acoustic features for SER that are resilient to environmental differences. Exploring this idea, we propose to enhance only weak features that degrade the emotion recognition performance. Our model first identifies weak feature sets by using multiple models trained with one acoustic feature at a time using clean speech. After training the single-feature models, we rank each speech feature by measuring three criteria: performance, robustness, and a joint rank ranking that combines performance and robustness. We group the weak features by cumulatively incrementing the features from the bottom to the top of each rank. Once the weak feature set is defined, we only enhance those weak features, keeping the resilient features unchanged. We implement these ideas with the low-level descriptors (LLDs). We show that directly enhancing the weak LLDs leads to better performance than extracting LLDs from an enhanced speech signal. Our experiment with clean and noisy versions of the MSP-Podcast corpus shows that the proposed approach yields a 17.7% (arousal), 21.2% (dominance), and 3.3% (valence) performance gains over a system that enhances all the LLDs for the 10dB signal-to-noise ratio (SNR) condition.

Research on the Path of English Translation Teaching and External Communication Based on the Multimodal Analysis Method

Abstract With the development of the times and the continuous innovation of modern education technology, the multimodal theory is developing deeply in the field of education. The first step in this study is to use the optimization principle to select modes that meet the needs of English translation teaching and then develop a multimodal analysis framework for external communication that incorporates multimedia communication methods. Then, we constructed an emotion recognition model for English translation teaching by using speech recognition and expression recognition combined with convolutional neural networks for the students’ classroom situation and attitude and carried out the empirical analysis. The recognition accuracy of the emotion recognition system constructed in this paper is 86.4%. In the multimodal analysis of English translation teaching, the teacher who uses the linguistic modality type for the most number of times 387 times, and the teacher who uses the mixed modality for the most number of times 271 times, the students have the best classroom status. It has been stable at about 5 points, which is a “more positive” state. The students in the class had the highest level of motivation. Through the analysis, it is found that multimodal teaching can effectively improve the students’ status in class and keep their attention, which verifies the appropriateness of the external communication path combined with multimedia communication in multimodal English translation teaching. It provides a practical path for the development of English translation teaching.

Automated human emotion recognition system using TQWT based EEG sub-bands

Automated human emotion recognition system using TQWT based EEG sub-bands

Triangular Region Cut-Mix Augmentation Algorithm-Based Speech Emotion Recognition System With Transfer Learning Approach

Triangular Region Cut-Mix Augmentation Algorithm-Based Speech Emotion Recognition System With Transfer Learning Approach

Ethical Considerations and Checklist for Affective Research With Wearables

As the popularity of wearables increases, so does their utility for studying emotions. Using new technologies points to several ethical challenges to be considered to improve research designs. There are several ethical recommendations for utilizing wearables to study human emotions, but they focus on emotion recognition systems applications rather than research design and implementation. To address this gap, we have developed a perspective on wearables, especially in daily life, adapting the ReCODE Health - Digital Health Framework and companion checklist. Therefore, our framework consists of four domains: (1) participation experience, (2) privacy, (3) data management, and (4) access and usability. We identified 33 primary risks of using wearables to study emotions, including research-related negative emotions, collecting, processing, storing, sharing personal and biological information, commercial technology validity and reliability, and exclusivity issues. We also proposed possible strategies for minimizing risks. We consulted the new ethical guidelines with members of ethics committees and relevant researchers. The judges ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> = 26) positively rated our solutions and provided useful feedback that helped us refine our guidance. Finally, we summarized our proposals with a checklist for researchers' convenience. Our guidelines contribute to future research by providing improved protection of participants' and scientists' interests.

Human Behaviour Analysis Using CNN

Emotion recognition has been the subject of extensive research due to its significant impact on various domains, including healthcare, human-computer interaction, and marketing. Traditional methods of emotion recognition rely on visual cues, such as facial expressions, to decipher emotional states. However, these methods often fall short when dealing with individuals who have limited ability to express emotions through facial expressions, such as individuals with certain neurological disorders. This research paper proposes a novel approach to emotion recognition by combining facial expression analysis with electroencephalography (EEG) data. Deep learning techniques are applied to extract features from facial expressions captured through video analysis, while simultaneously analyzing the corresponding EEG signals. The goal is to improve emotion recognition accuracy by utilizing the complementary information offered by the interaction between facial expressions and EEG data. Emotion recognition is a challenging task that has collected considerable recognition in the current years. Different and refined approaches to recognize emotions based on facial expressions, voice analysis, physiological signals, and behavioral patterns have been developed. While facial expression analysis has been a dominant approach, it falls short in instances where individuals cannot effectively express emotions through their faces. To overcome these limitations, there is a need to explore alternative methods that can provide a more accurate assessment of emotions. This research paper aims to investigate the collaboration and interaction between facial expressions and EEG data for emotion recognition. By combining the information from both modalities, it is expected to augment the accuracy and strength of emotion recognition systems. The proposed method can range from conducting literature reviews to designing and fine-tuning deep learning models for feature extraction, developing fusion models to combine features from facial expressions and EEG data, performing experimentation and evaluation, writing papers and documentation, preparing presentations for dissemination, and engaging in regular meetings and discussions for effective collaboration. Ethical considerations, robustness and generalizability, continual learning and skill development, and utilizing collaboration tools and platforms are also essential contributions to ensure the project’s success.

A novel methodology for emotion recognition through 62-lead EEG signals: multilevel heterogeneous recurrence analysis.

Recognizing emotions from electroencephalography (EEG) signals is a challenging task due to the complex, nonlinear, and nonstationary characteristics of brain activity. Traditional methods often fail to capture these subtle dynamics, while deep learning approaches lack explainability. In this research, we introduce a novel three-phase methodology integrating manifold embedding, multilevel heterogeneous recurrence analysis (MHRA), and ensemble learning to address these limitations in EEG-based emotion recognition. The proposed methodology was evaluated using the SJTU-SEED IV database. We first applied uniform manifold approximation and projection (UMAP) for manifold embedding of the 62-lead EEG signals into a lower-dimensional space. We then developed MHRA to characterize the complex recurrence dynamics of brain activity across multiple transition levels. Finally, we employed tree-based ensemble learning methods to classify four emotions (neutral, sad, fear, happy) based on the extracted MHRA features. Our approach achieved high performance, with an accuracy of 0.7885 and an AUC of 0.7552, outperforming existing methods on the same dataset. Additionally, our methodology provided the most consistent recognition performance across different emotions. Sensitivity analysis revealed specific MHRA metrics that were strongly associated with each emotion, offering valuable insights into the underlying neural dynamics. This study presents a novel framework for EEG-based emotion recognition that effectively captures the complex nonlinear and nonstationary dynamics of brain activity while maintaining explainability. The proposed methodology offers significant potential for advancing our understanding of emotional processing and developing more reliable emotion recognition systems with broad applications in healthcare and beyond.

Adaptive Spatial–Temporal Aware Graph Learning for EEG-Based Emotion Recognition

An intelligent emotion recognition system based on electroencephalography (EEG) signals shows considerable potential in various domains such as healthcare, entertainment, and education, thanks to its portability, high temporal resolution, and real-time capabilities. However, the existing research in this field faces limitations stemming from the nonstationary nature and individual variability of EEG signals. In this study, we present a novel EEG emotion recognition model, named GraphEmotionNet, designed to enhance the accuracy of EEG-based emotion recognition through the incorporation of a spatiotemporal attention mechanism and transfer learning. The proposed GraphEmotionNet model can effectively learn the intrinsic connections between EEG channels and construct an adaptive graph. This graph’s adaptive nature is crucial in optimizing spatial–temporal graph convolutions, which in turn enhances spatial–temporal feature characterization and contributes to the process of emotion classification. Moreover, an integration of domain adaptation aligns the extracted features across different domains, further alleviating the impact of individual EEG variability. We evaluate the model performance on two benchmark databases, employing two types of cross-validation protocols: within-subject cross-validation and cross-subject cross-validation. The experimental results affirm the model’s efficacy in extracting EEG features linked to emotional semantics and demonstrate its promising performance in emotion recognition.

Beyond superficial emotion recognition: Modality-adaptive emotion recognition system

Beyond superficial emotion recognition: Modality-adaptive emotion recognition system

IndEmoVis: A Multimodal Repository for In-Depth Emotion Analysis in Conversational Contexts

Emotion recognition holds significant importance in human communication, benefiting various domains like human-computer interaction, affective computing, and social robotics. Recent interest lies in exploiting multimodal data, encompassing audio, visual, and other cues, to enhance emotion recognition systems. However, most available datasets predominantly focus on Western cultures, overlooking the diverse emotional expressions in regions like India. Moreover, existing datasets often neglect complex emotions like sympathy and awe. To address these limitations, we introduce "IndEMoVis," a novel multimodal dataset of Indian emotions. It comprises 122 recorded audio visual responses during conversations between pairs of individuals. The dataset includes 61 participants, consisting of 25 females and 36 males aged 18 to 21, primarily from Maharashtra and Gujarat states in India. It encompasses nine emotions: Neutral, Happiness, Sadness, Surprise, Disgust, Anger, Fear, Awe, and Sympathy. The annotation process involves a three-step procedure, ensuring accurate emotion labeling. Additionally, annotations are provided for intensity and confidence levels. IndEMoVis dataset aims to support the research community in affective computing by improving conversation abilities, analyzing emotional intelligence, and evaluating responses in debates. Its cultural relevance and inclusion of complex emotions offer valuable insights into emotion recognition for diverse contexts.

Affective computing methods for simulation of action scenarios in video games

Video games are becoming a part of the lives of many, many people. They are able to evoke a wide range of emotions, from joy and excitement to fear and empathy, creating deep immersion and memorable impressions for players. How do they manage it and is it possible to use the emotions themselves in the game itself?This article presents the concept of developing a system of dynamic procedural content generation (PCG) with game mechanics of interaction using emotions based on methods affective computing. To achieve this goal, The article developed an emotion recognition system based on fuzzy logic. And PCG performed by Artificial Neural Networks.In the video game industry, affective computing methods, such as interaction with the expression of emotions, can bring great benefits, since the emotional involvement of the player is very important. This will make a new contribution to the diversity of human-computer interaction.

Multidimensional Scale Analysis and Modeling of Emotional Space in Yao Music

Abstract The research content of this paper gives a detailed analysis of the system structure of the music emotion recognition system. In the process of music feature extraction and recognition, the system first reads the note queue and tempo information queue of the format music and then analyzes and calculates the basic features of the music, tonal and melodic features, etc., respectively. The content of the text set is projected into a three-dimensional visual space using the multidimensional scale analysis method, and then the emotional theme is obtained by dividing the class groups. The samples from the Yao music database are selected as the data source for music emotion research, and performance analysis is performed. The results prove that the method of this paper can effectively perform feature extraction on Yao music files, with an average misdetection rate of only 0.09% and a recognition rate of more than 83% in all cases. The musical structure of Yao Village Impression can be broken down into five parts. The musical melodies mostly consist of major second and minor third intervals. The emotional labeling of structures 1 and 4 in the cluster analysis is distributed in four quadrants, with a preference for pleasurable emotions. Structure 2 was more concentrated in quadrants 1, 4. These emotion distribution results are consistent with the music depicting the beauty of Yao village and the emotional life of Yao people.

A subject-independent portable emotion recognition system using synchrosqueezing wavelet transform maps of EEG signals and ResNet-18

ObjectiveDesigning a portable Brain-Computer Interface (aBCI) using EEG signals is challenging due to the numerous channels, though not all are vital for emotional recognition. We aimed to simplify this by creating a two-channel portable aBCI using advanced time-frequency analysis and deep learning. MethodOur approach involved utilizing the time-frequency analysis named synchrosqueezing wavelet transform (SSWT), which provides better frequency resolution for fluctuations of EEG signal than common wavelet transform. Using the ResNet-18 Convolutional Neural Network, we fine-tuned for sadness and happiness classification. The two best channels were identified across four databases: SEED-IV, SEED-V, SEED-GER, and SEED-FRA, using the Leave-One-Subject-Out method. ResultsFinally, we achieved an average accuracy over sad and happy emotions using the SSWT-ResNet18 model of 76.66%, 78.12%, 81.25%, and 75.00% for the SEED-IV, SEED-V, SEED-GER, and SEED-FRA databases, respectively. ConclusionOverall, our study demonstrates the potential for developing a rapid aBCI by utilizing a precise time–frequency method and deep learning technique from the least number of channels. SignificanceOur approach has promising implications for future real-world applications in emotional recognition.

ENHANCING HUMAN-ROBOT INTERACTION THROUGH GROUP EMOTION RECOGNITION

Abstract - This article explores within the field of Human-Robot Interaction (HRI), focusing on the complicated relationship between emotions, decision-making, and robot behaviors. Emotions are essential to effective communication and interaction, requiring the development of emotion recognition systems in robots. The article explores both individual and group emotion recognition, including microexpressions and macroexpressions. Group emotion dynamics, encompassing phenomena like emotional contagion, convergence, and social influence, are separated to understand how emotions combine within collective settings. A concept, Group Emotion Recognition (GER), is introduced, providing a framework for recognizing emotions within groups. GER involves proximity metrics, emotion classification, and entropy-based analysis to quantify emotion diversity. The article also outlines how GER can enhance user engagement, personalize interactions, improve group dynamics, and foster social acceptance in various human-robot interaction scenarios. Decision-making based on GER, driven by positive or negative emotion labels, is discussed, highlighting the adaptability and sensitivity required for effective human-robot interactions. Ethical considerations regarding the use of emotion recognition technology are addressed throughout the article, emphasizing responsible implementation. Overall, this work lays a solid foundation for advancing the field of HRI by integrating emotion recognition and decision-making to create emotionally intelligent and socially aware robots.

Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions

In recent years, affective computing has become a topic of considerable interest, driven by its ability to enhance several domains, such as mental health monitoring, human–computer interaction, and personalized advertising. The progress of affective computing has been extensively supported by the emergence of sub-domains such as sentiment analysis and emotion recognition. Furthermore, Deep Learning (DL) techniques have made significant advancements in the realm of emotion recognition, resulting in the emergence of Multimodal Emotion Recognition (MER) systems that are capable of effectively processing data from various sources, such as audio, video, and text. However, despite the considerable progress made, there are still several challenges that persist in MER systems. Moreover, existing surveys often lack a specific focus on MER and the associated DL architectures. To address these research gaps, this study provides an in-depth systematic review of DL-based MER systems. This review encompasses the recent state-of-the-art models, foundational theories, DL architectures, mechanisms for fusing multimodal information, relevant datasets, performance evaluation, and practical applications. Additionally, the study identifies key challenges and limitations in MER systems and suggests future research opportunities. The main objective of this review is to provide a thorough comprehension of the present cutting-edge MER, thus enabling researchers in both academia and industry to stay up to date with the most recent developments in this rapidly evolving domain.

NA

NA

NA

NA

Emotion recognition with multi-modal peripheral physiological signals

IntroductionHealthcare wearables allow researchers to develop various system approaches that recognize and understand the human emotional experience. Previous research has indicated that machine learning classifiers, such as Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and Decision Tree (DT), can improve the accuracy of physiological signal analysis and emotion recognition. However, various emotions can have distinct effects on physiological signal alterations. Therefore, solely relying on a single type of physiological signal analysis is insufficient for accurately recognizing and understanding human emotional experiences.MethodsResearch on multi-modal emotion recognition systems (ERS) has commonly gathered physiological signals using expensive devices, which required participants to remain in fixed positions in the lab setting. This limitation restricts the potential for generalizing the ERS technology for peripheral use in daily life. Therefore, considering the convenience of data collection from everyday devices, we propose a multi-modal physiological signals-based ERS based on peripheral signals, utilizing the DEAP database. The physiological signals selected for analysis include photoplethysmography (PPG), galvanic skin response (GSR), and skin temperature (SKT). Signal features were extracted using the “Toolbox for Emotional Feature Extraction from Physiological Signals” (TEAP) library and further analyzed with three classifiers: SVM, KNN, and DT.ResultsThe results showed improved accuracy in the proposed system compared to a single-modal ERS application, which also outperformed current DEAP multi-modal ERS applications.DiscussionThis study sheds light on the potential of combining multi-modal peripheral physiological signals in ERS for ubiquitous applications in daily life, conveniently captured using smart devices.