Emotion Recognition Method Research Articles

Data augmentation and enhancement for multimodal speech emotion recognition

Humans’ fundamental need is interaction with each other such as using conversation or speech. Therefore, it is crucial to analyze speech using computer technology to determine emotions. The speech emotion recognition (SER) method detects emotions in speech by examining various aspects. SER is a supervised method to decide the emotion class in speech. This research proposed a multimodal SER model using one of the deep learning based enhancement techniques, which is the attention mechanism. Additionally, this research addresses the imbalanced dataset problem in the SER field using generative adversarial networks (GAN) as a data augmentation technique. The proposed model achieved an excellent evaluation performance of 0.96 or 96% for the proposed GAN configuration. This work showed that the GAN method in the multimodal SER model could enhance performance and create a balanced dataset.

ICaps-ResLSTM: Improved capsule network and residual LSTM for EEG emotion recognition

Electroencephalography (EEG) emotion recognition is an important task for brain–computer interfaces. The time, frequency, and spatial domains of EEG signals have been widely studied. However, these methods often ignore the spatial and temporal correlations in dual modules, resulting in insufficient emotional representations. In this paper, a dual module EEG emotion recognition method based on an improved capsule network and residual Long-Short Term Memory (ResLSTM) is proposed. Using an improved capsule network as the spatial module is more advantageous in learning specific EEG spatial representations. The ResLSTM of the temporal module inherits the information flow from the upper spatial module and conducts complementary learning of the spatiotemporal dual module features through residual connections, thus obtaining more discriminative EEG features and ultimately boosting the classification capabilities of the model. The average accuracy of arousal, valence, and dominance on the DEAP dataset reached 98.06%, 97.94%, and 98.15%, respectively. The DREAMER dataset’s average accuracy of arousal, valence, and dominance reached 94.97%, 94.71%, and 94.96%, respectively. The results of our experiments indicate that our method outperforms state-of-the-art approaches.

Research on Emotion Recognition Method of Flight Training Based on Multimodal Fusion

The emotional activities of the human body are mainly regulated by the autonomic nervous system, the central nervous system, and the advanced cognition of the human brain. This paper proposes an emotional state recognition method in pilot training tasks based on multimodal information fusion. A set of emotion perception recognition systems, a two-dimensional valence-arousal emotional model, and a multimodal information intelligent perception model were established based on the human-computer interaction mode during flight training; an intelligent perception system was designed to collect and intelligently perceive four kinds of peripheral physiological signals of pilots in real-time. And based on traditional machine learning models, a binary tree support vector machine was designed to optimize and improve the multimodal information co-integration decision model, which increased the accuracy of emotional state recognition in flight training by 37.58% on average. The experimental result showed that it realizes accurate monitoring and identification of real-time emotional state, helps to improve the effect of flight training and flight safety, maintains the efficiency of operation, and has important research significance and application prospects in the field of pilot training.

Machine learning methods for speech emotion recognition on telecommunication systems

Machine learning methods for speech emotion recognition on telecommunication systems

Construction of an Emotional Lexicon of Patients With Breast Cancer: Development and Sentiment Analysis.

The innovative method of sentiment analysis based on an emotional lexicon shows prominent advantages in capturing emotional information, such as individual attitudes, experiences, and needs, which provides a new perspective and method for emotion recognition and management for patients with breast cancer (BC). However, at present, sentiment analysis in the field of BC is limited, and there is no emotional lexicon for this field. Therefore, it is necessary to construct an emotional lexicon that conforms to the characteristics of patients with BC so as to provide a new tool for accurate identification and analysis of the patients' emotions and a new method for their personalized emotion management. This study aimed to construct an emotional lexicon of patients with BC. Emotional words were obtained by merging the words in 2 general sentiment lexicons, the Chinese Linguistic Inquiry and Word Count (C-LIWC) and HowNet, and the words in text corpora acquired from patients with BC via Weibo, semistructured interviews, and expressive writing. The lexicon was constructed using manual annotation and classification under the guidance of Russell's valence-arousal space. Ekman's basic emotional categories, Lazarus' cognitive appraisal theory of emotion, and a qualitative text analysis based on the text corpora of patients with BC were combined to determine the fine-grained emotional categories of the lexicon we constructed. Precision, recall, and the F1-score were used to evaluate the lexicon's performance. The text corpora collected from patients in different stages of BC included 150 written materials, 17 interviews, and 6689 original posts and comments from Weibo, with a total of 1,923,593 Chinese characters. The emotional lexicon of patients with BC contained 9357 words and covered 8 fine-grained emotional categories: joy, anger, sadness, fear, disgust, surprise, somatic symptoms, and BC terminology. Experimental results showed that precision, recall, and the F1-score of positive emotional words were 98.42%, 99.73%, and 99.07%, respectively, and those of negative emotional words were 99.73%, 98.38%, and 99.05%, respectively, which all significantly outperformed the C-LIWC and HowNet. The emotional lexicon with fine-grained emotional categories conforms to the characteristics of patients with BC. Its performance related to identifying and classifying domain-specific emotional words in BC is better compared to the C-LIWC and HowNet. This lexicon not only provides a new tool for sentiment analysis in the field of BC but also provides a new perspective for recognizing the specific emotional state and needs of patients with BC and formulating tailored emotional management plans.

A deep interpretable representation learning method for speech emotion recognition

This paper focuses on the active interpretability for deep learning-based speech emotion recognition (SER). To achieve this, we propose an explicit feature constrained model, the interpretable group convolutional neural network (IG-CNN) model. In the proposed model, we first introduce the interpretability constraint to learn human-understandable interpretable representations. The emotion prediction decision can be active interpreted via the model coefficients. To acquire more representations beyond interpretable ones, and ensure they are useful for SER, we then design the uncorrelation constraint between interpretable and autonomous representations and introduce group CNN structure. We test the model on IEMOCAP, RAVDESS, eNTERFACE’05, and CREMA-D datasets. Experimental results show that our model outperforms all the baselines. In addition, the proposed model can also learn the patterns of human perception of speech emotion and provide explanation for the recognition results.

Semi-supervised cross-lingual speech emotion recognition

Performance in Speech Emotion Recognition (SER) on a single language has increased greatly in the last few years thanks to the use of deep learning techniques. However, cross-lingual SER remains a challenge in real-world applications due to two main factors: the first is the big gap among the source and the target domain distributions; the second factor is the major availability of unlabeled utterances in contrast to the labeled ones for the new language. Taking into account previous aspects, we propose a Semi-Supervised Learning (SSL) method for cross-lingual emotion recognition when only few labeled examples in the target domain (i.e. the new language) are available. Our method is based on a Transformer and it adapts to the new domain by exploiting a pseudo-labeling strategy on the unlabeled utterances. In particular, the use of a hard and soft pseudo-labels approach is investigated. We thoroughly evaluate the performance of the proposed method in a speaker-independent setup on both the source and the new language and show its robustness across five languages belonging to different linguistic strains. The experimental findings indicate that the unweighted accuracy is increased by an average of 40% compared to state-of-the-art methods.

Brain-Machine Coupled Learning Method for Facial Emotion Recognition.

Neural network models of machine learning have shown promising prospects for visual tasks, such as facial emotion recognition (FER). However, the generalization of the model trained from a dataset with a few samples is limited. Unlike the machine, the human brain can effectively realize the required information from a few samples to complete the visual tasks. To learn the generalization ability of the brain, in this paper, we propose a novel brain-machine coupled learning method for facial emotion recognition to let the neural network learn the visual knowledge of the machine and cognitive knowledge of the brain simultaneously. The proposed method utilizes visual images and electroencephalogram (EEG) signals to couple training the models in the visual and cognitive domains. Each domain model consists of two types of interactive channels, common and private. Since the EEG signals can reflect brain activity, the cognitive process of the brain is decoded by a model following reverse engineering. Decoding the EEG signals induced by the facial emotion images, the common channel in the visual domain can approach the cognitive process in the cognitive domain. Moreover, the knowledge specific to each domain is found in each private channel using an adversarial strategy. After learning, without the participation of the EEG signals, only the concatenation of both channels in the visual domain is used to classify facial emotion images based on the visual knowledge of the machine and the cognitive knowledge learned from the brain. Experiments demonstrate that the proposed method can produce excellent performance on several public datasets. Further experiments show that the proposed method trained from the EEG signals has good generalization ability on new datasets and can be applied to other network models, illustrating the potential for practical applications.

Comprehensive review and analysis on facial emotion recognition methods

Comprehensive review and analysis on facial emotion recognition methods

Heterogeneous Convolutional Neural Networks for Emotion Recognition Combined with Multimodal Factorised Bilinear Pooling and Mobile Application Recommendation

The field of emotion recognition has garnered considerable interest due to its diverse applications in mental health, personalised advertising and enhancing user experiences. This research paper introduces a unique and innovative method for emotion recognition by integrating heterogeneous convolutional neural networks (CNNs) with multimodal factorised bilinear pooling. Furthermore, the paper also incorporates the integration of mobile application recommendations as part of the overall approach. The proposed method leverages the power of CNNs to extract high-level features from different modalities, including facial expressions, speech signals and physiological signals. By using heterogeneous CNNs, each modality is processed independently to capture modality-specific emotional cues effectively. To fuse the extracted features, multimodal factorised bilinear pooling is employed, which captures the complex interactions between different modalities while reducing the computational complexity. This pooling technique efficiently combines the modality-specific features, resulting in a compact and discriminative representation of the emotional state. In addition to emotion recognition, this paper also introduces the integration of mobile app recommendations. By leveraging the recognised emotion, the system recommends relevant mobile applications that are tailored to the user’s emotional state. This integration enhances user experience and facilitates emotion regulation through the utilisation of appropriate mobile apps. Experimental evaluations are conducted on benchmark emotion recognition datasets, including the DEAP and MAHNOB_HCI datasets. The findings of the study highlight the effectiveness of the proposed methodology in terms of accuracy and robustness, surpassing existing approaches in the field. Additionally, the integration of the mobile app recommendation system showcases encouraging outcomes by offering personalised recommendations tailored to the user’s emotional state.

A multimodal fusion emotion recognition method based on multitask learning and attention mechanism

With new developments in the field of human–computer interaction, researchers are now paying attention to emotion recognition, especially multimodal emotion recognition, as emotion is a multidimensional expression. In this study, we propose a multimodal fusion emotion recognition method (MTL-BAM) based on multitask learning and the attention mechanism to tackle the major problems encountered in multimodal emotion recognition tasks regarding the lack of consideration of emotion interactions among modalities and the focus on emotion similarity among modalities while ignoring the differences. By improving the attention mechanism, the emotional contribution of each modality is further analyzed so that the emotional representations of each modality can learn from and complement each other to achieve better interactive fusion effect, thereby building a multitask learning framework. By introducing three types of monomodal emotion recognition tasks as auxiliary tasks, the model can detect emotion differences. Simultaneously, the label generation unit is introduced into the auxiliary tasks, and the monomodal emotion label value can be obtained more accurately through two proportional formulas while preventing the zero value problem. Our results show that the proposed method outperforms selected state-of-the-art methods on four evaluation indexes of emotion classification (i.e., accuracy, F1 score, MAE, and Pearson correlation coefficient). The proposed method achieved accuracy rates of 85.36% and 84.61% on the published multimodal datasets of CMU-MOSI and CMU-MOSEI, respectively, which are 2–6% higher than those of existing state-of-the-art models, demonstrating good multimodal emotion recognition performance and strong generalizability.

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals.

Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.

Multi-channel EEG emotion recognition through residual graph attention neural network.

In this paper, a novel EEG emotion recognition method based on residual graph attention neural network is proposed. The method constructs a three-dimensional sparse feature matrix according to the relative position of electrode channels, and inputs it into the residual network to extract high-level abstract features containing electrode spatial position information. At the same time, the adjacency matrix representing the connection relationship of electrode channels is constructed, and the time-domain features of multi-channel EEG are modeled using graph. Then, the graph attention neural network is utilized to learn the intrinsic connection relationship between EEG channels located in different brain regions from the adjacency matrix and the constructed graph structure data. Finally, the high-level abstract features extracted from the two networks are fused to judge the emotional state. The experiment is carried out on DEAP data set. The experimental results show that the spatial domain information of electrode channels and the intrinsic connection relationship between different channels contain salient information related to emotional state, and the proposed model can effectively fuse these information to improve the performance of multi-channel EEG emotion recognition.

Simplicial Homology Global Optimization of EEG Signal Extraction for Emotion Recognition

Emotion recognition is a vital part of human functioning. textcolorredIt enables individuals to respond suitably to environmental events and develop self-awareness. The fast-paced developments in brain–computer interfacing (BCI) technology necessitate that intelligent machines of the future be able to digitize and recognize human emotions. To achieve this, both humans and machines have relied on facial expressions, in addition to other visual cues. While facial expressions are effective in recognizing emotions, they can be artificially replicated and require constant monitoring. In recent years, the use of Electroencephalography (EEG) signals has become a popular method for emotion recognition, thanks to advances in deep learning and machine learning techniques. EEG-based systems for recognizing emotions involve measuring electrical activity in the brain of a subject who is exposed to emotional stimuli such as images, sounds, or videos. Machine learning algorithms are then used to extract features from the electrical activity data that correspond to specific emotional states. The quality of the extracted EEG signal is crucial, as it affects the overall complexity of the system and the accuracy of the machine learning algorithm. This article presents an approach to improve the accuracy of EEG-based emotion recognition systems while reducing their complexity. The approach involves optimizing the number of EEG channels, their placement on the human scalp, and the target frequency band of the measured signal to maximize the difference between high and low arousal levels. The optimization method, called the simplicial homology global optimization (SHGO), is used for this purpose. Experimental results demonstrate that a six-electrode configuration optimally placed can achieve a better level of accuracy than a 14-electrode configuration, resulting in an over 60% reduction in complexity in terms of the number of electrodes. This method demonstrates promising results in improving the efficiency and accuracy of EEG-based emotion recognition systems, which could have implications for various fields, including healthcare, psychology, and human–computer interfacing.

Determining emotions from students' facial expressions using CNN

Recognizing and understanding human emotions, particularly in educational settings, is of great importance. This research focuses on utilizing Convolutional Neural Networks (CNNs) to accurately identify students' emotions based on their facial expressions. By leveraging facial cues, an automated system can be developed to effectively recognize and interpret emotions in educational contexts. A diverse dataset of facial images featuring students expressing various emotions is carefully curated for this study. Facial landmarks and action units are extracted to capture essential information from different facial regions. These images are meticulously annotated with ground truth labels, ensuring precise training and evaluation of the CNN model. CNNs are chosen as the core technology for feature extraction and emotion classification due to their ability to learn intricate spatial patterns and hierarchical representations. Extensive training, including techniques like data augmentation and transfer learning, enables the model to generalize and adapt to a wide range of emotional expressions. The performance of the CNN model is evaluated using metrics such as accuracy, precision, recall, and F1 score. Thorough experiments compare the proposed CNN approach with existing methods for facial emotion recognition, demonstrating the superior performance of the CNN model in accurately identifying students' emotions from facial expressions.

Adaptive neural decision tree for EEG based emotion recognition

An adaptive neural decision tree is investigated to recognize electroencephalogram (EEG) emotion signal with ability of intelligently selecting network structure. Firstly, to overcome lack of position information of the vectorized input EEG signal, the original one-dimensional EEG vector signal is cast into a two-dimensional matrix signal added with channel position information. Secondly, the depthwise convolution is adopted in transformers to deal with each channel by one convolution kernel, thus decreases the number of parameters. Then, to overcome model interpretability problem, an adaptive neural decision tree (ANT) based emotion recognition method is explored by embedding neural networks into the decision tree to perform feature extracting, path selecting, and label classifying. Further, ANT can automatically search optimized parameters by using the adaptive moment estimation algorithm, and explore tree architectures by using the exploration–exploitation trade-off reinforcement learning method to get the global optimal network structure without manually setting complex hyperparameters. Finally, by 5-fold cross-validation, binary, four-class and eight-class classification experiments are carried out on DEAP datasets, the average accuracy of the ANT algorithm are 99.14 ± 0.456%, 98.95 ± 0.84%, 97.58 ± 2.311%, respectively, which verifies the effectiveness of the proposed method, compared with the traditional decision tree method.

Multimodal transformer augmented fusion for speech emotion recognition.

Speech emotion recognition is challenging due to the subjectivity and ambiguity of emotion. In recent years, multimodal methods for speech emotion recognition have achieved promising results. However, due to the heterogeneity of data from different modalities, effectively integrating different modal information remains a difficulty and breakthrough point of the research. Moreover, in view of the limitations of feature-level fusion and decision-level fusion methods, capturing fine-grained modal interactions has often been neglected in previous studies. We propose a method named multimodal transformer augmented fusion that uses a hybrid fusion strategy, combing feature-level fusion and model-level fusion methods, to perform fine-grained information interaction within and between modalities. A Model-fusion module composed of three Cross-Transformer Encoders is proposed to generate multimodal emotional representation for modal guidance and information fusion. Specifically, the multimodal features obtained by feature-level fusion and text features are used to enhance speech features. Our proposed method outperforms existing state-of-the-art approaches on the IEMOCAP and MELD dataset.

A novel one-dimensional convolutional neural network-based method for emotion recognition of electric power industry workers

When the workers of electric power industry are going to perform high-risk or dispatching operations, it is important to assess the emotional state of the workers. Emotion recognition is helpful to justify whether the works are suitable to grasp and handle the operation status of the power system in time. In this paper, we develop a simple and efficient automatic emotion recognition method based on deep learning theory using ECG signals. Firstly, since Z-score method which is quite popular for emotion recognition exists high missed outlier detection rate, we define a basic beat and removed the abnormal beats according to the similarities between the truncated beats and the basic beat. Secondly, in order to make it easily for Convolutional Neural Network (CNN) to extract the periodicity features of one-dimensional signal in time domain, we present stacking operation of the truncated segments to increase the dimension of ECG data from one to three. Finally, aiming at decreasing the computation burden, a one-dimensional CNN only consisted of two convolution blocks is designed. The results of applying the proposed approach to the benchmark Wearable Stress and Affect Data set (WESAD) demonstrate that our proposal is capable of classifying four different kinds of emotions with an averaged accuracy of 98.57% and F1 value of 98.73%, respectively. Compared against some literature, the performance of our proposal is superior to them.

EEG-Based Multi-Class Emotion Recognition using Hybrid LSTM Approach

Emotion recognition is a crucial task in human-computer interaction, psychology, and neuroscience. Electroencephalogram (EEG)-based multi-class emotion recognition is a novel approach that aims to identify and classify human emotions by analysing EEG signals. Traditional methods of emotion recognition often face challenges in accurately identifying and classifying human emotions due to their complexity and subjectivity. EEG-based emotion recognition provides a direct and objective measure of three emotional states (positive, neutral, and negative), making it a promising tool for emotion recognition. The proposed hybrid LSTM approach combines the strengths of different traditional machine learning algorithms: Gaussian Naive Bayes (GNB), Support Vector Machine (SVM), Logistic Regression (LR), and Decision Tree (DT). The approach was tested on the EEG brainwave dataset, and LSTM achieved an accuracy of 95%, while the proposed hybrid LSTM-GNB, LSTM-SVM, LSTM-LR, and LSTM-DT models achieved 65%, 96%, 97%, and 96% accuracy, respectively. The contribution of this study is the development of a hybrid LSTM approach that combines the strengths of two different algorithms, resulting in higher accuracy for multi-class emotion recognition using EEG signals. The results demonstrate the potential of the hybrid LSTM approach for real-world applications such as emotion-based human-computer interaction and mental health diagnosis.

Detection and analysis of graduate students' academic emotions in the online academic forum based on text mining with a deep learning approach.

The possibility of mental illness caused by the academic emotions and academic pressure of graduate students has received widespread attention. Discovering hidden academic emotions by mining graduate students' speeches in social networks has strong practical significance for the mental state discovery of graduate students. Through data collected from online academic forum, a text based BiGRU-Attention model was conducted to achieve academic emotion recognition and classification, and a keyword statistics and topic analysis was performed for topic discussion among graduate posts. Female graduate students post more than male students, and graduates majoring in chemistry post the most. Using the BiGRU-Attention model to identify and classify academic emotions has a performance with precision, recall and F1 score of more than 95%, the category of PA (Positive Activating) has the best classification performance. Through the analysis of post topics and keywords, the academic emotions of graduates mainly come from academic pressure, interpersonal relationships and career related. A BiGRU-Attention model based on deep learning method is proposed to combine classical academic emotion classification and categories to achieve a text academic emotion recognition method based on user generated content.