Signal-based Emotion Recognition Research Articles

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations. Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram, electrooculogram, electromyography, and galvanic skin response. Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition. Main results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification. Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

Graph-Enhanced Emotion Neural Decoding.

Brain signal-based emotion recognition has recently attracted considerable attention since it has powerful potential to be applied in human-computer interaction. To realize the emotional interaction of intelligent systems with humans, researchers have made efforts to decode human emotions from brain imaging data. The majority of current efforts use emotion similarities (e.g., emotion graphs) or brain region similarities (e.g., brain networks) to learn emotion and brain representations. However, the relationships between emotions and brain regions are not explicitly incorporated into the representation learning process. As a result, the learned representations may not be informative enough to benefit specific tasks, e.g., emotion decoding. In this work, we propose a novel idea of graph-enhanced emotion neural decoding, which takes advantage of a bipartite graph structure to integrate the relationships between emotions and brain regions into the neural decoding process, thus helping learn better representations. Theoretical analyses conclude that the suggested emotion-brain bipartite graph inherits and generalizes the conventional emotion graphs and brain networks. Comprehensive experiments on visually evoked emotion datasets demonstrate the effectiveness and superiority of our approach.

Happy or sad? Recognizing emotions with wavelet coefficient energy mean of EEG signals.

Emotional intelligence plays a vital role in human-computer interaction, and EEG signals are an objective response to human emotions. We propose a method to extract the energy means of detail coefficients as feature values for emotion recognition helps to improve EEG signal-based emotion recognition accuracy. We used movie clips as the eliciting material to stimulate the real emotions of the subjects, preprocessed the collected EEG signals, extracted the feature values, and classified the emotions based on them using Support Vector Machine (SVM) and Stacked Auto-Encoder (SAE). The method was verified based on the SJTU emotion EEG database (SEED) and the self-acquisition experiment. The results show that the accuracy is better using SVM. The results based on the SEED database are 89.06% and 79.90% for positive-negative and positive-neutral-negative, respectively. The results based on the self-acquisition data are 98.05% and 89.83% for the same, with an average recognition rate of 86.57% for the four categories of fear, sad (negative), peace (neutral) and happy (positive). The results demonstrate the validity of the feature values and provide a theoretical basis for implementing human-computer interaction.

A Multimodal Framework for Large-Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Considerable attention has been paid to physiological signal-based emotion recognition in the field of affective computing. For reliability and user-friendly acquisition, electrodermal activity (EDA) has a great advantage in practical applications. However, EDA-based emotion recognition with large-scale subjects is still a tough problem. The traditional well-designed classifiers with hand-crafted features produce poorer results because of their limited representation abilities. And the deep learning models with auto feature extraction suffer the overfitting drop-off because of large-scale individual differences. Since music has a strong correlation with human emotion, static music can be involved as the external benchmark to constrain various dynamic EDA signals. In this article, we make an attempt by fusing the subject’s individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the one-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the channel-temporal attention mechanism for EDA-based emotion recognition is first involved in mine the temporal and channel-wise dynamic and steady features. The comparisons with single EDA-based SOTA models on DEAP and AMIGOS datasets prove the effectiveness of RTCAN-1D to mine EDA features. For music features, we simply process the music signal with the open-source toolkit openSMILE to obtain external feature vectors. We conducted systematic and extensive evaluations. The experiments on the current largest music emotion dataset PMEmo validate that the fusion of EDA and music is a reliable and efficient solution for large-scale emotion recognition.

Emotion recognition based on fusion of long short-term memory networks and SVMs

This paper proposes a multimodal fusion emotion recognition method based on Dempster-Shafer evidence theory, which includes electroencephalogram (EEG) and electrocardiogram (ECG). For EEG, we use the SVM classifier to classify features, and for ECG, we establish the corresponding Bi-directional Long Short-Term Memory network emotion recognition structure, which is fused with EEG classification results through the evidence theory. We selected 25 video clips with five emotions (happy, relaxed, angry, sad, and disgusted), and a total of 20 subjects participated in our emotional experiment. The experimental results prove that the performance of the multi-modal fusion model proposed in this paper is superior to the single-modal emotion recognition model. In the Arousal and Valance dimensions, the average accuracy is improved by 2.64% and 2.75% compared with the EEG signal-based emotion recognition model. Compared with the emotion recognition model based on the ECG signal, the accuracy is improved by 7.37% and 8.73%.

Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

In order to solve the problem that the emotion recognition rate of single-mode physiological signals is not high in the physiological signals based emotion recognition, in this paper, we propose a convolutional recurrent neural network based method for multi-modal physiological signal emotion recognition task. The method used convolutional neural network to learn the spatial representations of multi-channel EEG signals and the Long Short-term Memory network to learn the temporal representations of peripheral physiological signals (EOG, EMG, GSR, RSP, BVP, and TMP). The two representations are combined for emotion recognition and classification. In the two emotion dimensions of Arousal and Valence, our experiments conducted on the open source dataset DEAP shows that, this method achieve 89.68% and 89.19% average accuracy in the EEG emotion classification, 63.06% and 62.41% average accuracy in the peripheral physiological signal emotion classification, 93.06% and 91.95% average accuracy in the combined feature emotion classification. The experimental results show that the convolutional recurrent neural network based method that we proposed efficiently extract multi-modal physiological signal feature to improve the emotion recognition performance.

Selection of Optimal EEG Electrodes for Human Emotion Recognition

In recent years, emotion recognition has attracted increasing interest from researchers from diverse fields. Because of their intrinsic correlation with emotions, physiological signals based emotion recognition method is not susceptible to the so-called social masking and thus more objective than traditional visual, audio or text data based methods. In particular, EEG signals are more responsive to emotion fluctuations than other peripheral physiological signals. In this paper, a 4-class EEG-based emotion classification problem is considered. Firstly the subjective data clustering is performed to identify the optimal number of emotional states. Then wavelet and nonlinear dynamics analyses are used to extract EEG features of emotions. Finally, we consider the brain areas for emotion generation and show that the use of only a small number of EEG electrodes placed on the frontal area of scalp can achieve a 4-class emotion classification accuracy of higher than 90%.

Interpretable Emotion Recognition Using EEG Signals

Electroencephalogram (EEG) signal-based emotion recognition has attracted wide interests in recent years and has been broadly adopted in medical, affective computing, and other relevant fields. However, the majority of the research reported in this field tends to focus on the accuracy of classification whilst neglecting the interpretability of emotion progression. In this paper, we propose a new interpretable emotion recognition approach with the activation mechanism by using machine learning and EEG signals. This paper innovatively proposes the emotional activation curve to demonstrate the activation process of emotions. The algorithm first extracts features from EEG signals and classifies emotions using machine learning techniques, in which different parts of a trial are used to train the proposed model and assess its impact on emotion recognition results. Second, novel activation curves of emotions are constructed based on the classification results, and two emotion coefficients, i.e., the correlation coefficients and entropy coefficients. The activation curve can not only classify emotions but also reveals to a certain extent the emotional activation mechanism. Finally, a weight coefficient is obtained from the two coefficients to improve the accuracy of emotion recognition. To validate the proposed method, experiments have been carried out on the DEAP and SEED dataset. The results support the point that emotions are progressively activated throughout the experiment, and the weighting coefficients based on the correlation coefficient and the entropy coefficient can effectively improve the EEG-based emotion recognition accuracy.

A Review of Emotion Recognition Using Physiological Signals.

Emotion recognition based on physiological signals has been a hot topic and applied in many areas such as safe driving, health care and social security. In this paper, we present a comprehensive review on physiological signal-based emotion recognition, including emotion models, emotion elicitation methods, the published emotional physiological datasets, features, classifiers, and the whole framework for emotion recognition based on the physiological signals. A summary and comparation among the recent studies has been conducted, which reveals the current existing problems and the future work has been discussed.

IAware: A Real-Time Emotional Biofeedback System Based on Physiological Signals

Self-awareness is the foundation of emotional intelligence. Most people can recognize their own and others' emotions. However, many people suffer from a common infirmity that prevents them from recognizing emotion within themselves and are therefore unable to experience a life that fulfills them emotionally. We propose a real-time mobile biofeedback system that uses wearable sensors to depict five basic emotions and provides the user with emotional feedback. We also present empirical results for the configuration of a physiological signal-based emotion recognition system in two experimental scenarios involving controlled and noncontrolled environmental settings. In our evaluation, we show that iAware helps increase emotional self-awareness by reducing the predictive error by 3.333% for women and by 16.673% for men. The primary results suggest the usefulness and necessity of the iAware system to provide users with real-time biofeedback based on physiological signals.

Dynamical recursive feature elimination technique for neurophysiological signal-based emotion recognition

The machine learning-based classification model can predict the operator emotional states in human–machine system based on nonlinear, multidimensional neurophysiological features. However, the dynamical properties of the testing physiological data regarding the time series may influence the feature distribution variation and inter-class discrimination across different time steps. To overcome this shortcoming, we propose a novel EEG feature selection method, dynamical recursive feature elimination (D-RFE), to find the optimal but different feature rankings at each time instant for arousal and valence recognition. With the classification framework implemented via a model-selected least square support vector machine, the participant-specific classification performance has been significantly improved against conventional RFE model and several common classifiers. The optimal classification accuracy and F1-score elicited by the proposed method are 0.7896, 0.7991, 0.7143, and 0.7257 for arousal and valence dimensions, respectively, which are quite competitive among recent reported works on the same EEG database.

A Review on EEG Signals Based Emotion Recognition

Emotion recognition has become a very controversial issue in brain-computer interfaces (BCIs). Moreover, numerous studies have been conducted in order to recognize emotions. Also, there are several important definitions and theories about human emotions. In this paper we try to cover important topics related to the field of emotion recognition. We review several studies which are based on analyzing electroencephalogram (EEG) signals as a biological marker in emotion changes. Considering low cost, good time and spatial resolution, EEG has become very common and is widely used in most BCI applications and studies. First, we state some theories and basic definitions related to emotions. Then some important steps of an emotion recognition system like different kinds of biologic measurements (EEG, electrocardiogram [EEG], respiration rate, etc), offline vs online recognition methods, emotion stimulation types and common emotion models are described. Finally, the recent and most important studies are reviewed.

A Study of Physiological Signals-based Emotion Recognition Systems

The use of physiological signals is relatively recent development in human emotion recognition. Interest in this field has been motivated by the unbiased nature of such signals, which are generated autonomously from the central nervous system. Generally, these signals can be collected from the cardiovascular system, respiratory system, electrodermal activities, muscular system and brain activities. This paper presents an overview of emotion recognition using physiological signals. The main components of a physiological signals-based emotion recognirion system are explained, including discussion regarding the concepts and problems about the various stages involved in its framework.

Speech signal-based emotion recognition and its application to entertainment robots

By recognizing sensory information, through touch, vision, or voice sensory modalities, a robot can interact with people in a more intelligent manner. In human–robot interaction (HRI), emotion recognition has been a popular research topic in recent years. This paper proposes a method for emotion recognition, using a speech signal to recognize several basic human emotional states, for application in an entertainment robot. The proposed method uses voice signal processing and classification. Firstly, end-point detection and frame setting are accomplished in the pre-processing stage. Then, the statistical features of the energy contour are computed. Fisher’s linear discriminant analysis (FLDA) is used to enhance the recognition rate. In the final stage, a support vector machine (SVM) is used to complete the emotional state classification. In order to determine the effectiveness of emotional HRI, an embedded system was constructed and integrated with a self-built entertainment robot. The experimental results for the entertainment robot show that the robot interacts with a person in a responsive manner. The average recognition rate for five emotional states is 73.8% using the database constructed in the authors’ lab.

Research on Affective Recognition with Physiological Signals Based on IAGA and wkNN Classification

The ability to understand human emotions is desirable for the computer in many applications recently. Recording and recognizing physiological signals of emotion has become an increasingly important field of research in affective computing and human computer interaction. For the problem of feature redundancy of physiological signals-based emotion recognition and low efficiency of traditional feature reduction algorithms on great sample data, this paper proposed an improved adaptive genetic algorithm (IAGA) to solve the problem of emotion feature selection, and then presented a weighted kNN classifier (wkNN) to classify features by making full use of emotion sample information. We demonstrated a case study of emotion recognition application and verified the algorithm's validity by the analysis of experimental simulation data and the comparison of several recognition methods.

Emotion recognition system using short-term monitoring of physiological signals.

A physiological signal-based emotion recognition system is reported. The system was developed to operate as a user-independent system, based on physiological signal databases obtained from multiple subjects. The input signals were electrocardiogram, skin temperature variation and electrodermal activity, all of which were acquired without much discomfort from the body surface, and can reflect the influence of emotion on the autonomic nervous system. The system consisted of preprocessing, feature extraction and pattern classification stages. Preprocessing and feature extraction methods were devised so that emotion-specific characteristics could be extracted from short-segment signals. Although the features were carefully extracted, their distribution formed a classification problem, with large overlap among clusters and large variance within clusters. A support vector machine was adopted as a pattern classifier to resolve this difficulty. Correct-classification ratios for 50 subjects were 78.4% and 61.8%, for the recognition of three and four categories, respectively.