Subject-independent Emotion Recognition Research Articles

FBN-TCN: Temporal convolutional neural network based on spatial domain fusion brain networks for affective brain–computer interfaces

Background and objective: Electroencephalogram (EEG) emotion recognition is increasingly popular for visualizing and quantifying brain networks, with potential applications in medical cognition and human–computer interaction. However, single brain space is limited, and multi-channel recognition poses challenges with calculation volume and information redundancy. Therefore, this paper focuses on studying the affective brain–computer interface (aBCI) and critical lead acquisition of the fusion brain network. Method: This paper employs two correlation algorithms to build functional and causal networks. A fusion brain network (FBN) was proposed to capture emotional information in the spatial domain. Additionally, a deep network recognition model (FBN-TCN) is introduced to capture the dynamic changes of spatial domain emotional information over time. A critical channel combination with high emotional correlation among multiple channels is obtained by the brain causal networks. All proposed models were performed on the SEED dataset through subject-dependent and subject-independent experiments. Results: Experimental results demonstrate that FBN can cover more emotional information to improve emotional recognition. The critical channel aBCI achieves comparable recognition performance to the full-channel aBCI, with recognition results of 95.51%/3.15% and 96.34%/1.62% in the Total band, respectively. The generalization and effectiveness of the model were verified through ablation experiments, which involved subject-dependent and subject-independent emotion recognition of DEAP, and emotion recognition of SEED on model block transformation. Conclusion and significance: This paper can simplify the electrode cap device and maximizes the utilization of EEG spatial domain emotional information. It is highly significant for enhancing the performance of aBCI in human–computer interaction.

Subject-Independent Emotion Recognition Based on EEG Frequency Band Features and Self-Adaptive Graph Construction.

Emotion is one of the most important higher cognitive functions of the human brain and plays an important role in transaction processing and decisions. In traditional emotion recognition studies, the frequency band features in EEG signals have been shown to have a high correlation with emotion production. However, traditional emotion recognition methods cannot satisfactorily solve the problem of individual differences in subjects and data heterogeneity in EEG, and subject-independent emotion recognition based on EEG signals has attracted extensive attention from researchers. In this paper, we propose a subject-independent emotion recognition model based on adaptive extraction of layer structure based on frequency bands (BFE-Net), which is adaptive in extracting EEG map features through the multi-graphic layer construction module to obtain a frequency band-based multi-graphic layer emotion representation. To evaluate the performance of the model in subject-independent emotion recognition studies, extensive experiments are conducted on two public datasets including SEED and SEED-IV. The experimental results show that in most experimental settings, our model has a more advanced performance than the existing studies of the same type. In addition, the visualization of brain connectivity patterns reveals that some of the findings are consistent with previous neuroscientific validations, further validating the model in subject-independent emotion recognition studies.

Functional connectivity-enhanced feature-grouped attention network for cross-subject EEG emotion recognition

Electroencephalogram (EEG)-based automatic emotion recognition technologies are gaining significant attention and have become crucial in the field of brain–computer interfaces (BCIs). In particular, deep learning methods have been widely used in emotion recognition in recent years. However, most existing methods generally tend to focus on EEG spatiotemporal information, which ignores the potential relationships between brain activity signals and the differences in functional connectivity under different emotions. Here, we raise a functional connectivity-enhanced feature-grouped attention network (FC-FAN) for cross-subject emotion recognition. The FC-FAN model developed is a dual-input model. One input consists of the differential entropy data derived from the original EEG signals, while the other input comprises the functional connectivity data obtained through the calculation of the phase synchronization index. Then the primary EEG features of the two groups’ input data are extracted through two specific residual blocks. Next, the designed time-series feature grouped attention module (TFGAM) and functional connectivity feature grouped attention model (F2GAM) are utilized to mark interested information or suppress uninterested features for the two groups’ features, respectively. Finally, generated information interacts through a fusion operator. The designed framework could not only sufficiently learn the spatiotemporal features of EEG signals but also clearly analyze nonlinear correlations between electrode signals. Comprehensive tests confirm that the FC-FAN has an excellent effect on subject-independent emotion recognition tasks.

EEG-Based Subject-Independent Emotion Recognition Using Gated Recurrent Unit and Minimum Class Confusion

Automatic emotion recognition based on electroencephalogram (EEG) has attracted rapidly increasing interests. Due to large inter-subject variabilities, subject-independent emotion recognition faces great challenges. Recently, domain adaptation methods have been successfully applied in this field due to their ability to align features from different subjects. However, since EEG signals corresponding to some emotions have similar oscillation patterns, they are often confused and aligned to the wrong categories, which limits the generalization ability of the model across subjects. Besides, almost all methods only support offline applications, which require collecting a large number of samples of new subjects. To achieve online recognition, a simpler model is needed. In this paper, a novel Gated Recurrent Unit-Minimum Class Confusion (GRU-MCC) model is proposed. Specifically, a simple feature extractor based on gated recurrent unit (GRU) is firstly applied to model the spatial dependence of multiple electrodes and obtain high-level discriminative features. Then, during training, minimum class confusion (MCC) loss is introduced to reduce the confusion between the correct and ambiguous classes for the target subject and increase the transfer gains. We conduct both offline and online experiments on two public datasets: SEED and MPED. The results indicate that our method can obtain the superior performance.

Convolutional gated recurrent unit-driven multidimensional dynamic graph neural network for subject-independent emotion recognition

Electroencephalogram (EEG) could directly reflect human brain activities. Recently, EEG-based emotion recognition technology has attracted widespread attention. However, although most of the existing methods can extract temporal and spatial features of EEG signals, the extraction process of the time-series information and spatial features is separate from each other. Some spatial features may be lost when extracting time-series information, and vice versa. Additionally, the non-Euclidean representations between the EEG electrodes cannot be ignored. In response to these issues, we construct a convolutional gated recurrent unit-driven multidimensional dynamic graph neural network (CGRU-MDGN), which mines its local spatial characteristics of EEG signals while learning the temporal information and captures non-Euclidean spatial features between EEG channels to classify emotions more accurately. Specifically, we substitute the fully connected layers of the GRU with convolution layers, which can take into account both the local spatial features and time-series information of EEG signals. Next, multiple non-Euclidean spatial representations are generated using multidimensional dynamic graph convolution. Furthermore, the characteristics of different dimensions or views are merged through the adaptive-weighted summation technique to realize the maximized feature extraction as much as possible. Comprehensive experiments reveal that the constructed CGRU-MDGN could achieve competitive recognition results on subject-independent emotion recognition tasks.

A deep learning approach for subject-dependent & subject-independent emotion recognition using brain signals with dimensional emotion model

This paper aims to design a deep-learning based approach in combination with machine learning classifiers for two different perspectives. In first perspective, the performance is evaluated when training and testing are performed on same subject called as subject–dependent evaluation criteria. In second perspective, the performance is evaluated when training and testing are performed on different subjects called as subject–independent evaluation criteria. For each perspective, three label cases are made using valence, arousal, and dominance for recognizing human emotions: i) Binary/ 2-class, ii) Quad/ 4-class, and iii) Octal/ 8-class classifications. The experiment is performed on two publicly available datasets DEAP and DREAMER. For emotion recognition, firstly the brain signals are processed and then features are extracted using our proposed deep convolutional neural network (DCNN) architecture. These extracted features are used for emotion recognition using classifiers namely Naive Bayes (NB), decision tree (DT), k-Nearest Neighborhood (KNN), Support Vector Machine (SVM), AdaBoost (AB), Random Forest (RF), Neural Networks (NN), Long-short term memory (LSTM), and Bidirectional-LSTM (BiLSTM). The experimental results give more robust classification for subject-independent emotion recognition in comparison to subject-dependent emotion recognition, with DCNN + NN for binary and DCNN + SVM for quad & octal classification. Moreover, experimental results show that arousal and dominance play an important role in emotion recognition in contrary to valence and arousal as reported in literature.

Deep time-frequency features and semi-supervised dimension reduction for subject-independent emotion recognition from multi-channel EEG signals

In recent years, human emotion recognition has received great attention since it plays an essential role in human-computer interactions. Traditional methods focused on electroencephalogram(EEG) analysis in either time or frequency domain are unsuitable since EEG signals are nonlinear. This paper proposes subject-independent human emotion recognition from multi-channel EEG signals. The proposed method first obtains the time-frequency content of each channel using the modified Stockwell transform and then extracts the deep features from each time-frequency content by a deep convolutional neural network (CNN). Since there is a huge number of deep features, semi-supervised dimension reduction (SSDR) is utilized to reduce them, and reduced features of all channels are fused to construct the final feature vector. Several CNNs and classifiers are examined respectively for deep feature extraction and classification. The classification considering two-class and four-class scenarios on the DEAP dataset and three-class scenario on the SEED dataset show that the Inception-V3 CNN and support vector machine (SVM) classifier yields the highest accuracy. We present the extensive simulation to present the efficiency of the proposed method. Also, performance comparison with current methods based on time-frequency analysis demonstrates that the proposed method outperforms the others.

DAGAM: a domain adversarial graph attention model for subject-independent EEG-based emotion recognition

Objective. Due to individual differences in electroencephalogram (EEG) signals, the learning model built by the subject-dependent technique from one person’s data would be inaccurate when applied to another person for emotion recognition. Thus, the subject-dependent approach for emotion recognition may result in poor generalization performance when compared to the subject-independent approach. However, existing studies have attempted but have not fully utilized EEG’s topology, nor have they solved the problem caused by the difference in data distribution between the source and target domains. Approach. To eliminate individual differences in EEG signals, this paper proposes the domain adversarial graph attention model, a novel EEG-based emotion recognition model. The basic idea is to generate a graph using biological topology to model multichannel EEG signals. Graph theory can topologically describe and analyze EEG channel relationships and mutual dependencies. Then, unlike other graph convolutional networks, self-attention pooling is used to benefit from the extraction of salient EEG features from the graph, effectively improving performance. Finally, following graph pooling, the domain adversarial model based on the graph is used to identify and handle EEG variation across subjects, achieving good generalizability efficiently. Main Results. We conduct extensive evaluations on two benchmark data sets (SEED and SEED IV) and obtain cutting-edge results in subject-independent emotion recognition. Our model boosts the SEED accuracy to 92.59% (4.06% improvement) with the lowest standard deviation (STD) of 3.21% (2.46% decrements) and SEED IV accuracy to 80.74% (6.90% improvement) with the lowest STD of 4.14% (3.88% decrements), respectively. The computational complexity is drastically reduced in comparison to similar efforts (33 times lower). Significance. We have developed a model that significantly reduces the computation time while maintaining accuracy, making EEG-based emotion decoding more practical and generalizable.

TMLP+SRDANN: A domain adaptation method for EEG-based emotion recognition

Electroencephalogram (EEG) contains emotion information, but usually undergoes severe signal variations. In the literature, for the issue of EEG-based emotion recognition, not only have diverse features been designed to capture the representative characteristics of EEG, but also various domain adaptation techniques have been devised to reduce the distributional discrepancy between the source and target domains. However, existing domain adaptation techniques for this issue treat all the source samples equally, but ignore their transferability difference, which more or less limits the performance of learned domain-invariant and class-discriminative EEG features for emotion recognition. To cope with this problem, we propose Transposition Multi-Layer Perceptron (TMLP) and Sample-Reweighted Domain Adaptation Neural Network (SRDANN) in one whole learning framework. TMLP extracts the robust multi-channel EEG features related to emotions; SRDANN transfers the discriminative knowledge from the source domain to the target domain for classifying emotions. By concentrating on the source samples with stronger transferability in domain adaptation, TMLP+SRDANN can learn the more domain-invariant and class-discriminative features from EEG, thus providing a more effective solution to emotion recognition in the challenging cross-subject scenario. The subject-independent emotion recognition experiments on two benchmark datasets have demonstrated the effectiveness of our method: an accuracy of 81.04% for emotion classification on SEED, 61.88% for valence classification and 57.70% for arousal classification on DEAP.

An EEG-based subject-independent emotion recognition model using a differential-evolution-based feature selection algorithm

An EEG-based subject-independent emotion recognition model using a differential-evolution-based feature selection algorithm

Subject independent emotion recognition using EEG and physiological signals – a comparative study

PurposeThe aim of this study is to investigate subject independent emotion recognition capabilities of EEG and peripheral physiological signals namely: electroocoulogram (EOG), electromyography (EMG), electrodermal activity (EDA), temperature, plethysmograph and respiration. The experiments are conducted on both modalities independently and in combination. This study arranges the physiological signals in order based on the prediction accuracy obtained on test data using time and frequency domain features.Design/methodology/approachDEAP dataset is used in this experiment. Time and frequency domain features of EEG and physiological signals are extracted, followed by correlation-based feature selection. Classifiers namely – Naïve Bayes, logistic regression, linear discriminant analysis, quadratic discriminant analysis, logit boost and stacking are trained on the selected features. Based on the performance of the classifiers on the test set, the best modality for each dimension of emotion is identified.Findings The experimental results with EEG as one modality and all physiological signals as another modality indicate that EEG signals are better at arousal prediction compared to physiological signals by 7.18%, while physiological signals are better at valence prediction compared to EEG signals by 3.51%. The valence prediction accuracy of EOG is superior to zygomaticus electromyography (zEMG) and EDA by 1.75% at the cost of higher number of electrodes. This paper concludes that valence can be measured from the eyes (EOG) while arousal can be measured from the changes in blood volume (plethysmograph). The sorted order of physiological signals based on arousal prediction accuracy is plethysmograph, EOG (hEOG + vEOG), vEOG, hEOG, zEMG, tEMG, temperature, EMG (tEMG + zEMG), respiration, EDA, while based on valence prediction accuracy the sorted order is EOG (hEOG + vEOG), EDA, zEMG, hEOG, respiration, tEMG, vEOG, EMG (tEMG + zEMG), temperature and plethysmograph.Originality/valueMany of the emotion recognition studies in literature are subject dependent and the limited subject independent emotion recognition studies in the literature report an average of leave one subject out (LOSO) validation result as accuracy. The work reported in this paper sets the baseline for subject independent emotion recognition using DEAP dataset by clearly specifying the subjects used in training and test set. In addition, this work specifies the cut-off score used to classify the scale as low or high in arousal and valence dimensions. Generally, statistical features are used for emotion recognition using physiological signals as a modality, whereas in this work, time and frequency domain features of physiological signals and EEG are used. This paper concludes that valence can be identified from EOG while arousal can be predicted from plethysmograph.

Subject independent emotion recognition using EEG signals employing attention driven neural networks

Electroencephalogram (EEG) based emotional analysis has been employed in medical science, security and human–computer interaction with good success. In the recent past, deep learning-based approaches have significantly improved the classification accuracy when compared to classical signal processing and machine learning based frameworks. But most of them were subject-dependent studies which were not able to generalize on the subject-independent tasks due to the inter-subject variability present in EEG data. In this work, a novel deep learning framework capable of doing subject-independent emotion recognition is presented, consisting of two parts. First, an unsupervised Long Short-Term Memory (LSTM) with channel-attention autoencoder is proposed for getting a subject-invariant latent vector subspace i.e., intrinsic variables present in the EEG data of each individual. Secondly, a convolutional neural network (CNN) with attention framework is presented for performing the task of subject-independent emotion recognition on the encoded lower dimensional latent space representations obtained from the proposed LSTM with channel-attention autoencoder. With the attention mechanism, the proposed approach could highlight the significant time-segments of the EEG signal, which contributes to the emotion under consideration as validated by the results. The proposed approach has been validated using publicly available datasets for EEG signals such as DEAP dataset, SEED dataset and CHB-MIT dataset. With the proposed methodology, average subject independent accuracies of 65.9%, 69.5% for valence and arousal classification in the DEAP dataset and 76.7% for positive–negative classification in SEED dataset are obtained. Further for the CHB-MIT dataset, average subject independent accuracies of 69.1%, 67.6%, 72.3% for Pre-Ictal Vs Ictal, Inter-Ictal Vs Ictal, Pre-Ictal Vs Inter-Ictal classification are obtained, which are state-of-the-art to the best of our knowledge. The proposed end-to-end deep learning framework removes the requirement of different hand engineered features and provides a single comprehensive task agnostic EEG analysis tool capable of performing various kinds of EEG analysis on subject independent data.

Learning DenseNet features from EEG based spectrograms for subject independent emotion recognition

Subject-independent emotion recognition (SIER) using electroencephalogram (EEG) signals has always been a challenge among the biomedical research community. One of the major reasons behind this is the chaotic and non-stationary nature of EEG signals which may show varying signal characteristics for the same emotions for different individuals. Therefore, it is a daunting challenge for the researchers to enhance the emotion recognition performance of the existing SIER systems, which is still quite low. To address these challenges, in this work, we have proposed a novel deep learning-based approach to efficiently extract and classify emotion-related information from the 2D spectrograms obtained from the 1D EEG signals. To uncover the hidden deep features of EEG signals we proposed a Deep Convolution neural network for Emotion Recognition (DCERNet) with dense connections among layers. The proposed DCERNet is developed with a sequence of customizations over a pre-trained Densenet121 model integrated with softmax and Support Vector Machine (SVM) classifiers on top. The EEG signals are transformed to 2D spectrograms and fed to the proposed model for identifying emotions. To evaluate the performance of the proposed models, comprehensive simulations are conducted using two publicly available databases SJTU Emotion EEG Dataset (SEED) and Database for Emotion Analysis of Physiological Signals (DEAP). Experimental outcomes illustrate that the proposed DCERNet model boosts the emotion recognition accuracy by almost 8% compared to the state-of-the-art methods using DEAP database.

Automatic subject-specific spatiotemporal feature selection for subject-independent affective BCI.

The dimensionality of the spatially distributed channels and the temporal resolution of electroencephalogram (EEG) based brain-computer interfaces (BCI) undermine emotion recognition models. Thus, prior to modeling such data, as the final stage of the learning pipeline, adequate preprocessing, transforming, and extracting temporal (i.e., time-series signals) and spatial (i.e., electrode channels) features are essential phases to recognize underlying human emotions. Conventionally, inter-subject variations are dealt with by avoiding the sources of variation (e.g., outliers) or turning the problem into a subject-deponent. We address this issue by preserving and learning from individual particularities in response to affective stimuli. This paper investigates and proposes a subject-independent emotion recognition framework that mitigates the subject-to-subject variability in such systems. Using an unsupervised feature selection algorithm, we reduce the feature space that is extracted from time-series signals. For the spatial features, we propose a subject-specific unsupervised learning algorithm that learns from inter-channel co-activation online. We tested this framework on real EEG benchmarks, namely DEAP, MAHNOB-HCI, and DREAMER. We train and test the selection outcomes using nested cross-validation and a support vector machine (SVM). We compared our results with the state-of-the-art subject-independent algorithms. Our results show an enhanced performance by accurately classifying human affection (i.e., based on valence and arousal) by 16%–27% compared to other studies. This work not only outperforms other subject-independent studies reported in the literature but also proposes an online analysis solution to affection recognition.

NeuroSense: Short-term emotion recognition and understanding based on spiking neural network modelling of spatio-temporal EEG patterns

Emotion recognition still poses a challenge lying at the core of the rapidly growing area of affective computing and is crucial for establishing a successful human–computer interaction. Identification and understanding of emotions are achieved through various measures, such as subjective self-reports, face-tracking, voice analysis, gaze-tracking, as well as the analysis of autonomic and central neurophysiological measurements. Current approaches to emotion recognition based on electroencephalography (EEG) mostly rely on various handcrafted features extracted over relatively long time windows of EEG during participants exposure to appropriate affective stimuli. In this paper, we present a short-term emotion recognition framework based on spiking neural network (SNN) modelling of spatio-temporal EEG patterns. Our method relies on EEG signal segmentation based on detection of short-term changes in facial landmarks, and as such includes no computation of handcrafted EEG features. Differences between participants’ EEG properties are taken into account via subject-dependent spike encoding in the formulated subject-independent emotion recognition task. We test our methods on the publicly available DEAP and MAHNOB-HCI databases due to the availability of both EEG and frontal face video data. Through an exhaustive hyperparameter optimisation strategy, we show that the proposed SNN-based representation of EEG spiking patterns provides valuable information for short- term emotion recognition. The obtained accuracies are 78.97% and 79.39% in arousal classification, and 67.76% and 72.12% in valence classification, on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, through the application of a brain-inspired SNN model, this study provides novel insight and helps in the understanding of the neural mechanisms involved in emotional processing in the context of audiovisual stimuli, such as affective videos. The presented results encourage the use of the proposed EEG processing methodology as a complement to existing features and methods commonly used for EEG-based emotion recognition, especially for short-term arousal recognition.

FLDNet: Frame-Level Distilling Neural Network for EEG Emotion Recognition.

Based on the current research on EEG emotion recognition, there are some limitations, such as hand-engineered features, redundant and meaningless signal frames and the loss of frame-to-frame correlation. In this paper, a novel deep learning framework is proposed, named the frame-level distilling neural network (FLDNet), for learning distilled features from the correlations of different frames. A layer named the frame gate is designed to integrate weighted semantic information on multiple frames to remove redundant and meaningless signal frames. A triple-net structure is introduced to distill the learned features net by net to replace the hand-engineered features with professional knowledge. Specifically, one neural network is normally trained for several epochs. Then, a second network of the same structure will be initialized again to learn the extracted features from the frame gate of the first neural network based on the output of the first net. Similarly, the third net improves the features based on the frame gate of the second network. To utilize the representation ability of the triple neural network, an ensemble layer is conducted to integrate the discriminative ability of the proposed framework for final decisions. Consequently, the proposed FLDNet provides an effective method for capturing the correlation between different frames and automatically learn distilled high-level features for emotion recognition. The experiments are carried out in a subject-independent emotion recognition task on public emotion datasets of DEAP and DREAMER benchmarks, which have demonstrated the effectiveness and robustness of the proposed FLDNet.

Subject-Independent Emotion Recognition of EEG Signals Based on Dynamic Empirical Convolutional Neural Network

Affective computing is one of the key technologies to achieve advanced brain-machine interfacing. It is increasingly concerning research orientation in the field of artificial intelligence. Emotion recognition is closely related to affective computing. Although emotion recognition based on electroencephalogram (EEG) has attracted more and more attention at home and abroad, subject-independent emotion recognition still faces enormous challenges. We proposed a subject-independent emotion recognition algorithm based on dynamic empirical convolutional neural network (DECNN) in view of the challenges. Combining the advantages of empirical mode decomposition (EMD) and differential entropy (DE), we proposed a dynamic differential entropy (DDE) algorithm to extract the features of EEG signals. After that, the extracted DDE features were classified by convolutional neural networks (CNN). Finally, the proposed algorithm is verified on SJTU Emotion EEG Dataset (SEED). In addition, we discuss the brain area closely related to emotion and design the best profile of electrode placements to reduce the calculation and complexity. Experimental results show that the accuracy of this algorithm is 3.53 percent higher than that of the state-of-the-art emotion recognition methods. What's more, we studied the key electrodes for EEG emotion recognition, which is of guiding significance for the development of wearable EEG devices.

Subject independent emotion recognition system for people with facial deformity: an EEG based approach

Emotion recognition from Electroencephalography (EEG) is a better choice for the people with facial deformity like where facial data is not accurate or not available for example burned or paralyzed faces. This research exploits the image processing capability of convolutional neural network (CNN) and proposes a CNN model to classify different emotions from the scalogram images of EEG data. Scalogram images from EEG obtained by applying continuous wavelet transform used for the study. The proposed model is subject independent where the objective is to extract emotion specific features from EEG data irrespective of the source of the data. The proposed emotion recognition model is evaluated on two benchmark public databases namely DEAP and SEED. In order to show the model as a purely subject independent one, the cross data base criteria is also used for evaluation. The various performance evaluation experiments show that the proposed model is comparable in terms of emotion classification accuracy.

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine.

These days, physiological signals have been studied more broadly for emotion recognition to realize emotional intelligence in human-computer interaction. However, due to the complexity of emotions and individual differences in physiological responses, how to design reliable and effective models has become an important issue. In this article, we propose a regularized deep fusion framework for emotion recognition based on multimodal physiological signals. After extracting the effective features from different types of physiological signals, we construct ensemble dense embeddings of multimodal features using kernel matrices, and then utilize a deep network architecture to learn task-specific representations for each kind of physiological signal from these ensemble dense embeddings. Finally, a global fusion layer with a regularization term, which can efficiently explore the correlation and diversity among all of the representations in a synchronous optimization process, is designed to fuse generated representations. Experiments on two benchmark datasets show that this framework can improve the performance of subject-independent emotion recognition compared to single-modal classifiers or other fusion methods. Data visualization also demonstrates that the final fusion representation exhibits higher class-separability power for emotion recognition.

A Systematic Exploration of Deep Neural Networks for EDA-Based Emotion Recognition

Subject-independent emotion recognition based on physiological signals has become a research hotspot. Previous research has proved that electrodermal activity (EDA) signals are an effective data resource for emotion recognition. Benefiting from their great representation ability, an increasing number of deep neural networks have been applied for emotion recognition, and they can be classified as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a combination of these (CNN+RNN). However, there has been no systematic research on the predictive power and configurations of different deep neural networks in this task. In this work, we systematically explore the configurations and performances of three adapted deep neural networks: ResNet, LSTM, and hybrid ResNet-LSTM. Our experiments use the subject-independent method to evaluate the three-class classification on the MAHNOB dataset. The results prove that the CNN model (ResNet) reaches a better accuracy and F1 score than the RNN model (LSTM) and the CNN+RNN model (hybrid ResNet-LSTM). Extensive comparisons also reveal that our three deep neural networks with EDA data outperform previous models with handcraft features on emotion recognition, which proves the great potential of the end-to-end DNN method.