Harnessing Deep Learning for EEG Emotion Recognition: A Hybrid Approach with Attention Mechanisms

Emotion is the human brain reacting to objective things. In real life, human emotions are complex and changeable, so research into emotion recognition is of great significance in real life applications. Recently, many deep learning and machine learning methods have been widely applied in emotion recognition based on EEG signals. However, the traditional machine learning method has a major disadvantage in that the feature extraction process is usually cumbersome, which relies heavily on human experts. Then, end-to-end deep learning methods emerged as an effective method to address this disadvantage with the help of raw signal features and time-frequency spectrums. Here, we investigated the application of several deep learning models to the research field of EEG-based emotion recognition, including deep neural networks (DNN), convolutional neural networks (CNN), long short-term memory (LSTM), and a hybrid model of CNN and LSTM (CNN-LSTM). The experiments were carried on the well-known DEAP dataset. Experimental results show that the CNN and CNN-LSTM models had high classification performance in EEG-based emotion recognition, and their accurate extraction rate of RAW data reached 90.12 and 94.17%, respectively. The performance of the DNN model was not as accurate as other models, but the training speed was fast. The LSTM model was not as stable as the CNN and CNN-LSTM models. Moreover, with the same number of parameters, the training speed of the LSTM was much slower and it was difficult to achieve convergence. Additional parameter comparison experiments with other models, including epoch, learning rate, and dropout probability, were also conducted in the paper. Comparison results prove that the DNN model converged to optimal with fewer epochs and a higher learning rate. In contrast, the CNN model needed more epochs to learn. As for dropout probability, reducing the parameters by ~50% each time was appropriate.

EEG emotion recognition based on TQWT-features and hybrid convolutional recurrent neural network

Through emotion recognition with EEG signals, brain responses can be analyzed to monitor and identify individual emotional states. The success of emotion recognition relies on comprehensive emotion information extracted from EEG signals and the constructed emotion identification model. In this work, we proposed an innovative approach, called spatial-spectral-temporal-based convolutional recurrent neural network (CRNN) with lightweight attention mechanism (SST-CRAM). Firstly, we combined power spectral density (PSD) with differential entropy (DE) features to construct four-dimensional (4D) EEG feature maps and obtain more spatial, spectral, and temporal information. Additional, with a spatial interpolation algorithm, the utilization of the obtained valuable information was enhanced. Next, the constructed 4D EEG feature map was input into the convolutional neural network (CNN) integrated with convolutional block attention module (CBAM) and efficient channel attention module (ECA-Net) for extracting spatial and spectral features. CNN was used to learn spatial and spectral information and CBAM was employed to prioritize global information and obtain detailed and accurate features. ECA-Net was also used to further highlight key brain regions and frequency bands. Finally, a bidirectional long short-term memory (LSTM) network was used to explore the temporal correlation of EEG feature maps for comprehensive feature extraction. To assess the performance of our model, we tested it on the publicly available DEAP dataset. Our model demonstrated excellent performance and achieved high accuracy (98.63% for arousal classification and 98.66% for valence classification). These results indicated that SST-CRAM could fully utilize spatial, spectral, and temporal information to improve the emotion recognition performance.

Emotion is a fundamental aspect of daily life and is crucial for human interactions. This study suggests a unique electroencephalogram (EEG)-based technique for identifying human emotions. For EEG-based emotion analysis, the proposed model is tested on the SEED and DEAP datasets. For the DEAP dataset, we consider valence and arousal emotions for classification purposes, and for the SEED dataset, three emotions, neutral, positive, and negative, have been considered. The differential entropy (DE) is used for the SEED dataset, and for the DEAP dataset, the power spectral density (PSD) is used as a feature. For precise emotion recognition, an EmHM (Emotion Hybrid Model) based on long short-term memory (LSTM) and a convolutional neural network (CNN) are constructed. Furthermore, we applied the CNN, LSTM, and EmHM models, and all three models for emotion recognition are fed with the retrieved information. Various methods improved on already-existing models to accurately classify human emotion. To get better accuracy than the existing techniques, we suggested a model that uses a different approach known as EmHM. By applying all three models such as CNN, LSTM and EmHM, we got the highest accuracy 86.50%, 87.98% and 91.56% respectively on dataset. To improve prediction results, the CNN and the LSTM models are combined to make the EmHM hybrid model.

In recent years, deep learning has been widely used in emotion recognition, but the models and algorithms in practical applications still have much room for improvement. With the development of graph convolutional neural networks, new ideas for emotional recognition based on EEG have arisen. In this paper, we propose a novel deep learning model-based emotion recognition method. First, the EEG signal is spatially filtered by using the common spatial pattern (CSP), and the filtered signal is converted into a time–frequency map by continuous wavelet transform (CWT). This is used as the input data of the network; then the feature extraction and classification are performed by the deep learning model. We called this model CNN-BiLSTM-MHSA, which consists of a convolutional neural network (CNN), bi-directional long and short-term memory network (BiLSTM), and multi-head self-attention (MHSA). This network is capable of learning the time series and spatial information of EEG emotion signals in depth, smoothing EEG signals and extracting deep features with CNN, learning emotion information of future and past time series with BiLSTM, and improving recognition accuracy with MHSA by reassigning weights to emotion features. Finally, we conducted experiments on the DEAP dataset for sentiment classification, and the experimental results showed that the method has better results than the existing classification. The accuracy of high and low valence, arousal, dominance, and liking state recognition is 98.10%, and the accuracy of four classifications of high and low valence-arousal recognition is 89.33%.

Emotions serve various functions. The traditional emotion recognition methods are based primarily on readily accessible facial expressions, gestures, and voice signals. However, it is often challenging to ensure that these non-physical signals are valid and reliable in practical applications. Electroencephalogram (EEG) signals are more successful than other signal recognition methods in recognizing these characteristics in real-time since they are difficult to camouflage. Although EEG signals are commonly used in current emotional recognition research, the accuracy is low when using traditional methods. Therefore, this study presented an optimized hybrid pattern with an attention mechanism (FFT_CLA) for EEG emotional recognition. First, the EEG signal was processed via the fast fourier transform (FFT), after which the convolutional neural network (CNN), long short-term memory (LSTM), and CNN-LSTM-attention (CLA) methods were used to extract and classify the EEG features. Finally, the experiments compared and analyzed the recognition results obtained via three DEAP dataset models, namely FFT_CNN, FFT_LSTM, and FFT_CLA. The final experimental results indicated that the recognition rates of the FFT_CNN, FFT_LSTM, and FFT_CLA models within the DEAP dataset were 87.39%, 88.30%, and 92.38%, respectively. The FFT_CLA model improved the accuracy of EEG emotion recognition and used the attention mechanism to address the often-ignored importance of different channels and samples when extracting EEG features.

Emotion recognition (ER) is crucial for understanding human behaviours, social interactions, and psychological well-being. Electroencephalography (EEG) has emerged as a promising tool for capturing the neural correlates of emotions. This work is a systematic review of articles in ER using EEG signals. A total of 120 articles from 1041 articles were selected based on PRISMA guidelines using defined inclusion and exclusion criteria, published between 2018 and 2024. This article aims to provide an in-depth understanding of the current landscape of ER from EEG signals utilizing deep learning (DL). This review offers valuable guidance for researchers and practitioners seeking more refined and reliable emotion classification systems. To explore the effectiveness of DL models in EEG-based ER, several potential DL models, such as convolutional neural network, long short-term memory (LSTM), gated recurrent unit (GRU), hybrid bidirectional LSTM (BiLSTM), bidirectional GRU, and advanced DL models such as convolutional recurrent neural network and EEG-Conformer models are applied to two popular datasets, SEED and GAMEEMO, respectively, to depict the full process of ER. Additionally, the performance of DL models is also compared with the performance of basic machine learning (ML) models such as SVM, k-nearest neighbors, logistic regression, and boosting algorithms such as AdaBoost, XGBoost and LightGBM. Through extensive experiments and performance evaluations, the performance of different models when applied to the datasets mentioned above is compared. The accuracy, precision, recall, and F1-scores are analysed to determine the most effective model for EEG-based ER. The findings of this study demonstrate that the performance of hybrid DL models is more efficacious than that of ML models. The best-performing model (BiLSTM) classified the emotions, with an accuracy of 90.54% when applied to the GAMEEMO dataset. This research contributes to the growing body of literature on ER and provides insights into the feasibility of using EEG signals to understand emotional states, and presents a structured roadmap for future exploration. The findings can aid in the development of more accurate and reliable ER systems, which can have wide-ranging applications in psychology, social sciences, and human-computer interactions.

EEG-based emotion recognition through artificial intelligence is one of the major areas of biomedical and machine learning, which plays a key role in understanding brain activity and developing decision-making systems. However, the traditional EEG-based emotion recognition is a single feature input mode, which cannot obtain multiple feature information, and cannot meet the requirements of intelligent and high real-time brain computer interface. And because the EEG signal is nonlinear, the traditional methods of time domain or frequency domain are not suitable. In this paper, a CNN-DSC-Bi-LSTM-Attention (CDBA) model based on EEG signals for automatic emotion recognition is presented, which contains three feature-extracted channels. The normalized EEG signals are used as an input, the feature of which is extracted by multi-branching and then concatenated, and each channel feature weight is assigned through the attention mechanism layer. Finally, Softmax was used to classify EEG signals. To evaluate the performance of the proposed CDBA model, experiments were performed on SEED and DREAMER datasets, separately. The validation experimental results show that the proposed CDBA model is effective in classifying EEG emotions. For triple-category (positive, neutral and negative) and four-category (happiness, sadness, fear and neutrality), the classification accuracies were respectively 99.44% and 99.99% on SEED datasets. For five classification (Valence 1-Valence 5) on DREAMER datasets, the accuracy is 84.49%. To further verify and evaluate the model accuracy and credibility, the multi-classification experiments based on ten-fold cross-validation were conducted, the elevation indexes of which are all higher than other models. The results show that the multi-branch feature fusion deep learning model based on attention mechanism has strong fitting and generalization ability and can solve nonlinear modeling problems, so it is an effective emotion recognition method. Therefore, it is helpful to the diagnosis and treatment of nervous system diseases, and it is expected to be applied to emotion-based brain computer interface systems.

In this paper, we propose an emotion recognition model based on convolutional neural network (CNN), long short term memory (LSTM) and channel attention mechanism, aiming at the low classification accuracy of machine learning methods and the uneven spatial distribution of electroencephalogram (EEG) electrodes. This model can effectively integrate the frequency, space and time information of EEG signals, and improve the accuracy of emotion recognition by adding channel attention mechanism after the last convolutional layer of the model. Firstly, construct a 4-dimensional structure representing EEG signals. Then, a CLSTM model structure combining CNN and LSTM is designed. CNN is used to extract frequency and spatial information from 4-dimensional input, and LSTM is used to extract time information. Finally, the channel attention module is added after the last convolutional layer of CLSTM model structure to allocate the weight of different electrodes. In this paper, an emotion recognition model based on CLSTM and channel attention mechanism was proposed from the perspective of integrating the frequency, space and time 3-dimensional information of EEG signals. The average classification accuracy of the model on SEED public data set reached 93.36%, which was significantly improved over the existing CNN and LSTM emotion recognition models.

In this paper, we present a novel method, called four-dimensional convolutional recurrent neural network, which integrating frequency, spatial and temporal information of multichannel EEG signals explicitly to improve EEG-based emotion recognition accuracy. First, to maintain these three kinds of information of EEG, we transform the differential entropy features from different channels into 4D structures to train the deep model. Then, we introduce CRNN model, which is combined by convolutional neural network (CNN) and recurrent neural network with long short term memory (LSTM) cell. CNN is used to learn frequency and spatial information from each temporal slice of 4D inputs, and LSTM is used to extract temporal dependence from CNN outputs. The output of the last node of LSTM performs classification. Our model achieves state-of-the-art performance both on SEED and DEAP datasets under intra-subject splitting. The experimental results demonstrate the effectiveness of integrating frequency, spatial and temporal information of EEG for emotion recognition.

Abstract. Emotion recognition is a branch of artificial intelligence that analyzes human emotional states through facial expressions, voice, or physiological signals. It enhances human-computer interaction, facilitating more personalized and empathetic technology experiences, crucial for fields like mental health, customer service, and human-robot interaction. In recent years, research on emotion recognition using these tools has grown rapidly, involving multiple interdisciplinary fields. With the aid of electroencephalogram (EEG)-based brain-computer interfaces (BCIs), the emotional states of users can be sensed and analyzed. It offers a direct, non-intrusive insight into user emotions, enhancing user experience and system responsiveness. This approach is crucial for developing adaptive artificial intelligence (AI) in fields like healthcare for personalized treatments and in entertainment for immersive experiences, advancing human-technology symbiosis. This paper compares five current machine learning (ML)-based emotion recognition methods leveraging EEG signals, aiming to evaluate their effectiveness and applicability in emotion recognition. The paper concludes that while both Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) have their strengths, the combination of them provides the best performance in EEG-based emotion recognition.

EEG-based emotion recognition can help achieve more natural human-computer interaction, but the temporal non-stationarity of EEG signals affects the robustness of EEG-based emotion recognition models. Most existing studies use the emotional EEG data collected in the same trial to train and test models, once this kind of model is applied to the data collected at different times of the same subject, its recognition accuracy will decrease significantly. To address the problem of EEG-based cross-day emotion recognition, this paper has constructed a database of emotional EEG signals collected over six days for each subject using the Chinese Affective Video System and self-built video library stimuli materials, and the database is the largest number of days collected for a single subject so far. To study the neural patterns of emotions based on EEG signals cross-day, the brain topography has been analyzed in this paper, which show there is a stable neural pattern of emotions cross-day. Then, Transfer Component Analysis (TCA) algorithm is used to adaptively determine the optimal dimensionality of the TCA transformation and match domains of the best correlated motion features in multiple time domains by using EEG signals from different time (days). The experimental results show that the TCA-based domain adaptation strategy can effectively improve the accuracy of cross-day emotion recognition by 3.55% and 2.34%, respectively, in the classification of joy-sadness and joy-anger emotions. The emotion recognition model and brain topography in this paper, verify that the database can provide a reliable data basis for emotion recognition across different time domains. This EEG database will be open to more researchers to promote the practical application of emotion recognition.

Emotion plays a nuclear part in human attention, decision-making, and communication. Electroencephalogram (EEG)-based emotion recognition has developed a lot due to the application of Brain-Computer Interface (BCI) and its effectiveness compared to body expressions and other physiological signals. Despite significant progress in affective computing, emotion recognition is still an unexplored problem. This paper introduced Logistic Regression (LR) with Gaussian kernel and Laplacian prior for EEG-based emotion recognition. The Gaussian kernel enhances the EEG data separability in the transformed space. The Laplacian prior promotes the sparsity of learned LR regressors to avoid over-specification. The LR regressors are optimized using the logistic regression via variable splitting and augmented Lagrangian (LORSAL) algorithm. For simplicity, the introduced method is noted as LORSAL. Experiments were conducted on the dataset for emotion analysis using EEG, physiological and video signals (DEAP). Various spectral features and features by combining electrodes (power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), rational asymmetry (RASM), and differential caudality (DCAU)) were extracted from different frequency bands (Delta, Theta, Alpha, Beta, Gamma, and Total) with EEG signals. The Naive Bayes (NB), support vector machine (SVM), linear LR with L1-regularization (LR_L1), linear LR with L2-regularization (LR_L2) were used for comparison in the binary emotion classification for valence and arousal. LORSAL obtained the best classification accuracies (77.17% and 77.03% for valence and arousal, respectively) on the DE features extracted from total frequency bands. This paper also investigates the critical frequency bands in emotion recognition. The experimental results showed the superiority of Gamma and Beta bands in classifying emotions. It was presented that DE was the most informative and DASM and DCAU had lower computational complexity with relatively ideal accuracies. An analysis of LORSAL and the recently deep learning (DL) methods is included in the discussion. Conclusions and future work are presented in the final section.

This work aims to investigate the performance of the Long Short-Term Memory (LSTM) Model for EEG-Based Emotion Recognition. For the experimentation, we use the publicly available DEAP dataset, which consists of preprocessed EEG and physiological signals. Our work limits itself to the study of only the EEG signals to have a scope for developing an efficient headgear model for real-time monitoring of emotions. In this study, we extract the band power, a frequency-domain feature, from the EEG signals and compare the classification accuracies for Valence and Arousal domain for different classifiers. The proposed Long Short-Term Memory (LSTM) model achieves the best classification accuracy of 94.69% and 93.13% for Valence and Arousal scales, respectively, illustrating a significant average increment of 16% in valence and 18% in arousal in comparison to other classifiers.

The emotional state of people plays a key role in physiological and behavioral human interaction. Emotional state analysis entails many fields such as neuroscience, cognitive sciences, and biomedical engineering because the parameters of interest contain the complex neuronal activities of the brain. Electroencephalogram (EEG) signals are processed to communicate brain signals with external systems and make predictions over emotional states. This paper proposes a novel method for emotion recognition based on deep convolutional neural networks (CNNs) that are used to classify Valence, Arousal, Dominance, and Liking emotional states. Hence, a novel approach is proposed for emotion recognition with time series of multi-channel EEG signals from a Database for Emotion Analysis and Using Physiological Signals (DEAP). We propose a new approach to emotional state estimation utilizing CNN-based classification of multi-spectral topology images obtained from EEG signals. In contrast to most of the EEG-based approaches that eliminate spatial information of EEG signals, converting EEG signals into a sequence of multi-spectral topology images, temporal, spectral, and spatial information of EEG signals are preserved. The deep recurrent convolutional network is trained to learn important representations from a sequence of three-channel topographical images. We have achieved test accuracy of 90.62% for negative and positive Valence, 86.13% for high and low Arousal, 88.48% for high and low Dominance, and finally 86.23% for like-unlike. The evaluations of this method on emotion recognition problem revealed significant improvements in the classification accuracy when compared with other studies using deep neural networks (DNNs) and one-dimensional CNNs.