Cross-subject Emotion Recognition Research Articles

MSS-JDA: Multi-source self-selected joint domain adaptation method based on cross-subject EEG emotion recognition

The electroencephalogram (EEG) is widely used to identify different emotional states. However, cross-subject emotion recognition based on EEG data is challenging due to physiological differences among participants. Many studies have proposed using multi-source domain adaptation methods to address cross-subject issues and have achieved promising results. multi-source domain adaptation involves constructing independent branches for EEG data from multiple subjects, using one-to-one domain adaptation, and extracting domain-specific features, followed by reasoning through multiple branches. However, a current practical obstacle is the increasing computational time cost as the number of source domains grows. Moreover, too many source domains may lead to domain shift issues. To address these challenges, we propose a novel emotion recognition method based on Multi-Source Domain Branch Self-Selected Joint Domain Adaptation (MSS-JDA). Initially, we construct a shared feature extractor for multiple source domain branches to extract common low-level features. Next, we use a one-to-one associated joint adaptation method to extract features specific to each domain. During training, our method of multi-source domain branch self-selection, guided by prior knowledge in the early training phase, prunes source domain branches that deviate significantly from the target domain, retaining only a certain number of source domain branches close to the target domain for continued training. Finally, domain-specific features are input into their respective classifiers to obtain N predictions, and their average is taken as the final result. Extensive experiments on the SEED and SEED-IV datasets demonstrate that the MSS-JDA model achieves accuracy and standard deviation scores of 93.78%, 3.39 and 78.93%, 7.39 respectively, surpassing most existing models. The experiments indicate that the proposed MSS-JDA method improves the accuracy of EEG emotion recognition classification, thus benefiting practical applications in EEG classification.

Multi-source Selective Graph Domain Adaptation Network for cross-subject EEG emotion recognition

Affective brain-computer interface is an important part of realizing emotional human–computer interaction. However, existing objective individual differences among subjects significantly hinder the application of electroencephalography (EEG) emotion recognition. Existing methods still lack the complete extraction of subject-invariant representations for EEG and the ability to fuse valuable information from multiple subjects to facilitate the emotion recognition of the target subject. To address the above challenges, we propose a Multi-source Selective Graph Domain Adaptation Network (MSGDAN), which can better utilize data from different source subjects and perform more robust emotion recognition on the target subject. The proposed network extracts and selects the individual information specific to each subject, where public information refers to subject-invariant components from multi-source subjects. Moreover, the graph domain adaptation network captures both functional connectivity and regional states of the brain via a dynamic graph network and then integrates graph domain adaptation to ensure the invariance of both functional connectivity and regional states. To evaluate our method, we conduct cross-subject emotion recognition experiments on the SEED, SEED-IV, and DEAP datasets. The results demonstrate that the MSGDAN has superior classification performance.

Cross-subject emotion recognition in brain-computer interface based on frequency band attention graph convolutional adversarial neural networks

Background:Emotion is an important area in neuroscience. Cross-subject emotion recognition based on electroencephalogram (EEG) data is challenging due to physiological differences between subjects. Domain gap, which refers to the different distributions of EEG data at different subjects, has attracted great attention for cross-subject emotion recognition. Comparison with existing methods:This study focuses on narrowing the domain gap between subjects through the emotional frequency bands and the relationship information between EEG channels. Emotional frequency band features represent the energy distribution of EEG data in different frequency ranges, while relationship information between EEG channels provides spatial distribution information about EEG data. New method:To achieve this, this paper proposes a model called the Frequency Band Attention Graph convolutional Adversarial neural Network (FBAGAN). This model includes three components: a feature extractor, a classifier, and a discriminator. The feature extractor consists of a layer with a frequency band attention mechanism and a graph convolutional neural network. The mechanism effectively extracts frequency band information by assigning weights and Graph Convolutional Networks can extract relationship information between EEG channels by modeling the graph structure. The discriminator then helps minimize the gap in the frequency information and relationship information between the source and target domains, improving the model’s ability to generalize. Results:The FBAGAN model is extensively tested on the SEED, SEED-IV, and DEAP datasets. The accuracy and standard deviation scores are 88.17% and 4.88, respectively, on the SEED dataset, and 77.35% and 3.72 on the SEED-IV dataset. On the DEAP dataset, the model achieves 69.64% for Arousal and 65.18% for Valence. These results outperform most existing models. Conclusions:The experiments indicate that FBAGAN effectively addresses the challenges of transferring EEG channel domain and frequency band domain, leading to improved performance.

Label-refined EEG emotion classification based on intelligent media communication

With the development of cognitive computing and intelligent media communication, recognizing the users emotion using electroencephalography (EEG) has garnered increasing attention. However, building a cross-subject emotion recognition model with good generalization performance is really difficult. To achieve remarkable performance, a label-refined domain adversarial neural network is designed and EEG samples are subdivided into 6 groups according to the differences in users' brain activities under different movie stimuli. Based on the experiment of SEED and SEED-IV datasets, the classification accuracy of our method reaches 93.354.14% and 84.086.01%, achieving the state-of-the-art performance. Experiments show that using different movie stimuli to refine labels can achieve efficient cross-subject emotion recognition.

SFT-SGAT: A semi-supervised fine-tuning self-supervised graph attention network for emotion recognition and consciousness detection

Emotional recognition is highly important in the field of brain-computer interfaces (BCIs). However, due to the individual variability in electroencephalogram (EEG) signals and the challenges in obtaining accurate emotional labels, traditional methods have shown poor performance in cross-subject emotion recognition. In this study, we propose a cross-subject EEG emotion recognition method based on a semi-supervised fine-tuning self-supervised graph attention network (SFT-SGAT). First, we model multi-channel EEG signals by constructing a graph structure that dynamically captures the spatiotemporal topological features of EEG signals. Second, we employ a self-supervised graph attention neural network to facilitate model training, mitigating the impact of signal noise on the model. Finally, a semi-supervised approach is used to fine-tune the model, enhancing its generalization ability in cross-subject classification. By combining supervised and unsupervised learning techniques, the SFT-SGAT maximizes the utility of limited labeled data in EEG emotion recognition tasks, thereby enhancing the model’s performance. Experiments based on leave-one-subject-out cross-validation demonstrate that SFT-SGAT achieves state-of-the-art cross-subject emotion recognition performance on the SEED and SEED-IV datasets, with accuracies of 92.04% and 82.76%, respectively. Furthermore, experiments conducted on a self-collected dataset comprising ten healthy subjects and eight patients with disorders of consciousness (DOCs) revealed that the SFT-SGAT attains high classification performance in healthy subjects (maximum accuracy of 95.84%) and was successfully applied to DOC patients, with four patients achieving emotion recognition accuracies exceeding 60%. The experiments demonstrate the effectiveness of the proposed SFT-SGAT model in cross-subject EEG emotion recognition and its potential for assessing levels of consciousness in patients with DOC.

Evolutionary Ensemble Learning for EEG-Based Cross-Subject Emotion Recognition.

Electroencephalogram (EEG) has been widely utilized in emotion recognition due to its high temporal resolution and reliability. However, the individual differences and non-stationary characteristics of EEG, along with the complexity and variability of emotions, pose challenges in generalizing emotion recognition models across subjects. In this paper, an end-to-end framework is proposed to improve the performance of cross-subject emotion recognition. A novel evolutionary programming (EP)-based optimization strategy with neural network (NN) as the base classifier termed NN ensemble with EP (EPNNE) is designed for cross-subject emotion recognition. The effectiveness of the proposed method is evaluated on the publicly available DEAP, FACED, SEED, and SEED-IV datasets. Numerical results demonstrate that the proposed method is superior to state-of-the-art cross-subject emotion recognition methods. The proposed end-to-end framework for cross-subject emotion recognition aids biomedical researchers in effectively assessing individual emotional states, thereby enabling efficient treatment and interventions.

Semi-supervised bipartite graph construction with active EEG sample selection for emotion recognition.

Electroencephalogram (EEG) signals are derived from the central nervous system and inherently difficult to camouflage, leading to the recent popularity of EEG-based emotion recognition. However, due to the non-stationary nature of EEG, inter-subject variabilities become obstacles for recognition models to well adapt to different subjects. In this paper, we propose a novel approach called semi-supervised bipartite graph construction with active EEG sample selection (SBGASS) for cross-subject emotion recognition, which offers two significant advantages. Firstly, SBGASS adaptively learns a bipartite graph to characterize the underlying relationships between labeled and unlabeled EEG samples, effectively implementing the semantic connection for samples from different subjects. Secondly, we employ active sample selection technique in this paper to reduce the impact of negative samples (outliers or noise in the data) on bipartite graph construction. Drawing from the experimental results with the SEED-IV data set, we have gained the following three insights. (1) SBGASS actively rejects negative labeled samples, which helps mitigate the impact of negative samples when constructing the optimal bipartite graph and improves the model performance. (2) Through the learned optimal bipartite graph in SBGASS, the transferability of labeled EEG samples is quantitatively analyzed, which exhibits a decreasing tendency as the distance between each labeled sample and the corresponding class centroid increases. (3) Besides the improved recognition accuracy, the spatial-frequency patterns in emotion recognition are investigated by the acquired projection matrix.

Generative Listener EEG for Speech Emotion Recognition Using Generative Adversarial Networks with Compressed Sensing.

Currently, emotional features in speech emotion recognition are typically extracted from the speeches, However, recognition accuracy can be influenced by factors such as semantics, language, and cross-speech datasets. Achieving consistent emotional judgment with human listeners is a key challenge for AI to address. Electroencephalography (EEG) signals prove to be an effective means of capturing authentic and meaningful emotional information in humans. This positions EEG as a promising tool for detecting emotional cues conveyed in speech. In this study, we proposed a novel approach named CS-GAN that generates listener EEGs in response to a speaker's speech, specifically aimed at enhancing cross-subject emotion recognition. We utilized generative adversarial networks (GANs) to establish a mapping relationship between speech and EEGs to generate stimulus-induced EEGs. Furthermore, we integrated compressive sensing theory (CS) into the GAN-based EEG generation method, thereby enhancing the fidelity and diversity of the generated EEGs. The generated EEGs were then processed using a CNN-LSTM model to identify the emotional categories conveyed in the speech. By averaging these EEGs, we obtained the event-related potentials (ERPs) to improve the cross-subject capability of the method. The experimental results demonstrate that the generated EEGs by this method outperform real listener EEGs by 9.31% in cross-subject emotion recognition tasks. Furthermore, the ERPs show an improvement of 43.59%, providing evidence for the effectiveness of this method in cross-subject emotion recognition.

Online multi-hypergraph fusion learning for cross-subject emotion recognition

Multimodal fusion for emotion recognition has received increasing attention from researchers because of its ability to effectively leverage multimodal complementary information. However, there are two main challenges lead to performance degradation of existing emotion recognition models, which limits the practical use of existing models. One is that multimodal signals are difficult to fuse effectively to respond to the complexity of emotions. The other is that the individual variability and non-stationarity of physiological signals lead to poor performance of the model on new subjects. In particular, existing methods will not work well when faced with emotion recognition of new subjects in online scenarios. In this paper, we propose a novel online multi-hypergraph fusion learning method (OnMHF) to effectively fuse multimodal information and to reduce the difference between training data and test data for online cross-subject emotion recognition. Specifically, in a training phase, a multi-hypergraph fusion is proposed to fuse multimodal physiological signals to effectively obtain emotion-aware information via leveraging multimodal complementary information and high-order correlations among multimodal signals. In an online recognizing phase, an online multi-hypergraph learning is designed to learn online multimodal information from online multimodal data by updating hypergraph structure. As a result, the proposed method can be more effective for emotion recognition of target subjects when target data arrive in an online manner. Experimental results have demonstrated that the proposed method outperforms the baselines and compared state-of-the-art methods in online emotion recognition tasks.

ST-SCGNN: A Spatio-Temporal Self-Constructing Graph Neural Network for Cross-Subject EEG-Based Emotion Recognition and Consciousness Detection.

In this paper, a novel spatio-temporal self-constructing graph neural network (ST-SCGNN) is proposed for cross-subject emotion recognition and consciousness detection. For spatio-temporal feature generation, activation and connection pattern features are first extracted and then combined to leverage their complementary emotion-related information. Next, a self-constructing graph neural network with a spatio-temporal model is presented. Specifically, the graph structure of the neural network is dynamically updated by the self-constructing module of the input signal. Experiments based on the SEED and SEED-IV datasets showed that the model achieved average accuracies of 85.90% and 76.37%, respectively. Both values exceed the state-of-the-art metrics with the same protocol. In clinical besides, patients with disorders of consciousness (DOC) suffer severe brain injuries, and sufficient training data for EEG-based emotion recognition cannot be collected. Our proposed ST-SCGNN method for cross-subject emotion recognition was first attempted in training in ten healthy subjects and testing in eight patients with DOC. We found that two patients obtained accuracies significantly higher than chance level and showed similar neural patterns with healthy subjects. Covert consciousness and emotion-related abilities were thus demonstrated in these two patients. Our proposed ST-SCGNN for cross-subject emotion recognition could be a promising tool for consciousness detection in DOC patients.

Cross-subject EEG-based emotion recognition through dynamic optimization of random forest with sparrow search algorithm.

The objective of EEG-based emotion recognition is to classify emotions by decoding signals, with potential applications in the fields of artificial intelligence and bioinformatics. Cross-subject emotion recognition is more difficult than intra-subject emotion recognition. The poor adaptability of classification model parameters is a significant factor of low accuracy in cross-subject emotion recognition. We propose a model of a dynamically optimized Random Forest based on the Sparrow Search Algorithm (SSA-RF). The decision trees number (DTN) and the leave minimum number (LMN) of the RF are dynamically optimized by the SSA. 12 features are used to construct feature combinations for selecting the optimal feature combination. DEAP and SEED datasets are employed for testing the performance of SSA-RF. The experimental results show that the accuracy of binary classification is 76.81% on DEAP, and the accuracy of triple classification is 75.96% on SEED based on SSA-RF, which are both higher than that of traditional RF. This study provides new insights for the development of cross-subject emotion recognition, and has significant theoretical value.

CiABL: Completeness-induced Adaptative Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals

CiABL: Completeness-induced Adaptative Broad Learning for Cross-Subject Emotion Recognition with EEG and Eye Movement Signals

Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration

Objective. Due to individual differences, it is greatly challenging to realize the multiple types of emotion identification across subjects. Approach. In this research, a hierarchical feature optimization method is proposed in order to represent emotional states effectively based on peripheral physiological signals. Firstly, sparse learning combined with binary search is employed to achieve feature selection of single signals. Then an improved fast correlation-based filter is proposed to implement fusion optimization of multi-channel signal features. Aiming at overcoming the limitations of the support vector machine (SVM), which uses a single kernel function to make decisions, the multi-kernel function collaboration strategy is proposed to improve the classification performance of SVM. Main results. The effectiveness of the proposed method is verified on the DEAP dataset. Experimental results show that the proposed method presents a competitive performance for four cross-subject types of emotion identification with an accuracy of 84% (group 1) and 85.07% (group 2). Significance. The proposed model with hierarchical feature optimization and SVM with multi-kernel function collaboration demonstrates superior emotion recognition accuracy compared to state-of-the-art techniques. In addition, the analysis based on DEAP dataset composition characteristics presents a novel perspective to explore the emotion recognition issue more objectively and comprehensively.

Functional connectivity-enhanced feature-grouped attention network for cross-subject EEG emotion recognition

Electroencephalogram (EEG)-based automatic emotion recognition technologies are gaining significant attention and have become crucial in the field of brain–computer interfaces (BCIs). In particular, deep learning methods have been widely used in emotion recognition in recent years. However, most existing methods generally tend to focus on EEG spatiotemporal information, which ignores the potential relationships between brain activity signals and the differences in functional connectivity under different emotions. Here, we raise a functional connectivity-enhanced feature-grouped attention network (FC-FAN) for cross-subject emotion recognition. The FC-FAN model developed is a dual-input model. One input consists of the differential entropy data derived from the original EEG signals, while the other input comprises the functional connectivity data obtained through the calculation of the phase synchronization index. Then the primary EEG features of the two groups’ input data are extracted through two specific residual blocks. Next, the designed time-series feature grouped attention module (TFGAM) and functional connectivity feature grouped attention model (F2GAM) are utilized to mark interested information or suppress uninterested features for the two groups’ features, respectively. Finally, generated information interacts through a fusion operator. The designed framework could not only sufficiently learn the spatiotemporal features of EEG signals but also clearly analyze nonlinear correlations between electrode signals. Comprehensive tests confirm that the FC-FAN has an excellent effect on subject-independent emotion recognition tasks.

DA-CapsNet: A multi-branch capsule network based on adversarial domain adaption for cross-subject EEG emotion recognition

Due to inter-individual variances, cross-subject electroencephalogram (EEG)-based emotion recognition is a challenging task. In this paper, we construct a multi-branch Capsule network (named DA-CapsNet) based on domain adaptation to improve the performance of cross-subject EEG emotion recognition. To fully capture the various intensity characteristics of a single emotion, firstly, DA-CapsNet decomposes the source and the target domain EEG signals into four frequency bands and homomorphically groups the data in each band, and then extracts the differential entropy (DE) features for each group separately. Taking into account the spatial arrangement of the electrodes, the DE features are mapped into a two-dimensional matrix to form a homomorphic difference cube sequence (HDCS). Second, to enhance the feature information of the same emotion and accelerate the run efficiency of the network, a parallel structured multi-branch primary Capsual network (CapsNet) is constructed in this paper. The multi-branch primary CapsNet can effectively extract the aforementioned sequence discriminative features and fuse them as the input features of the capsule emotion classifier. Finally, to lessen inter-domain distribution discrepancies, we brought adversarial domain adaptation to improve the performance of cross-subject emotion recognition. Numerous tests are run on the three public datasets of EEG, and the results show that the proposed algorithm in this paper works well.

Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing.

Joint EEG Feature Transfer and Semisupervised Cross-Subject Emotion Recognition

Due to the weak and non-stationary properties, Electroencephalogram (EEG) data presents significant individual differences. To align data distributions of different subjects, transfer learning showed promising performance in cross-subject EEG emotion recognition. However, most of the existing models sequentially learned the domain-invariant features and estimated the target domain label information. Such a two-stage strategy breaks the inner connections of both processes, inevitably causing the sub-optimality. In this paper, we propose a Joint EEG feature Transfer and Semi-supervised cross-subject emotion Recognition (JTSR) model in which the shared subspace projection matrix and target label are jointly optimized towards the optimum. Extensive experiments are conducted on SEED-IV and SEED, and the results show that 1) the emotion recognition performance is significantly enhanced by the joint learning mode, and 2) the spatial-frequency activation patterns of critical EEG frequency bands and brain regions in cross-subject emotion expression are quantitatively identified by analyzing the learned shared subspace.

Learning a robust unified domain adaptation framework for cross-subject EEG-based emotion recognition

Over the last few years, unsupervised domain adaptation (UDA) based on deep learning has emerged as a solution to build cross-subject emotion recognition models from Electroencephalogram (EEG) signals, aligning the subject distributions within a latent feature space. However, most reported works have a common intrinsic limitation: the subject distribution alignment is coarse-grained, but not all of the feature space is shared between subjects. In this paper, we propose a robust unified domain adaptation framework, named Multi-source Feature Alignment and Label Rectification (MFA-LR), which performs a fine-grained domain alignment at subject and class levels, while inter-class separation and robustness against input perturbations are encouraged in coarse grain. As a complementary step, a pseudo-labeling correction procedure is used to rectify mislabeled target samples. Our proposal was assessed over two public datasets, SEED and SEED-IV, on each of the three available sessions, using leave-one-subject-out cross-validation. Experimental results show an accuracy performance of up to 89.11 ± 07.72% and 74.99 ± 12.10% for the best session on SEED and SEED-IV, as well as an average accuracy of 85.27% and 69.58% on all three sessions, outperforming state-of-the-art results.

Study on Driver Cross-Subject Emotion Recognition Based on Raw Multi-Channels EEG Data

In our life, emotions often have a profound impact on human behavior, especially for drivers, as negative emotions can increase the risk of traffic accidents. As such, it is imperative to accurately discern the emotional states of drivers in order to preemptively address and mitigate any negative emotions that may otherwise manifest and compromise driving behavior. In contrast to many current studies that rely on complex and deep neural network models to achieve high accuracy, this research aims to explore the potential of achieving high recognition accuracy using shallow neural networks through restructuring the structure and dimensions of the data. In this study, we propose an end-to-end convolutional neural network (CNN) model called simply ameliorated CNN (SACNN) to address the issue of low accuracy in cross-subject emotion recognition. We extracted features and converted dimensions of EEG signals from the SEED dataset from the BCMI Laboratory to construct 62-dimensional data, and obtained the optimal model configuration through ablation experiments. To further improve recognition accuracy, we selected the top 10 channels with the highest accuracy by separately training the EEG data of each of the 62 channels. The results showed that the SACNN model achieved an accuracy of 88.16% based on raw cross-subject data, and an accuracy of 91.85% based on EEG channel data from the top 10 channels. In addition, we explored the impact of the position of the BN and dropout layers on the model through experiments, and found that a targeted shallow CNN model performed better than deeper and larger perceptual field CNN models. Furthermore, we discuss herein the future issues and challenges of driver emotion recognition in promising smart city applications.

STGATE: Spatial-temporal graph attention network with a transformer encoder for EEG-based emotion recognition.

Electroencephalogram (EEG) is a crucial and widely utilized technique in neuroscience research. In this paper, we introduce a novel graph neural network called the spatial-temporal graph attention network with a transformer encoder (STGATE) to learn graph representations of emotion EEG signals and improve emotion recognition performance. In STGATE, a transformer-encoder is applied for capturing time-frequency features which are fed into a spatial-temporal graph attention for emotion classification. Using a dynamic adjacency matrix, the proposed STGATE adaptively learns intrinsic connections between different EEG channels. To evaluate the cross-subject emotion recognition performance, leave-one-subject-out experiments are carried out on three public emotion recognition datasets, i.e., SEED, SEED-IV, and DREAMER. The proposed STGATE model achieved a state-of-the-art EEG-based emotion recognition performance accuracy of 90.37% in SEED, 76.43% in SEED-IV, and 76.35% in DREAMER dataset, respectively. The experiments demonstrated the effectiveness of the proposed STGATE model for cross-subject EEG emotion recognition and its potential for graph-based neuroscience research.