Affective Brain-computer Interface Research Articles

Effective Emotion Recognition by Learning Discriminative Graph Topologies in EEG Brain Networks.

Multichannel electroencephalogram (EEG) is an array signal that represents brain neural networks and can be applied to characterize information propagation patterns for different emotional states. To reveal these inherent spatial graph features and increase the stability of emotion recognition, we propose an effective emotion recognition model that performs multicategory emotion recognition with multiple emotion-related spatial network topology patterns (MESNPs) by learning discriminative graph topologies in EEG brain networks. To evaluate the performance of our proposed MESNP model, we conducted single-subject and multisubject four-class classification experiments on two public datasets, MAHNOB-HCI and DEAP. Compared with existing feature extraction methods, the MESNP model significantly enhances the multiclass emotional classification performance in the single-subject and multisubject conditions. To evaluate the online version of the proposed MESNP model, we designed an online emotion monitoring system. We recruited 14 participants to conduct the online emotion decoding experiments. The average online experimental accuracy of the 14 participants was 84.56%, indicating that our model can be applied in affective brain-computer interface (aBCI) systems. The offline and online experimental results demonstrate that the proposed MESNP model effectively captures discriminative graph topology patterns and significantly improves emotion classification performance. Moreover, the proposed MESNP model provides a new scheme for extracting features from strongly coupled array signals.

Analysis and recognition of a novel experimental paradigm for musical emotion brain-computer interface

Musical emotions have received increasing attention over the years. To better recognize the emotions by brain-computer interface (BCI), the random music-playing and sequential music-playing experimental paradigms are proposed and compared in this paper. Two experimental paradigms consist of three positive pieces, three neutral pieces and three negative pieces of music. Ten subjects participate in two experimental paradigms. The features of electroencephalography (EEG) signals are firstly analyzed in the time, frequency and spatial domains. To improve the effect of emotion recognition, a recognition model is proposed with the optimal channels selecting by Pearson’s correlation coefficient, and the feature fusion combining differential entropy and wavelet packet energy. According to the analysis results, the features of sequential music-playing experimental paradigm are more different among three emotions. The classification results of sequential music-playing experimental paradigm are also better, and its average results of positive, neutral and negative emotions are 78.53%, 72.81% and 77.35%, respectively. The more obvious the changes of EEG induced by the emotions, the higher the classification accuracy will be. After analyzing two experimental paradigms, a better way for music to induce the emotions can be explored. Therefore, our research offers a novel perspective on affective BCIs.

FBN-TCN: Temporal convolutional neural network based on spatial domain fusion brain networks for affective brain–computer interfaces

Background and objective: Electroencephalogram (EEG) emotion recognition is increasingly popular for visualizing and quantifying brain networks, with potential applications in medical cognition and human–computer interaction. However, single brain space is limited, and multi-channel recognition poses challenges with calculation volume and information redundancy. Therefore, this paper focuses on studying the affective brain–computer interface (aBCI) and critical lead acquisition of the fusion brain network. Method: This paper employs two correlation algorithms to build functional and causal networks. A fusion brain network (FBN) was proposed to capture emotional information in the spatial domain. Additionally, a deep network recognition model (FBN-TCN) is introduced to capture the dynamic changes of spatial domain emotional information over time. A critical channel combination with high emotional correlation among multiple channels is obtained by the brain causal networks. All proposed models were performed on the SEED dataset through subject-dependent and subject-independent experiments. Results: Experimental results demonstrate that FBN can cover more emotional information to improve emotional recognition. The critical channel aBCI achieves comparable recognition performance to the full-channel aBCI, with recognition results of 95.51%/3.15% and 96.34%/1.62% in the Total band, respectively. The generalization and effectiveness of the model were verified through ablation experiments, which involved subject-dependent and subject-independent emotion recognition of DEAP, and emotion recognition of SEED on model block transformation. Conclusion and significance: This paper can simplify the electrode cap device and maximizes the utilization of EEG spatial domain emotional information. It is highly significant for enhancing the performance of aBCI in human–computer interaction.

PR-PL: A Novel Prototypical Representation Based Pairwise Learning Framework for Emotion Recognition Using EEG Signals

Affective brain-computer interface based on electroencephalography (EEG) is an important branch in the field of affective computing. However, the individual differences in EEG emotional data and the noisy labeling problem in the subjective feedback seriously limit the effectiveness and generalizability of existing models. To tackle these two critical issues, we propose a novel transfer learning framework with Prototypical Representation based Pairwise Learning ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PR-PL</i> ). The discriminative and generalized EEG features are learned for emotion revealing across individuals and the emotion recognition task is formulated as pairwise learning for improving the model tolerance to the noisy labels. More specifically, a prototypical learning is developed to encode the inherent emotion-related semantic structure of EEG data and align the individuals' EEG features to a shared common feature space under consideration of the feature separability of both source and target domains. Based on the aligned feature representations, pairwise learning with an adaptive pseudo labeling method is introduced to encode the proximity relationships among samples and alleviate the label noises effect on modeling. Extensive results on two benchmark databases (SEED and SEED-IV) under four different cross-validation evaluation protocols validate the model reliability and stability across subjects and sessions. Compared to the literature, the average enhancement of emotion recognition across four different evaluation protocols is 2.04% (SEED) and 2.58% (SEED-IV). The source code is available at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/KAZABANA/PR-PL</uri></i> .

An Affective Brain-Computer Interface Based on a Transfer Learning Method

An Affective Brain-Computer Interface Based on a Transfer Learning Method

Cross-Day Data Diversity Improves Inter-Individual Emotion Commonality of Spatio-Spectral EEG Signatures Using Independent Component Analysis

Electroencephalogram (EEG) variability poses a great challenge to the affective brain-computer interface (aBCI) for practical applications. Most aBCI frameworks have been demonstrated successfully but deliberated on single-day data, which can be realistically susceptible to psychophysiological changes and further hinder the exploration of inter-individual EEG commonality. This study proposes a multiple-day scenario that learns exclusively from diverse EEG correlates of emotional responses on different days (i.e., enriched data diversity) by using a unified independent components analysis framework. Given an eight-day dataset of 10 subjects ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., 80 sessions), the results demonstrated that the multiple-day scenario intensified the inter-subject emotion commonality ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., the percentage of subjects with the same signature) to a certain extent when considering sufficient cross-day sessions, whereas the most commonly adopted single-day analysis (i.e., diversity-confined) led to session-dependent inferior outcomes. Given the best case, the emotional valence dimension was associated with relatively reproducible frontal beta, central midline gamma, and occipital beta modulations with 30%–40% subject commonality, whereas the arousal counterpart suffered more substantially from EEG variability and barely returned representative signatures. These results suggest that EEG signature representation may be substantially compromised by limited data diversity, impeding the efficacy and generalizability of the aBCI model in real-life settings.

Analysis of emotion recognition based on brain-computer interface technology

The lifetime prevalence of social anxiety disorder is pretty high, and Brain-computer Interface (BCI) technology has become the main solution to the above problems. In order to solve this problem, this paper firstly introduced the sentiment classification and the theoretical basis of BCI emotion recognition, and bring up the specific mechanism and detailed process of emotion recognition. Secondly the selection of data sets and the detailed process of EEG signal analysis are mainly discussed, including pre-processing, feature extraction, classification of signal recognition, accuracy analysis and so on. Thirdly, the paper summarized some examples of current representative applications and the existing problems, improvements and breakthroughs in recent years. In the end, the future development trend of affective brain-computer interface is prospected. Although the brain computer interface (BCI) emotion recognition is still an immature technology in the initial stage and has so many challenging problems, but this paper wish and believe it can have a brilliant development prospect.

Possibilistic distribution distance metric: a robust domain adaptation learning method.

The affective Brain-Computer Interface (aBCI) systems, which achieve predictions for individual subjects through training on multiple subjects, often cannot achieve satisfactory results due to the differences in Electroencephalogram (EEG) patterns between subjects. One tried to use Subject-specific classifiers, but there was a lack of sufficient labeled data. To solve this problem, Domain Adaptation (DA) has recently received widespread attention in the field of EEG-based emotion recognition. Domain adaptation (DA) learning aims to solve the problem of inconsistent distributions between training and test datasets and has received extensive attention. Most existing methods use Maximum Mean Discrepancy (MMD) or its variants to minimize the problem of domain distribution inconsistency. However, noisy data in the domain can lead to significant drift in domain means, which can affect the adaptability performance of learning methods based on MMD and its variants to some extent. Therefore, we propose a robust domain adaptation learning method with possibilistic distribution distance measure. Firstly, the traditional MMD criterion is transformed into a novel possibilistic clustering model to weaken the influence of noisy data, thereby constructing a robust possibilistic distribution distance metric (P-DDM) criterion. Then the robust effectiveness of domain distribution alignment is further improved by a fuzzy entropy regularization term. The proposed P-DDM is in theory proved which be an upper bound of the traditional distribution distance measure method MMD criterion under certain conditions. Therefore, minimizing P-DDM can effectively optimize the MMD objective. Secondly, based on the P-DDM criterion, a robust domain adaptation classifier based on P-DDM (C-PDDM) is proposed, which adopts the Laplacian matrix to preserve the geometric consistency of instances in the source domain and target domain for improving the label propagation performance. At the same time, by maximizing the use of source domain discriminative information to minimize domain discrimination error, the generalization performance of the learning model is further improved. Finally, a large number of experiments and analyses on multiple EEG datasets (i.e., SEED and SEED-IV) show that the proposed method has superior or comparable robustness performance (i.e., has increased by around 10%) in most cases.

Affective Brain–Computer Interfaces (aBCIs): A Tutorial

A brain–computer interface (BCI) enables a user to communicate directly with a computer using only the central nervous system. An affective BCI (aBCI) monitors and/or regulates the emotional state of the brain, which could facilitate human cognition, communication, decision-making, and health. The last decade has witnessed rapid progress in aBCI research and applications, but there does not exist a comprehensive and up-to-date tutorial on aBCIs. This tutorial fills the gap. It introduces first the basic concepts of BCIs and then, in detail, the individual components in a closed-loop aBCI system, including signal acquisition, signal processing, feature extraction, emotion recognition, and brain stimulation. Next, it describes three representative applications of aBCIs, i.e., cognitive workload recognition, fatigue estimation, and depression diagnosis and treatment. Several challenges and opportunities in aBCI research and applications, including brain signal acquisition, emotion labeling, diversity and size of aBCI datasets, algorithm comparison, negative transfer in emotion recognition, and privacy protection and security of aBCIs, are also explained.

An Efficacy Analysis of Affect Representation Methodologies Used in EEG-Based Emotion Recognition System: An Unsupervised Approach

Brain-Computer Interface (BCI)-based affect recognition in humans utilizing physiological signals is typically guided by a supervised method that relies heavily on the presence of appropriate affect representations. In this paper, we assess the effectiveness of contemporary affect representation methodologies for establishing ground truth for reliable affective BCI systems. On the DEAP dataset, two experiments were conducted to compare the observed emotion clusters based on subjective ratings with the clustering of corresponding physiological responses based on EEG characteristics. The outcomes of the two investigations are inconsistent with one another. The findings suggest the subjective use of emotion categories as opposed to their general application in establishing ground truth for supervised approaches and must encourage researchers to examine affect representation techniques critically before considering them as concrete representations of a person’s affective state.

Transformer-based ensemble deep learning model for EEG-based emotion recognition

Emotion recognition is one of the most important research directions in the field of brain–computer interface (BCI). However, to conduct electroencephalogram (EEG)-based emotion recognition, there exist difficulties regarding EEG signal processing; moreover, the performance of classification models in this regard is restricted. To counter these issues, the 2022 World Robot Contest successfully held an affective BCI competition, thus promoting the innovation of EEG-based emotion recognition. In this paper, we propose the Transformer-based ensemble (TBEM) deep learning model. TBEM comprises two models: a pure convolutional neural network (CNN) model and a cascaded CNN-Transformer hybrid model. The proposed model won the abovementioned affective BCI competition’s final championship in the 2022 World Robot Contest, demonstrating the effectiveness of the proposed TBEM deep learning model for EEG-based emotion recognition.

Tagging Continuous Labels for EEG-based Emotion Classification.

EEG-based emotion classification has long been a critical task in the field of affective brain-computer interface (aBCI). The majority of leading researches construct supervised learning models based on labeled datasets. Several datasets have been released, including different kinds of emotions while utilizing various forms of stimulus materials. However, they adopt discrete labeling methods, in which the EEG data collected during the same stimulus material are given a same label. These methods neglect the fact that emotion changes continuously, and mislabeled data possibly exist. The imprecision of discrete labels may hinder the progress of emotion classification in concerned works. Therefore, we develop an efficient system in this paper to support continuous labeling by giving each sample a unique label, and construct a continuously labeled EEG emotion dataset. Using our dataset with continuous labels, we demonstrate the superiority of continuous labeling in emotion classification through experiments on several classification models. We further utilize the continuous labels to identify the EEG features under induced and non-induced emotions in both our dataset and a public dataset. Our experimental results reveal the learnability and generality of the relation between the EEG features and their continuous labels.

SSTM-IS: simplified STM method based on instance selection for real-time EEG emotion recognition.

EEG signals can non-invasively monitor the brain activities and have been widely used in brain-computer interfaces (BCI). One of the research areas is to recognize emotions objectively through EEG. In fact, the emotion of people changes over time, however, most of the existing affective BCIs process data and recognize emotions offline, and thus cannot be applied to real-time emotion recognition. In order to solve this problem, we introduce the instance selection strategy into transfer learning and propose a simplified style transfer mapping algorithm. In the proposed method, the informative instances are firstly selected from the source domain data, and then the update strategy of hyperparameters is also simplified for style transfer mapping, making the model training more quickly and accurately for a new subject. To verify the effectiveness of our algorithm, we carry out the experiments on SEED, SEED-IV and the offline dataset collected by ourselves, and achieve the recognition accuracies up to 86.78%, 82.55% and 77.68% in computing time of 7s, 4s and 10s, respectively. Furthermore, we also develop a real-time emotion recognition system which integrates the modules of EEG signal acquisition, data processing, emotion recognition and result visualization. Both the results of offline and online experiments show that the proposed algorithm can accurately recognize emotions in a short time, meeting the needs of real-time emotion recognition applications.

Stylized faces enhance ERP features used for the detection of emotional responses.

For their ease of accessibility and low cost, current Brain-Computer Interfaces (BCI) used to detect subjective emotional and affective states rely largely on electroencephalographic (EEG) signals. Public datasets are available for researchers to design models for affect detection from EEG. However, few designs focus on optimally exploiting the nature of the stimulus elicitation to improve accuracy. The RSVP protocol is used in this experiment to present human faces of emotion to 28 participants while EEG was measured. We found that artificially enhanced human faces with exaggerated, cartoonish visual features significantly improve some commonly used neural correlates of emotion as measured by event-related potentials (ERPs). These images elicit an enhanced N170 component, well known to relate to the facial visual encoding process. Our findings suggest that the study of emotion elicitation could exploit consistent, high detail, AI generated stimuli transformations to study the characteristics of electrical brain activity related to visual affective stimuli. Furthermore, this specific result might be useful in the context of affective BCI design, where a higher accuracy in affect decoding from EEG can improve the experience of a user.

Cross-subject EEG emotion recognition using multi-source domain manifold feature selection

Recent researches on emotion recognition suggests that domain adaptation, a form of transfer learning, has the capability to solve the cross-subject problem in Affective brain-computer interface (aBCI) field. However, traditional domain adaptation methods perform single to single domain transfer or simply merge different source domains into a larger domain to realize the transfer of knowledge, resulting in negative transfer. In this study, a multi-source transfer learning framework was proposed to promote the performance of multi-source electroencephalogram (EEG) emotion recognition. The method first used the data distribution similarity ranking (DDSA) method to select the appropriate source domain for each target domain off-line, and reduced data drift between domains through manifold feature mapping on Grassmann manifold. Meanwhile, the minimum redundancy maximum correlation algorithm (mRMR) was employed to select more representative manifold features and minimized the conditional distribution and marginal distribution of the manifold features, and then learned the domain-invariant classifier by summarizing structural risk minimization (SRM). Finally, the weighted fusion criterion was applied to further improve recognition performance. We compared our method with several state-of-the-art domain adaptation techniques using the SEED and DEAP dataset. Results showed that, compared with the conventional MEDA algorithm, the recognition accuracy of our proposed algorithm on SEED and DEAP dataset were improved by 6.74% and 5.34%, respectively. Besides, compared with TCA, JDA, and other state-of-the-art algorithms, the performance of our proposed method was also improved with the best average accuracy of 86.59% on SEED and 64.40% on DEAP. Our results demonstrated that the proposed multi-source transfer learning framework is more effective and feasible than other state-of-the-art methods in recognizing different emotions by solving the cross-subject problem.

A survey on encoding and decoding technology of non-invasive brain-computer interface

A survey on encoding and decoding technology of non-invasive brain-computer interface

Cross-Subject Emotion Recognition Brain-Computer Interface Based on fNIRS and DBJNet.

Functional near-infrared spectroscopy (fNIRS) is a noninvasive brain imaging technique that has gradually been applied in emotion recognition research due to its advantages of high spatial resolution, real time, and convenience. However, the current research on emotion recognition based on fNIRS is mainly limited to within-subject, and there is a lack of related work on emotion recognition across subjects. Therefore, in this paper, we designed an emotion evoking experiment with videos as stimuli and constructed the fNIRS emotion recognition database. On this basis, deep learning technology was introduced for the first time, and a dual-branch joint network (DBJNet) was constructed, creating the ability to generalize the model to new participants. The decoding performance obtained by the proposed model shows that fNIRS can effectively distinguish positive versus neutral versus negative emotions (accuracy is 74.8%, F1 score is 72.9%), and the decoding performance on the 2-category emotion recognition task of distinguishing positive versus neutral (accuracy is 89.5%, F1 score is 88.3%), negative versus neutral (accuracy is 91.7%, F1 score is 91.1%) proved fNIRS has a powerful ability to decode emotions. Furthermore, the results of the ablation study of the model structure demonstrate that the joint convolutional neural network branch and the statistical branch achieve the highest decoding performance. The work in this paper is expected to facilitate the development of fNIRS affective brain-computer interface.

Unraveling the Development of an Algorithm for Recognizing Primary Emotions Through Electroencephalography.

The large range of potential applications, not only for patients but also for healthy people, that could be achieved by affective brain-computer interface (aBCI) makes more latent the necessity of finding a commonly accepted protocol for real-time EEG-based emotion recognition. Based on wavelet package for spectral feature extraction, attending to the nature of the EEG signal, we have specified some of the main parameters needed for the implementation of robust positive and negative emotion classification. Twelve seconds has resulted as the most appropriate sliding window size; from that, a set of 20 target frequency-location variables have been proposed as the most relevant features that carry the emotional information. Lastly, QDA and KNN classifiers and population rating criterion for stimuli labeling have been suggested as the most suitable approaches for EEG-based emotion recognition. The proposed model reached a mean accuracy of 98% (s.d. 1.4) and 98.96% (s.d. 1.28) in a subject-dependent (SD) approach for QDA and KNN classifier, respectively. This new model represents a step forward towards real-time classification. Moreover, new insights regarding subject-independent (SI) approximation have been discussed, although the results were not conclusive.

Joint Feature Adaptation and Graph Adaptive Label Propagation for Cross-Subject Emotion Recognition From EEG Signals

Though Electroencephalogram (EEG) could objectively reflect emotional states of our human beings, its weak, non-stationary, and low signal-to-noise properties easily cause the individual differences. To enhance the universality of affective brain-computer interface systems, transfer learning has been widely used to alleviate the data distribution discrepancies among subjects. However, most of existing approaches focused mainly on the domain-invariant feature learning, which is not unified together with the recognition process. In this paper, we propose a joint feature adaptation and graph adaptive label propagation model (JAGP) for cross-subject emotion recognition from EEG signals, which seamlessly unifies the three components of domain-invariant feature learning, emotional state estimation and optimal graph learning together into a single objective. We conduct extensive experiments on two benchmark SEED_IV and SEED_V data sets and the results reveal that 1) the recognition performance is greatly improved, indicating the effectiveness of the triple unification mode; 2) the emotion metric of EEG samples are gradually optimized during model training, showing the necessity of optimal graph learning, and 3) the projection matrix-induced feature importance is obtained based on which the critical frequency bands and brain regions corresponding to subject-invariant features can be automatically identified, demonstrating the superiority of the learned shared subspace.

Classification of emotions using EEG activity associated with different areas of the brain

• For emotion classification , emotional brain activity should be analyzed for each brain lobe and hemisphere. • The four lobes of the brain give different emotion classification accuracies as they generate distinct emotional activity. • The hemispheric regions of brain give different classification accuracies as they generate distinct emotional activity. • Fusion of EEG and peripheral signals results in better emotion classification accuracy than using EEG signals alone. It is important that interactive Affective Brain-Computer Interface (aBCI) applications support some degree of emotional intelligence. Many of the previous works that have explored the use of the Electroencephalogram (EEG) for the purpose of emotion recognition have focused on using data coming from the entire brain. However, the emotional activity in humans is not constant across the brain but varies from one region to another. Therefore, this paper aims to classify human emotions into high/low arousal and high/low valence using EEG and other physiological data from three datasets - DEAP, DREAMER and DASPS, showing the differences in the classification accuracies for different regions of the brain such as frontal lobe, parietal lobe , temporal lobe, occipital lobe, left frontal region, right frontal region, left parietal-temporal-occipital region and the right parietal-temporal-occipital region. The classification experiments are performed using the 1D Convolutional LSTM network and its performance is then compared with two baseline machine learning (ML) algorithms K-Nearest Neighbor and Random Forest. The experimental results show the variation of classification accuracies from one brain region to another. Moreover, they also indicate that the fusion of EEG and peripheral data produces higher emotion classification accuracy as compared to using the EEG and peripheral data separately.