Valence Classification Research Articles

Emotion recognition from multichannel EEG signals based on low-rank subspace self-representation features

In recent years, emotion recognition based on electroencephalogram (EEG) has become the research focus in human–computer interaction (HCI), but deficiencies in EEG feature extraction and noise suppression are still challenging. In this paper, a novel robust low-rank subspace self-representation (RLSR) of EEG is developed for emotion recognition. Instead of using classical time–frequency EEG feature, the data-driven based EEG self-representation in low-rank subspace is extracted for emotion characterization. The Robust Principal Component Analysis (RPCA) is incorporated to separate the noise part in the process of solving self-representation. The accuracy and robustness of the result are improved because of the superior features and noise suppression. To fully exploit the effective knowledge of different EEG frequency bands, the Tucker decomposition based data dimensionality reduction is introduced. Experiments conducted on the public dataset DEAP reveal that the average accuracies of the proposed method can reach to 93.04 % and 93.13 % for binary classification of valence and arousal, respectively. The average accuracy reaches to 88.82 % of four-class classification.

End-to-end multimodal emotion recognition based on facial expressions and remote photoplethysmography signals.

Emotion is a complex physiological phenomenon, and a single modality may be insufficient for accurately determining human emotional states. This paper proposes an end-to-end multimodal emotion recognition method based on facial expressions and non-contact physiological signals. Facial expression features and remote photoplethysmography (rPPG) signals are extracted from facial video data, and a transformer-based cross-modal attention mechanism (TCMA) is used to learn the correlation between the two modalities. The results show that the accuracy of emotion recognition can be slightly improved by combining facial expressions with accurate rPPG signals. The performance is further improved with the use of TCMA, for which the binary classification accuracy of valence and arousal is 91.11% and 90.00%, respectively. Additionally, when experiments are conducted using the whole dataset, an increased accuracy of 7.31% and 4.23% for the binary classification of valence and arousal, and an improved accuracy of 5.36% for the four classifications of valence-arousal are achieved when TCMA is used in modal fusion, compared to using only facial expression modality, which fully demonstrates the effectiveness and robustness of TCMA. This method makes it possible to realize multimodal emotion recognition of facial expressions and contactless physiological signals in reality.

Efficient Decoding of Affective States from Video-elicited EEG Signals: An Empirical Investigation

Affect decoding through brain-computer interfacing (BCI) holds great potential to capture users’ feelings and emotional responses via non-invasive electroencephalogram (EEG) sensing. Yet, little research has been conducted to understand efficient decoding when users are exposed to dynamic audiovisual contents. In this regard, we study EEG-based affect decoding from videos in arousal and valence classification tasks, considering the impact of signal length, window size for feature extraction, and frequency bands. We train both classic Machine Learning models (SVMs and k -NNs) and modern Deep Learning models (FCNNs and GTNs). Our results show that: (1) affect can be effectively decoded using less than 1 minute of EEG signal; (2) temporal windows of 6 and 10 seconds provide the best classification performance for classic Machine Learning models but Deep Learning models benefit from much shorter windows of 2 seconds; and (3) any model trained on the Beta band alone achieves similar (sometimes better) performance than when trained on all frequency bands. Taken together, our results indicate that affect decoding can work in more realistic conditions than currently assumed, thus becoming a viable technology for creating better interfaces and user models.

EEG emotion recognition based on the TimesNet fusion model

In recent years, emotion recognition based on electroencephalogram (EEG) has become an important research field. This paper proposes an innovative multi-scale emotion recognition method (MS-ERM), which is based on a deep learning model. First, we divide the EEG signal into time windows of 0.5 s in different frequency bands to extract the differential entropy feature and embed the feature into the brain electrode map to express spatial information. Then, the features of each segment are used as input to the new deep learning model (MS-TimesNet). The model combines multi-scale convolution and TimesNet network to effectively extract dynamic time features, cross-channel spatial features, and complex time features in 2D space. Through extensive tests on the DEAP dataset, we prove that this method is superior to existing methods in terms of sentiment classification performance. In the arousal and valence classification, the average classification accuracy of subject-dependent tests reached 91.31% and 90.45%, respectively, while in subject-independent tests, the average classification accuracy was 86.66% and 85.40%, respectively. Code is available at this repository: https://github.com/hyao0827/MS-ERM.

Cerebral asymmetry representation learning-based deep subdomain adaptation network for electroencephalogram-based emotion recognition.

Objective.Extracting discriminative spatial information from multiple electrodes is a crucial and challenging problem for electroencephalogram (EEG)-based emotion recognition. Additionally, the domain shift caused by the individual differences degrades the performance of cross-subject EEG classification.Approach.To deal with the above problems, we propose the cerebral asymmetry representation learning-based deep subdomain adaptation network (CARL-DSAN) to enhance cross-subject EEG-based emotion recognition. Specifically, the CARL module is inspired by the neuroscience findings that asymmetrical activations of the left and right brain hemispheres occur during cognitive and affective processes. In the CARL module, we introduce a novel two-step strategy for extracting discriminative features through intra-hemisphere spatial learning and asymmetry representation learning. Moreover, the transformer encoders within the CARL module can emphasize the contributive electrodes and electrode pairs. Subsequently, the DSAN module, known for its superior performance over global domain adaptation, is adopted to mitigate domain shift and further improve the cross-subject performance by aligning relevant subdomains that share the same class samples.Main Results.To validate the effectiveness of the CARL-DSAN, we conduct subject-independent experiments on the DEAP database, achieving accuracies of 68.67% and 67.11% for arousal and valence classification, respectively, and corresponding accuracies of 67.70% and 67.18% on the MAHNOB-HCI database.Significance.The results demonstrate that CARL-DSAN can achieve an outstanding cross-subject performance in both arousal and valence classification.

Group Synchrony for Emotion Recognition Using Physiological Signals

During group interactions, we react and modulate our emotions and behaviour to the group through phenomena including emotion contagion and physiological synchrony. Previous work on emotion recognition through video/image has shown that group context information improves the classification performance. However, when using physiological data, literature mostly focuses on intrapersonal models that leave-out group information, while interpersonal models are unexplored. This paper introduces a new interpersonal Weighted Group Synchrony approach, which relies on Electrodermal Activity (EDA) and Heart-Rate Variability (HRV). We perform an analysis of synchrony metrics applied across diverse data representations (EDA and HRV morphology and features, recurrence plot, spectrogram), to identify which metrics and modalities better characterise physiological synchrony for emotion recognition. We explored two datasets (AMIGOS and K-EmoCon), covering different group sizes (4 vs dyad) and group-based activities (video-watching vs conversation). The experimental results show that integrating group information improves arousal and valence classification, across all datasets, with the exception of K-EmoCon on valence. The proposed method was able to attain mean M-F1 of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\approx$</tex-math></inline-formula> 72.15% arousal and 81.16% valence for AMIGOS, and M-F1 of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\approx$</tex-math></inline-formula> 52.63% arousal, 65.09% valence for K-EmoCon, surpassing previous work results for K-EmoCon on arousal, and providing a new baseline on AMIGOS for long-videos.

A spatiotemporal symmetrical transformer structure for EEG emotion recognition

In the field of brain-computer interface, automatic recognition of emotions based on electroencephalogram (EEG) signals is of great significance. At present, deep learning can deeply mine the information in the data, especially the convolutional neural network (CNN) and the recurrent neural network (LSTM), which has remarkably improved the accuracy in numerous fields, hence it is applied by researchers to EEG-based emotional identification research. Nonetheless, existing CNN and LSTM-based models still rely on data preprocessing and feature extraction. Furthermore, CNNs have limitations in perceiving global dependencies while LSTMs have problems such as vanishing gradients in the case of long sequences. In this paper, we proposed the Spatiotemporal Symmetric Transformer Model (STS-Transformer), an effective EEG emotion recognition model, to overcome the current problems. STS-Transformer is an end-to-end framework that directly recognizes emotion from raw EEG signals without data preprocessing and feature extraction. This method achieves 89.86% and 86.83% accuracy respectively in the binary classification of valence and arousal in the DEAP dataset; in the DREAMER dataset, the binary classification accuracy of valence and arousal is 85.09% and 82.32%. Therefore, our method exhibits remarkable advantages over other end-to-end models in similar studies.

MTDN: Learning Multiple Temporal Dynamics Representation for Emotional Valence Classification with EEG.

Emotion recognition from electroencephalogram (EEG) requires computational models to capture the crucial features of the emotional response to external stimulation. Spatial, spectral, and temporal information are relevant features for emotion recognition. However, learning temporal dynamics is a challenging task, and there is a lack of efficient approaches to capture such information. In this work, we present a deep learning framework called MTDN that is designed to capture spectral features with a filterbank module and to learn spatial features with a spatial convolution block. Multiple temporal dynamics are jointly learned with parallel long short-term memory (LSTM) embedding and self-attention modules. The LSTM module is used to embed the time segments, and then the self-attention is utilized to learn the temporal dynamics by intercorrelating every embedded time segment. Multiple temporal dynamics representations are then aggregated to form the final extracted features for classification. We experiment on a publicly available dataset, DEAP, to evaluate the performance of our proposed framework and compare MTDN with existing published results. The results demonstrate improvement over the current state-of-the-art methods on the valence dimension of the DEAP dataset.

EEG-based Emotion Recognition Using Sub-Band Time-Delay Correlations.

Emotion recognition is a challenging task with many potential applications in psychology, psychiatry, and human-computer interaction (HCI). The use of time-delay information in the controlled time-delay stability (cTDS) algorithm can help to capture the temporal dynamics of EEG signals, including sub-band information and bi-directional coupling that can aid in emotion recognition and identification of specific connectivity patterns between brain rhythms. Incorporating EEG frequency bands can be used to design better emotion recognition systems. This paper evaluates the cTDS algorithm for binary classification tasks of arousal and valence using EEG sub-band signals. This method achieved a high accuracy of 91.1% for arousal and 91.7% for valence based on one electrode recording site at Fp1. The cTDS algorithm is a promising approach to analyzing brain network interactions. It can be particularly applicable to arousal and valence classification tasks, especially within a complex, multimodal feature space associated with understanding psychiatric disorders and HCI applications.

Exploring the Relationship between EEG Features of Basic and Academic Emotions

This study aimed to explore the relationship between basic and academic emotions by analyzing their EEG patterns. Using MAHNOB-HCI (MH) and Academic Emotion (AE) datasets, we performed three experiments based on valence and discrete emotion models. Our analysis revealed no similarity between the valence of basic and academic emotion datasets. However, we found that three out of 84 features in the MH discrete emotion dataset had a statistically significant relationship with the AE frustration dataset, suggesting some commonality between basic and academic emotions, particularly in the case of frustration. We also used random forest (RF), multilayer perceptron (MLP), and support vector machine (SVM) models to validate our findings, with the RF model outperforming the others in terms of valence classification accuracy. Our study provides valuable insights into the relationship between basic and academic emotions and may inform future research in this area.

A Bi-Stream hybrid model with MLPBlocks and self-attention mechanism for EEG-based emotion recognition

Due to the instability and complex distribution of electroencephalography (EEG) signals and the great cross-subject variations, exploiting valuable and discriminative emotional information from EEG is still a significant challenging issue. In this paper, we propose Bi-Stream Multilayer Perceptron – Self-Attention Mixer (BiSMSM), a novel model for EEG-based emotion recognition. The proposed model consists of two streams: the spatial stream and the temporal stream. BiSMSM jointly captures the useful information from temporal, spatial, local and global angles, aiming to encode more discriminative features describing emotions. The spatial stream focuses on the spatial information, while the temporal stream concentrates on the correlation information in the time domain. The structures of the two streams are similar, either of which contains a Multilayer Perceptron (MLP) based module to extract the regional in-channel and cross-channel information. The MLP-based module is followed by a self-attention mechanism module to explore the global signal correlations. Finally, the subject-independent experiments on the public benchmark datasets DEAP and DREAMER have demonstrated the advantage of our model over the related advanced approaches. Specifically, BiSMSM obtains an accuracy of 63.10% for valence classification and 61.89% for aoursal classification on DEAP, and 61.88% for valence classification and 64.25% for arousal classification on DREAMER.

Unsupervised Cross-Corpus Speech Emotion Recognition Using a Multi-Source Cycle-GAN

Speech emotion recognition (SER) plays a crucial role in understanding user feelings when developing artificial intelligence services. However, the data mismatch and label distortion between the training (source) set and the testing (target) set significantly degrade the performances when developing the SER systems. Additionally, most emotion-related speech datasets are highly contextualized and limited in size. The manual annotation cost is often too high leading to an active investigation of unsupervised cross-corpus SER techniques. In this paper, we propose a framework in unsupervised cross-corpus emotion recognition using multi-source corpus in a data augmentation manner. We introduced Corpus-Aware Emotional CycleGAN (CAEmoCyGAN) including a corpus-aware attention mechanism to aggregate each source datasets to generate the synthetic target sample. We choose the widely used speech emotion corpora the IEMOCAP and the VAM as sources and the MSP-Podcast as the target. By generating synthetic target-aware samples to augment source datasets and by directly training on this augmented dataset, our proposed multi-source target-aware augmentation method outperforms other baseline models in activation and valence classification.

Topological EEG Nonlinear Dynamics Analysis for Emotion Recognition

Emotional recognition through exploring the electroencephalography (EEG) characteristics has been widely performed in recent studies. Nonlinear analysis and feature extraction methods for understanding the complex dynamical phenomena are associated with the EEG patterns of different emotions. The phase space reconstruction is a typical nonlinear technique to reveal the dynamics of the brain neural system. Recently, the topological data analysis (TDA) scheme has been used to explore the properties of space, which provides a powerful tool to think over the phase space. In this work, we proposed a topological EEG nonlinear dynamics analysis approach using the phase space reconstruction (PSR) technique to convert EEG time series into phase space, and the persistent homology tool explores the topological properties of the phase space. We perform the topological analysis of EEG signals in different rhythm bands to build emotion feature vectors, which shows high distinguishing ability. We evaluate the approach with two well-known benchmark datasets, the DEAP and DREAMER datasets. The recognition results achieved accuracies of 99.37% and 99.35% in arousal and valence classification tasks with DEAP, and 99.96%, 99.93%, and 99.95% in arousal, valence, and dominance classifications tasks with DREAMER, respectively. The performances are supposed to be outperformed current state-of-art approaches in DREAMER (improved by 1% to 10% depends on temporal length), while comparable to other related works evaluated in DEAP. The proposed work is the first investigation in the emotion recognition oriented EEG topological feature analysis, which brought a novel insight into the brain neural system nonlinear dynamics analysis and feature extraction.

STILN: A novel spatial-temporal information learning network for EEG-based emotion recognition

The spatial correlations and temporal contexts are indispensable in Electroencephalogram (EEG)-based emotion recognition. However, the learning of complex spatial correlations among several channels is a challenging problem. Besides, the temporal contexts learning is beneficial to emphasize the critical EEG frames because the subjects only reach the prospective emotion during part of stimuli. Hence, we propose a novel Spatial-Temporal Information Learning Network (STILN) to extract the discriminative features by capturing the spatial correlations and temporal contexts. Specifically, the generated 2D power topographic maps capture the dependencies among electrodes, and they are fed to the CNN-based spatial feature extraction network. Furthermore, Convolutional Block Attention Module (CBAM) recalibrates the weights of power topographic maps to emphasize the crucial brain regions and frequency bands. Meanwhile, Batch Normalizations (BNs) and Instance Normalizations (INs) are appropriately combined to relieve the individual differences. In the temporal contexts learning, we adopt the Bidirectional Long Short-Term Memory Network (Bi-LSTM) network to capture the dependencies among the EEG frames. To validate the effectiveness of the proposed method, subject-independent experiments are conducted on the public DEAP dataset. The proposed method has achieved the outstanding performance, and the accuracies of arousal and valence classification have reached 0.6831 and 0.6752 respectively.

CNN-LSTM for automatic emotion recognition using contactless photoplythesmographic signals

Numerous researchers in the field of automatic human emotion recognition have shown that physiological responses in humans are influenced by emotional changes. According to several previously conducted studies, the physiological data used are collected by sensors placed in the human body. In recent years, researchers have shown that the human heart rate can be estimated from human face videos. This technique is based on extracting the PPG signal by measuring the red–greenblue (RGB) color changes of the face of a person during the cardiac cycle. In this respect, a new framework based on contactless PPG signals for classifying human emotions is proposed in the present study. The experiment conducted in the present work used a widely popular emotional database, i.e. the MAHNOB-HCI database, to evaluate some PPG signal extraction methods. Regarding classification, a deep learning architecture, which combines a one-dimensional convolution neural network (1DCNN) and a long short-term memory (LSTM), after applying the normalization and signal segmentation steps, was adopted. It is important to emphasize that the method adopted here achieved a recognition rate of 73.33% and 60% for the binary classification of valence and arousal, respectively, with a signal segmentation of 4 s. It is worth recalling that the present paper aims primarily to propose a new study approach on automatic emotion recognition using deep learning techniques that are based on contactless sensing of physiological signals.© 2022 The Authors. Published by Elsevier B.V.

Practical hyperparameters tuning of convolutional neural networks for EEG emotional features classification

EEG-based emotional states recognition is an active research direction, rushed up by recent emerging applications of intelligent systems and smart environments. However, accurate EEG-based emotion recognition is still challenging even with the use of the most efficient recent approaches like deep learning. Whereas many studies showed that convolutional neural networks can achieve good results in this field, they did not offer a general guide to subsequent researchers in designing dedicated networks. For practical deployment of deep neural networks in emotion-aware systems, an efficient design method, in terms of hyperparameters selection facilities, is needed. This paper investigates the simplification of the design process of a convolutional neural network applied to a binary and subject-dependent emotional valence classification. The proposed approach is based on two key solutions which are network architecture modularity and procedural tuning method. The proposed tuning approach estimates, in a given search space, the hyperparameter tunability defined as its main effect on the classification accuracy. The search space is depicted by selected hyperparameters levels and a fractional orthogonal array according to the Taguchi robust Design of Experiments method. In this investigation, we include twelve hyperparameters related to the neural network architecture and the training process. The significance of the estimated hyperparameter tunability is validated by statistical variance analysis. Experimental results allowed the determination of the best performance hyperparameters combination, confirmed the well-established effect of some training hyperparameters and revealed the importance of the pooling layer type in our case.

Emotion Priming in People with Williams Syndrome.

Emotion categories configure the basic semantic knowledge of the human cognitive structure. Previous studies with people with Williams syndrome (WS) investigated their ability to process basic emotions and the dimensions of emotional valences. However, little is known about the categorization of emotions from the subordinate perspective of lexical words in people with WS. In this study, emotion priming was used as the research paradigm. Three types of emotional valence were used as stimuli: positive, neutral, and negative. Each emotional valence was used as a prime matched to a target in one of these same three types of emotional valence. All participants were asked to judge whether the prime and the target were matched in their emotional valence. People with WS (n = 14, 11M/3F, CA = 10.49, and MA = 6.57) showed priming patterns for emotion valences like those of the typically developing controls. When positive primes were presented, accuracy was higher for positive and negative targets than neutral targets. When neutral primes were presented, accuracy was highest for negative targets. When negative primes were presented, accuracy was the lowest for negative targets. All participants showed high priming accuracy for positive emotions; however, they confused neutral with positive targets. A negative priming effect was observed when negative primes preceded negative targets. Considering previous findings that people with WS show developmental delays in the basic emotions of anger and surprise, this study concludes that people with WS responded least accurately to the classification of emotional valence. The findings regarding the categorization of emotions in people with WS not only advance our understanding of their emotion knowledge and socioemotional cognition but also confirm the superficial enrichment of lexical semantics with weak conceptual change in people with WS. This weakness may result in impaired contextual integration in people with WS.

Multi-Stage Machine Learning Model for Hierarchical Tie Valence Prediction

Individuals interacting in organizational settings involving varying levels of formal hierarchy naturally form a complex network of social ties having different tie valences (e.g., positive and negative connections). Social ties critically affect employees’ satisfaction, behaviors, cognition, and outcomes—yet identifying them solely through survey data is challenging because of the large size of some organizations or the often hidden nature of these ties and their valences. We present a novel deep learning model encompassing NLP and graph neural network techniques that identifies positive and negative ties in a hierarchical network. The proposed model uses human resource attributes as node information and web-logged work conversation data as link information. Our findings suggest that the presence of conversation data improves the tie valence classification by 8.91% compared to employing user attributes alone. This gain came from accurately distinguishing positive ties, particularly for male, non-minority, and older employee groups. We also show a substantial difference in conversation patterns for positive and negative ties with positive ties being associated with more messages exchanged on weekends, and lower use of words related to anger and sadness. These findings have broad implications for facilitating collaboration and managing conflict within organizational and other social networks.

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations.

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. Major research efforts are dedicated to the development of emotion recognition methods. However, most of the affective computing models are based on images, audio, videos and brain signals. Literature lacks works that focus on utilizing only peripheral signals for emotion recognition (ER), which can be ideally implemented in daily life settings. Therefore, this paper present a framework for ER on the arousal and valence space, based on using multi-modal peripheral signals. The data used in this work were collected during a debate between two people using wearable devices. The emotions of the participants were rated by multiple raters and converted into classes in correspondence to the arousal and valence space. The use of a dynamic threshold for ratings conversion was investigated. An ER model is proposed that uses a Long Short-Term Memory (LSTM)-based architecture for classification. The model uses heart rate (HR), temperature (T), and electrodermal activity (EDA) signals as its inputs with emotional cues. Additionally, a post-processing prediction mechanism is introduced to enhance the recognition performance. The model is implemented to study the use of individual and different combinations of the peripheral signals, as well as utilizing annotations from different ratings. Additionally, it is employed for classification of valence and arousal in an independent and combined fashion, under subject dependent and independent scenarios. The experimental results have justified the efficient performance of the proposed framework, achieving classification accuracy 96% and 93% for the independent and combined classification scenarios, accordingly. The comparison of the achieved performance against the baseline methods shows the superiority of the proposed framework and the ability to recognize arousal-valance levels with high accuracy from peripheral signals, in real-life scenarios.

Multiview Feature Fusion Attention Convolutional Recurrent Neural Networks for EEG‐Based Emotion Recognition

Emotion recognition is essential for computers to understand human emotions. Traditional EEG emotion recognition methods have significant limitations. To improve the accuracy of EEG emotion recognition, we propose a multiview feature fusion attention convolutional recurrent neural network (multi‐aCRNN) model. Multi‐aCRNN combines CNN, GRU, and attention mechanisms to fuse features from multiple perspectives deeply. Specifically, multiscale CNN can unite elements in the frequency and spatial domains through the convolution of different scales. The role of the attention mechanism is to weigh the frequency domain and spatial domain information of different periods to find more valuable temporal perspectives. Finally, the implicit feature representation is learned from the time domain through the bidirectional GRU to achieve the profound fusion of features from multiple perspectives in the time domain, frequency domain, and spatial domain. At the same time, for the noise problem, we use label smoothing to reduce the influence of label noise to achieve a better emotion recognition classification effect. Finally, the model is validated on the EEG data of 32 subjects on a public dataset (DEAP) by fivefold cross‐validation. Multi‐aCRNN achieves an average classification accuracy of 96.43% and 96.30% in arousal and valence classification tasks, respectively. In conclusion, multi‐aCRNN can better integrate EEG features from different angles and provide better classification results for emotion recognition.