Features For Emotion Recognition Research Articles

Enhancing cross-subject emotion recognition precision through unimodal EEG: a novel emotion preceptor model

Affective computing is a key research area in computer science, neuroscience, and psychology, aimed at enabling computers to recognize, understand, and respond to human emotional states. As the demand for affective computing technology grows, emotion recognition methods based on physiological signals have become research hotspots. Among these, electroencephalogram (EEG) signals, which reflect brain activity, are highly promising. However, due to individual physiological and anatomical differences, EEG signals introduce noise, reducing emotion recognition performance. Additionally, the synchronous collection of multimodal data in practical applications requires high equipment and environmental standards, limiting the practical use of EEG signals. To address these issues, this study proposes the Emotion Preceptor, a cross-subject emotion recognition model based on unimodal EEG signals. This model introduces a Static Spatial Adapter to integrate spatial information in EEG signals, reducing individual differences and extracting robust encoding information. The Temporal Causal Network then leverages temporal information to extract beneficial features for emotion recognition, achieving precise recognition based on unimodal EEG signals. Extensive experiments on the SEED and SEED-V datasets demonstrate the superior performance of the Emotion Preceptor and validate the effectiveness of the new data processing method that combines DE features in a temporal sequence. Additionally, we analyzed the model’s data flow and encoding methods from a biological interpretability perspective and validated it with neuroscience research related to emotion generation and regulation, promoting further development in emotion recognition research based on EEG signals.

Emotion Recognition Based on a EEG-fNIRS Hybrid Brain Network in the Source Space.

Background/Objectives: Studies have shown that emotion recognition based on electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) multimodal physiological signals exhibits superior performance compared to that of unimodal approaches. Nonetheless, there remains a paucity of in-depth investigations analyzing the inherent relationship between EEG and fNIRS and constructing brain networks to improve the performance of emotion recognition. Methods: In this study, we introduce an innovative method to construct hybrid brain networks in the source space based on simultaneous EEG-fNIRS signals for emotion recognition. Specifically, we perform source localization on EEG signals to derive the EEG source signals. Subsequently, causal brain networks are established in the source space by analyzing the Granger causality between the EEG source signals, while coupled brain networks in the source space are formed by assessing the coupling strength between the EEG source signals and the fNIRS signals. The resultant causal brain networks and coupled brain networks are integrated to create hybrid brain networks in the source space, which serve as features for emotion recognition. Results: The effectiveness of our proposed method is validated on multiple emotion datasets. The experimental results indicate that the recognition performance of our approach significantly surpasses that of the baseline method. Conclusions: This work offers a novel perspective on the fusion of EEG and fNIRS signals in an emotion-evoked experimental paradigm and provides a feasible solution for enhancing emotion recognition performance.

An enhanced Coati Optimization Algorithm for global optimization and feature selection in EEG emotion recognition

Emotion recognition based on Electroencephalography (EEG) signals has garnered significant attention across diverse domains including healthcare, education, information sharing, and gaming, among others. Despite its potential, the absence of a standardized feature set poses a challenge in efficiently classifying various emotions. Addressing the issue of high dimensionality, this paper introduces an advanced variant of the Coati Optimization Algorithm (COA), called eCOA for global optimization and selecting the best subset of EEG features for emotion recognition. Specifically, COA suffers from local optima and imbalanced exploitation abilities as other metaheuristic methods. The proposed eCOA incorporates the COA and RUNge Kutta Optimizer (RUN) algorithms. The Scale Factor (SF) and Enhanced Solution Quality (ESQ) mechanism from RUN are applied to resolve the raised shortcomings of COA. The proposed eCOA algorithm has been extensively evaluated using the CEC'22 test suite and two EEG emotion recognition datasets, DEAP and DREAMER. Furthermore, the eCOA is applied for binary and multi-class classification of emotions in the dimensions of valence, arousal, and dominance using a multi-layer perceptron neural network (MLPNN). The experimental results revealed that the eCOA algorithm has more powerful search capabilities than the original COA and seven well-known counterpart methods related to statistical, convergence, and diversity measures. Furthermore, eCOA can efficiently support feature selection to find the best EEG features to maximize performance on four quadratic emotion classification problems compared to the methods of its counterparts. The suggested method obtains a classification accuracy of 85.17% and 95.21% in the binary classification of low and high arousal emotions in two public datasets: DEAP and DREAMER, respectively, which are 5.58% and 8.98% superior to existing approaches working on the same datasets for different subjects, respectively.

Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset

Emotion recognition using multimodal data is a widely adopted approach due to its potential to enhance human interactions and various applications. By leveraging multimodal data for emotion recognition, the quality of human interactions can be significantly improved. We present the Multimodal Emotion Lines Dataset (MELD) and a novel method for multimodal emotion recognition using a bi-lateral gradient graph neural network (Bi-LG-GNN) and feature extraction and pre-processing. The multimodal dataset uses fine-grained emotion labeling for textual, audio, and visual modalities. This work aims to identify affective computing states successfully concealed in the textual and audio data for emotion recognition and sentiment analysis. We use pre-processing techniques to improve the quality and consistency of the data to increase the dataset’s usefulness. The process also includes noise removal, normalization, and linguistic processing to deal with linguistic variances and background noise in the discourse. The Kernel Principal Component Analysis (K-PCA) is employed for feature extraction, aiming to derive valuable attributes from each modality and encode labels for array values. We propose a Bi-LG-GCN-based architecture explicitly tailored for multimodal emotion recognition, effectively fusing data from various modalities. The Bi-LG-GCN system takes each modality's feature-extracted and pre-processed representation as input to the generator network, generating realistic synthetic data samples that capture multimodal relationships. These generated synthetic data samples, reflecting multimodal relationships, serve as inputs to the discriminator network, which has been trained to distinguish genuine from synthetic data. With this approach, the model can learn discriminative features for emotion recognition and make accurate predictions regarding subsequent emotional states. Our method was evaluated on the MELD dataset, yielding notable results in terms of accuracy (80%), F1-score (81%), precision (81%), and recall (81%) when using the MELD dataset. The pre-processing and feature extraction steps enhance input representation quality and discrimination. Our Bi-LG-GCN-based approach, featuring multimodal data synthesis, outperforms contemporary techniques, thus demonstrating its practical utility.

Multiscale-multichannel feature extraction and classification through one-dimensional convolutional neural network for Speech emotion recognition

Multiscale-multichannel feature extraction and classification through one-dimensional convolutional neural network for Speech emotion recognition

Identifying sex differences in EEG-based emotion recognition using graph convolutional network with attention mechanism

Objective. Sex differences in emotions have been widely perceived via self-reports, peripheral physiological signals and brain imaging techniques. However, how sex differences are reflected in the electroencephalography (EEG) neural patterns of emotions remains unresolved. In this paper, we detect sex differences in emotional EEG patterns, investigate the consistency of such differences in various emotion datasets across cultures, and study how sex as a factor affects the performance of EEG-based emotion recognition models. Approach. We thoroughly assess sex differences in emotional EEG patterns on five public datasets, including SEED, SEED-IV, SEED-V, DEAP and DREAMER, systematically examine the sex-specific EEG patterns for happy, sad, fearful, disgusted and neutral emotions, and implement deep learning models for sex-specific emotion recognition. Main results. (1) Sex differences exist in various emotion types and both Western and Eastern cultures; (2) The emotion patterns of females are more stable than those of males, and the patterns of happiness from females are in sharp contrast with the patterns of sadness, fear and disgust, while the energy levels are more balanced for males; (3) The key features for emotion recognition are mainly located at the frontal and temporal sites for females and distributed more evenly over the whole brain for males, and (4) the same-sex emotion recognition models outperform the corresponding cross-sex models. Significance. These findings extend efforts to characterize sex differences in emotional brain activation, provide new physiological evidence for sex-specific emotion processing, and reinforce the message that sex differences should be carefully considered in affective research and precision medicine.

S-LSTM-ATT: a hybrid deep learning approach with optimized features for emotion recognition in electroencephalogram.

Human emotion recognition using electroencephalograms (EEG) is a critical area of research in human-machine interfaces. Furthermore, EEG data are convoluted and diverse; thus, acquiring consistent results from these signals remains challenging. As such, the authors felt compelled to investigate EEG signals to identify different emotions. A novel deep learning(DL) model stacked long short-term memory with attention (S-LSTM-ATT) model is proposed for emotion recognition (ER) in EEG signals. Long Short-Term Memory (LSTM) and attention networks effectively handle time-series EEG data and recognise intrinsic connections and patterns. Therefore, the model combined the strengths of the LSTM model and incorporated an attention network to enhance its effectiveness. Optimal features were extracted from the metaheuristic-based firefly optimisation algorithm (FFOA) to identify different emotions efficiently. The proposed approach recognised emotions in two publicly available standard datasets: SEED and EEG Brainwave. An outstanding accuracy of 97.83% in the SEED and 98.36% in the EEG Brainwave datasets were obtained for three emotion indices: positive, neutral and negative. Aside from accuracy, a comprehensive comparison of the proposed model's precision, recall, F1 score and kappa score was performed to determine the model's applicability. When applied to the SEED and EEG Brainwave datasets, the proposed S-LSTM-ATT achieved superior results to baseline models such as Convolutional Neural Networks(CNN), Gated Recurrent Unit(GRU) and LSTM. Combining an FFOA-based feature selection(FS) and an S-LSTM-ATT-based classification model demonstrated promising results with high accuracy. Other metrics like precision, recall, F1 score and kappa score proved the suitability of the proposed model for ER in EEG signals.

MTDN: Learning Multiple Temporal Dynamics Representation for Emotional Valence Classification with EEG.

Emotion recognition from electroencephalogram (EEG) requires computational models to capture the crucial features of the emotional response to external stimulation. Spatial, spectral, and temporal information are relevant features for emotion recognition. However, learning temporal dynamics is a challenging task, and there is a lack of efficient approaches to capture such information. In this work, we present a deep learning framework called MTDN that is designed to capture spectral features with a filterbank module and to learn spatial features with a spatial convolution block. Multiple temporal dynamics are jointly learned with parallel long short-term memory (LSTM) embedding and self-attention modules. The LSTM module is used to embed the time segments, and then the self-attention is utilized to learn the temporal dynamics by intercorrelating every embedded time segment. Multiple temporal dynamics representations are then aggregated to form the final extracted features for classification. We experiment on a publicly available dataset, DEAP, to evaluate the performance of our proposed framework and compare MTDN with existing published results. The results demonstrate improvement over the current state-of-the-art methods on the valence dimension of the DEAP dataset.

Intelligent Speech Emotion Classification Using Deep Learning Technique

Abstract: Speech emotion recognition is a very interesting but very challenging human-computer interaction task. In recent years, this topic has attracted a lot of attention. In the field of speech emotion recognition, many techniques have been used to extract emotion from signals, including many well-established speech analysis and classification techniques. In the traditional way of speech emotion recognition, the emotion recognition features are extracted from the speech signals, and then the features, which are collectively known as the selection module, are selected, and then the emotions are recognized, which is a very tedious and time-consuming process, so this paper provides an overview of the deep learning technique, which is based on a simple algorithm based on feature extraction and building a model that recognizes emotions

Emotion Recognition from Spatio-Temporal Representation of EEG Signals via 3D-CNN with Ensemble Learning Techniques

The recognition of emotions is one of the most challenging issues in human-computer interaction (HCI). EEG signals are widely adopted as a method for recognizing emotions because of their ease of acquisition, mobility, and convenience. Deep neural networks (DNN) have provided excellent results in emotion recognition studies. Most studies, however, use other methods to extract handcrafted features, such as Pearson correlation coefficient (PCC), Principal Component Analysis, Higuchi Fractal Dimension (HFD), etc., even though DNN is capable of generating meaningful features. Furthermore, most earlier studies largely ignored spatial information between the different channels, focusing mainly on time domain and frequency domain representations. This study utilizes a pre-trained 3D-CNN MobileNet model with transfer learning on the spatio-temporal representation of EEG signals to extract features for emotion recognition. In addition to fully connected layers, hybrid models were explored using other decision layers such as multilayer perceptron (MLP), k-nearest neighbor (KNN), extreme learning machine (ELM), XGBoost (XGB), random forest (RF), and support vector machine (SVM). Additionally, this study investigates the effects of post-processing or filtering output labels. Extensive experiments were conducted on the SJTU Emotion EEG Dataset (SEED) (three classes) and SEED-IV (four classes) datasets, and the results obtained were comparable to the state-of-the-art. Based on the conventional 3D-CNN with ELM classifier, SEED and SEED-IV datasets showed a maximum accuracy of 89.18% and 81.60%, respectively. Post-filtering improved the emotional classification performance in the hybrid 3D-CNN with ELM model for SEED and SEED-IV datasets to 90.85% and 83.71%, respectively. Accordingly, spatial-temporal features extracted from the EEG, along with ensemble classifiers, were found to be the most effective in recognizing emotions compared to state-of-the-art methods.

You're Not You When You're Angry: Robust Emotion Features Emerge by Recognizing Speakers

The robustness of an acoustic emotion recognition system hinges on first having access to features that represent an acoustic input signal. These representations should abstract extraneous low-level variations present in acoustic signals and only capture speaker characteristics relevant for emotion recognition. Previous research has demonstrated that, in other classification tasks, when large labeled datasets are available, neural networks trained on these data learn to extract robust features from the input signal. However, the datasets used for developing emotion recognition systems remain significantly smaller than those used for developing other speech systems. Thus, acoustic emotion recognition systems remain in need of robust feature representations. In this work, we study the utility of speaker embeddings, representations extracted from a trained speaker recognition network, as robust features for detecting emotions. We first study the relationship between emotions and speaker embeddings, and demonstrate how speaker embeddings highlight the differences that exist between neutral speech and emotionally expressive speech. We quantify the modulations that variations in emotional expression incur on speaker embeddings, and show how these modulations are greater than those incurred from lexical variations in an utterance. Finally, we demonstrate how speaker embeddings can be used as a replacement for traditional low-level acoustic features for emotion recognition.

SPEECH EMOTION RECOGNITION WITH FACIAL EXPRESSIONS

The speech emotion recognition is a very interesting yet very challenging task of human computer interaction. In the recent years this topic has grabbed so much attention. In the field of speech emotion recognition many techniques have been utilized to extract emotions from signals , including many well- established speech analysis and classification techniques. In the traditional way of speech emotion recognition features are extracted from the speech signals and then the features are selected which is collectively know as selection module and then the emotions are recognized this is a very lengthy and time taking process. To understand the importance of Speech emotion & facial expression and how it is important for emotion detection. Emotion detection from our project has vast study for further future research and development. To build automatically speech and facial expression recognition has an effective emerging for developments. This project extends study and development on emotion detection with speech and facial expressions using Convolution Neural network (CNN). This task is done by detecting the facial actions per every unit of face measurements as a sub part of facial action coding system. This project offers light on utilization of CNN from a live video stream as an input. Using various machine learning libraries like tensor flow and many more. With this development it's advantageous to various domains such as medical engineering, technology, marketing etc.. Key words:- Emotion , Features , Facial expressions , Speech , Techniques , Technology.

A Bimodal Emotion Recognition Approach through the Fusion of Electroencephalography and Facial Sequences.

In recent years, human-computer interaction (HCI) systems have become increasingly popular. Some of these systems demand particular approaches for discriminating actual emotions through the use of better multimodal methods. In this work, a deep canonical correlation analysis (DCCA) based multimodal emotion recognition method is presented through the fusion of electroencephalography (EEG) and facial video clips. A two-stage framework is implemented, where the first stage extracts relevant features for emotion recognition using a single modality, while the second stage merges the highly correlated features from the two modalities and performs classification. Convolutional neural network (CNN) based Resnet50 and 1D-CNN (1-Dimensional CNN) have been utilized to extract features from facial video clips and EEG modalities, respectively. A DCCA-based approach was used to fuse highly correlated features, and three basic human emotion categories (happy, neutral, and sad) were classified using the SoftMax classifier. The proposed approach was investigated based on the publicly available datasets called MAHNOB-HCI and DEAP. Experimental results revealed an average accuracy of 93.86% and 91.54% on the MAHNOB-HCI and DEAP datasets, respectively. The competitiveness of the proposed framework and the justification for exclusivity in achieving this accuracy were evaluated by comparison with existing work.

Comprehensive Analysis of Feature Extraction Methods for Emotion Recognition from Multichannel EEG Recordings

Advances in signal processing and machine learning have expedited electroencephalogram (EEG)-based emotion recognition research, and numerous EEG signal features have been investigated to detect or characterize human emotions. However, most studies in this area have used relatively small monocentric data and focused on a limited range of EEG features, making it difficult to compare the utility of different sets of EEG features for emotion recognition. This study addressed that by comparing the classification accuracy (performance) of a comprehensive range of EEG feature sets for identifying emotional states, in terms of valence and arousal. The classification accuracy of five EEG feature sets were investigated, including statistical features, fractal dimension (FD), Hjorth parameters, higher order spectra (HOS), and those derived using wavelet analysis. Performance was evaluated using two classifier methods, support vector machine (SVM) and classification and regression tree (CART), across five independent and publicly available datasets linking EEG to emotional states: MAHNOB-HCI, DEAP, SEED, AMIGOS, and DREAMER. The FD-CART feature-classification method attained the best mean classification accuracy for valence (85.06%) and arousal (84.55%) across the five datasets. The stability of these findings across the five different datasets also indicate that FD features derived from EEG data are reliable for emotion recognition. The results may lead to the possible development of an online feature extraction framework, thereby enabling the development of an EEG-based emotion recognition system in real time.

Cross-session Emotion Recognition by Joint Label-common and Label-specific EEG Features Exploration.

Since Electroencephalogram (EEG) is resistant to camouflage, it has been a reliable data source for objective emotion recognition. EEG is naturally multi-rhythm and multi-channel, based on which we can extract multiple features for further processing. In EEG-based emotion recognition, it is important to investigate whether there exist some common features shared by different emotional states, and the specific features associated with each emotional state. However, such fundamental problem is ignored by most of the existing studies. To this end, we propose a Joint label-Common and label-Specific Features Exploration (JCSFE) model for semi-supervised cross-session EEG emotion recognition in this paper. To be specific, JCSFE imposes the ℓ2,1-norm on the projection matrix to explore the label-common EEG features and simultaneously the ℓ1-norm is used to explore the label-specific EEG features. Besides, a graph regularization term is introduced to enforce the data local invariance property, i.e., similar EEG samples are encouraged to have the same emotional state. Results obtained from the SEED-IV and SEED-V emotional data sets experimentally demonstrate that JCSFE not only achieves superior emotion recognition performance in comparison with the state-of-the-art models but also provides us with a quantitative method to identify the label-common and label-specific EEG features in emotion recognition.

A Comparative Study on Prominent Connectivity Features for Emotion Recognition From EEG

Classifying distinct human emotions, the fundamental purpose of brain-computer interface research, is essential for providing instant personalized services and assistance to individuals. With such emerging applications for individuals, several techniques have been proposed recently to explore interactions between brain regions, such as correlation, synchronization, and dependence. Notably, functional and effective connectivity methods are applied to assess the relationships between different brain areas. The primary objective of this study is to compare the frequently used functional and effective connectivity methods to recognize emotion using Electroencephalogram (EEG) signals. This paper uses a benchmark emotional EEG dataset consisting of 32 channels of EEG signals collected from 32 subjects while they were watching 40 inspirational music videos. Specifically, correlation, phase synchronization, and mutual information are used to measure functional brain connectivity, and transfer entropy is used to acquire effective brain connectivity. After extracting the features, they are represented in a two-dimensional connectivity feature map (CFM). The CFMs are then used to classify emotions by a convolutional neural network model. The results of classified emotions are analyzed regarding compatible EEG bands, accuracy, and time. Notably, the Gamma band is found as the most compatible band. The comparative study has demonstrated that though the connectivity method named Pearson correlation coefficient requires less time, the normalized mutual information is the most accurate method with advantageous detecting capability of nonlinear dependencies.

TMLP+SRDANN: A domain adaptation method for EEG-based emotion recognition

TMLP+SRDANN: A domain adaptation method for EEG-based emotion recognition

Feature selection enhancement and feature space visualization for speech-based emotion recognition.

Robust speech emotion recognition relies on the quality of the speech features. We present speech features enhancement strategy that improves speech emotion recognition. We used the INTERSPEECH 2010 challenge feature-set. We identified subsets from the features set and applied principle component analysis to the subsets. Finally, the features are fused horizontally. The resulting feature set is analyzed using t-distributed neighbour embeddings (t-SNE) before the application of features for emotion recognition. The method is compared with the state-of-the-art methods used in the literature. The empirical evidence is drawn using two well-known datasets: Berlin Emotional Speech Dataset (EMO-DB) and Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) for two languages, German and English, respectively. Our method achieved an average recognition gain of 11.5% for six out of seven emotions for the EMO-DB dataset, and 13.8% for seven out of eight emotions for the RAVDESS dataset as compared to the baseline study.

EEG-based multi-frequency band functional connectivity analysis and the application of spatio-temporal features in emotion recognition

• Proposed a novel approach for extracting spatio-temporal features of EEG signals. • Performed functional connectivity analysis of relevant brain regions during emotional activation. • Explored the differences between EEG signals in various frequencies during the emotional process. • Average classification accuracy of greater than 83 % was achieved by the classifier SVM. The study of emotional states in brain-computer interface (BCI) has a wide range of applications in psychiatry, psychology, et al. However, there is few novel feature extraction method integrating time-domain and space-domain features in emotion classification. This study explored the connectivity patterns between brain regions over functional connectivity brain networks in different frequency bands of electroencephalogram (EEG) signals and proposed a novel feature extraction method to classify emotions, which provided a unique perspective on emotion recognition. We constructed phase locking value (PLV) matrices analyzed in different frequency bands. Then, three distance matrices, dF, dS, and dLE, were built using the corresponding three distance measures (the Frobenius norm, the spectral norm, and the log-Euclidean distance, respectively). And the complexity measures on those distance matrices were calculated. The distance matrices and complexity measures, as two features, were fed into the machine learning classifiers to validate the proposed method. Eventually, the dF matrix obtained an average classification accuracy of 83.96 % in the alpha band between positive and neutral emotions, the dLE matrix obtained an average classification accuracy of 84.12 % in the beta band between positive and negative emotions, and the dF matrix obtained an average classification accuracy of 83.56 % in the delta band between neutral and negative emotions. We conclude that the delta, alpha, and beta frequency bands correlate highly with emotions, and the brain's anterior and right temporal lobes are inextricably linked to emotions. In addition, the feature extraction method proposed in this paper can effectively improve the classification accuracy of emotions.

Multi-Channel EEG Emotion Recognition Based on Parallel Transformer and 3D-Convolutional Neural Network

Due to its covert and real-time properties, electroencephalography (EEG) has long been the medium of choice for emotion identification research. Currently, EEG-based emotion recognition focuses on exploiting temporal, spatial, and spatiotemporal EEG data for emotion recognition. Due to the lack of consideration of both spatial and temporal aspects of EEG data, the accuracy of EEG emotion detection algorithms employing solely spatial or temporal variables is low. In addition, approaches that use spatiotemporal properties of EEG for emotion recognition take temporal and spatial characteristics of EEG into account; however, these methods extract temporal and spatial information directly from EEG data. Since there is no reconstruction of the EEG data format, the temporal and spatial properties of the EEG data cannot be extracted efficiently. To address the aforementioned issues, this research proposes a multi-channel EEG emotion identification model based on the parallel transformer and three-dimensional convolutional neural networks (3D-CNN). First, parallel channel EEG data and position reconstruction EEG sequence data are created separately. The temporal and spatial characteristics of EEG are then retrieved using transformer and 3D-CNN models. Finally, the features of the two parallel modules are combined to form the final features for emotion recognition. On the DEAP, Dreamer, and SEED databases, the technique achieved greater accuracy in emotion recognition than other methods. It demonstrates the efficiency of the strategy described in this paper.