Network For Emotion Recognition Research Articles

A multi-task hybrid emotion recognition network based on EEG signals

Electroencephalogram (EEG)-based emotion recognition is getting a lot of attention in Brain–Computer Interface (BCI) research. However, the recognition model of cross-subject emotions requires effectively represented EEG data to capture diverse features corresponding to the spatial, frequency and temporal of information. In contrast, by calculating Power Spectral Density (PSD), Differential Entropy (DE), and the spaced interpolation algorithm, brain map construction has the ability to enhance the quality of frequency and spatial data features. In addition, DEAP is a dimensional dataset. We used the multi-task learning method while considering the characteristics of the dimensional dataset. Therefore, we proposed a novel hybrid emotion recognition deep learning network called the multi-task hybrid emotion recognition network(MTHNet), which fuses Bi-LSTM(Bidirectional long short-term memory) and improved UNet. In the hybrid network, improved UNet is utilized to extract spatial features. Then Bi-LSTM cells are applied to extract temporal features and memorize the change of sequential data in forward and backward directions. The proposed method is applied to the well-known 5 public databases and has given outperforming results than the current state of the art.

Cascaded Convolutional Recurrent Neural Networks for EEG Emotion Recognition Based on Temporal–Frequency–Spatial Features

Emotion recognition is a research area that spans multiple disciplines, including computational science, neuroscience, and cognitive psychology. The use of electroencephalogram (EEG) signals in emotion recognition is particularly promising due to their objective and nonartefactual nature. To effectively leverage the spatial information between electrodes, the temporal correlation of EEG sequences, and the various sub-bands of information corresponding to different emotions, we construct a 4D matrix comprising temporal–frequency–spatial features as the input to our proposed hybrid model. This model incorporates a residual network based on depthwise convolution (DC) and pointwise convolution (PC), which not only extracts the spatial–frequency information in the input signal, but also reduces the training parameters. To further improve performance, we apply frequency channel attention networks (FcaNet) to distribute weights to different channel features. Finally, we use a bidirectional long short-term memory network (Bi-LSTM) to learn the temporal information in the sequence in both directions. To highlight the temporal importance of the frame window in the sample, we choose the weighted sum of the hidden layer states at all frame moments as the input to softmax. Our experimental results demonstrate that the proposed method achieves excellent recognition performance. We experimentally validated all proposed methods on the DEAP dataset, which has authoritative status in the EEG emotion recognition domain. The average accuracy achieved was 97.84% for the four binary classifications of valence, arousal, dominance, and liking and 88.46% for the four classifications of high and low valence–arousal recognition.

Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognition

Recently, speech emotion recognition (SER) has become an active research area in speech processing, particularly with the advent of deep learning (DL). Numerous DL-based methods have been proposed for SER. However, most of the existing DL-based models are complex and require a large amounts of data to achieve a good performance. In this study, a new framework of deep attention-based dilated convolutional-recurrent neural networks coupled with a hybrid data augmentation method was proposed for addressing SER tasks. The hybrid data augmentation method constitutes an upsampling technique for generating more speech data samples based on the traditional and generative adversarial network approaches. By leveraging both convolutional and recurrent neural networks in a dilated form along with an attention mechanism, the proposed DL framework can extract high-level representations from three-dimensional log Mel spectrogram features. Dilated convolutional neural networks acquire larger receptive fields, whereas dilated recurrent neural networks overcome complex dependencies as well as the vanishing and exploding gradient issues. Furthermore, the loss functions are reconfigured by combining the SoftMax loss and the center-based losses to classify various emotional states. The proposed framework was implemented using the Python programming language and the TensorFlow deep learning library. To validate the proposed framework, the EmoDB and ERC benchmark datasets, which are imbalanced and/or small datasets, were employed. The experimental results indicate that the proposed framework outperforms other related state-of-the-art methods, yielding the highest unweighted recall rates of 88.03 ± 1.39 (%) and 66.56 ± 0.67 (%) for the EmoDB and ERC datasets, respectively.

STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition.

How to use the characteristics of EEG signals to obtain more complementary and discriminative data representation is an issue in EEG-based emotion recognition. Many studies have tried spatio-temporal or spatio-spectral feature fusion to obtain higher-level representations of EEG data. However, these studies ignored the complementarity between spatial, temporal and spectral domains of EEG signals, thus limiting the classification ability of models. This study proposed an end-to-end network based on ManifoldNet and BiLSTM networks, named STSNet. The STSNet first constructed a 4-D spatio-temporal-spectral data representation and a spatio-temporal data representation based on EEG signals in manifold space. After that, they were fed into the ManifoldNet network and the BiLSTM network respectively to calculate higher-level features and achieve spatio-temporal-spectral feature fusion. Finally, extensive comparative experiments were performed on two public datasets, DEAP and DREAMER, using the subject-independent leave-one-subject-out cross-validation strategy. On the DEAP dataset, the average accuracy of the valence and arousal are 69.38% and 71.88%, respectively; on the DREAMER dataset, the average accuracy of the valence and arousal are 78.26% and 82.37%, respectively. Experimental results show that the STSNet model has good emotion recognition performance.

Multi-loop graph convolutional network for multimodal conversational emotion recognition

Emotion recognition in conversations (ERC) has gained increasing research attention in recent years due to its wide applications in a surge of emerging tasks, such as social media analysis, dialog generation, and recommender systems. Since constituent utterances in a conversation are closely semantic-related, the constituent utterances’ emotional states are also closely related. In our consideration, this correlation could serve as a guide for the emotion recognition of constituent utterances. Accordingly, we propose a novel approach named Semantic-correlation Graph Convolutional Network (SC-GCN) to take advantage of this correlation for the ERC task in multimodal scenario. Specifically, we first introduce a hierarchical fusion module to model the dynamics among the textual, acoustic and visual features and fuse the multimodal information. Afterward, we construct a graph structure based on the speaker and temporal dependency of the dialog. We put forward a novel multi-loop architecture to explore the semantic correlations by the self-attention mechanism and enhance the correlation information via multiple loops. Through the graph convolution process, the proposed SC-GCN finally obtains a refined representation of each utterance, which is used for the final prediction. Extensive experiments are conducted on two benchmark datasets and the experimental results demonstrate the superiority of our SC-GCN.

HAAN-ERC: hierarchical adaptive attention network for multimodal emotion recognition in conversation

HAAN-ERC: hierarchical adaptive attention network for multimodal emotion recognition in conversation

STILN: A novel spatial-temporal information learning network for EEG-based emotion recognition

The spatial correlations and temporal contexts are indispensable in Electroencephalogram (EEG)-based emotion recognition. However, the learning of complex spatial correlations among several channels is a challenging problem. Besides, the temporal contexts learning is beneficial to emphasize the critical EEG frames because the subjects only reach the prospective emotion during part of stimuli. Hence, we propose a novel Spatial-Temporal Information Learning Network (STILN) to extract the discriminative features by capturing the spatial correlations and temporal contexts. Specifically, the generated 2D power topographic maps capture the dependencies among electrodes, and they are fed to the CNN-based spatial feature extraction network. Furthermore, Convolutional Block Attention Module (CBAM) recalibrates the weights of power topographic maps to emphasize the crucial brain regions and frequency bands. Meanwhile, Batch Normalizations (BNs) and Instance Normalizations (INs) are appropriately combined to relieve the individual differences. In the temporal contexts learning, we adopt the Bidirectional Long Short-Term Memory Network (Bi-LSTM) network to capture the dependencies among the EEG frames. To validate the effectiveness of the proposed method, subject-independent experiments are conducted on the public DEAP dataset. The proposed method has achieved the outstanding performance, and the accuracies of arousal and valence classification have reached 0.6831 and 0.6752 respectively.

Enhanced deep learning network for emotion recognition from GIF

The Graphic Interchange Format (GIF) is a bitmap picture format that has a series of perpetually repeating images or silent movies that may be viewed without the user having to click and start them. GIFs are frequently used to visually represent emotions that are expressed through body language such as gestures, movements, and facial expressions. Computing may be used to recognise thoughts and other emotions like desire, interest, sentiments, etc. by using emotional expressions or movements as face markers or properties in GIFs. The ability to predict emotions in GIFs may make it easier to express oneself on social media and convey a person’s attitude or personality. Emotion detection in GIFs may be utilised for a range of purposes, e.g., developing a recommendation system, detecting inappropriate content, sentiment identification from GIF-induced sentiment as perceived by person and creating a GIF tag generating system. This study discusses the prior contributions made towards emotion identification in GIFs and describes a method for detecting seven different emotion classes (Happy, Anger, Sad, Surprise, Disgust, Fear, and Neutral) in GIFs by combining an activity recognition network with face emotional expression. The suggested deep neural network, RNN, LSTM approach produced an F1-score of 0.89 and an accuracy of 88 percent.

GCF2-Net: global-aware cross-modal feature fusion network for speech emotion recognition.

Emotion recognition plays an essential role in interpersonal communication. However, existing recognition systems use only features of a single modality for emotion recognition, ignoring the interaction of information from the different modalities. Therefore, in our study, we propose a global-aware Cross-modal feature Fusion Network (GCF2-Net) for recognizing emotion. We construct a residual cross-modal fusion attention module (ResCMFA) to fuse information from multiple modalities and design a global-aware module to capture global details. More specifically, we first use transfer learning to extract wav2vec 2.0 features and text features fused by the ResCMFA module. Then, cross-modal fusion features are fed into the global-aware module to capture the most essential emotional information globally. Finally, the experiment results have shown that our proposed method has significant advantages than state-of-the-art methods on the IEMOCAP and MELD datasets, respectively.

Progressive graph convolution network for EEG emotion recognition

Studies in the area of neuroscience have revealed the relationship between emotional patterns and brain functional regions, demonstrating that the dynamic relationship between different brain regions is an essential factor affecting emotion recognition determined through electroencephalography (EEG). Moreover, in EEG emotion recognition, we can observe that clearer boundaries exist between coarse-grained emotions than those between fine-grained emotions, based on the same EEG data; this indicates the concurrence of large coarse- and small fine-grained emotion variations. The progressive classification process from coarse- to fine-grained categories may be helpful for EEG emotion recognition. Consequently, in this study, we proposed a progressive graph convolution network (PGCN) for capturing this inherent characteristic in EEG emotional signals and progressively learning the discriminative EEG features. To fit different EEG patterns, we constructed a dual-graph module to characterize the intrinsic relationship between different EEG channels, containing the dynamic functional connections and static spatial proximity information of brain regions from neuroscience research. Moreover, motivated by the observation of the relationship between coarse- and fine-grained emotions, we adopted a dual-head module that enabled the PGCN to progressively learn more discriminative EEG features, from coarse-grained (easy) to fine-grained categories (difficult), referring to the hierarchical characteristics of emotion. To verify the performance of our model, extensive experiments are conducted on three public datasets: SEED-IV, SEED-V, and MPED. The experiment results show that the PGCN achieves a state-of-the-art performance. Furthermore, we explored the effect of different frequency bands based on our model and visualized the activated brain regions. The experiment results reveal the relationship between human emotion and high-frequency EEG signals, as well as the importance of the frontal and temporal lobes for emotion expression.

A Domain Generative Graph Network for EEG-Based Emotion Recognition.

Emotion is a human attitude experience and corresponding behavioral response to objective things. Effective emotion recognition is important for the intelligence and humanization of brain-computer interface (BCI). Although deep learning has been widely used in emotion recognition in recent years, emotion recognition based on electroencephalography (EEG) is still a challenging task in practical applications. Herein, we proposed a novel hybrid model that employs generative adversarial networks to generate potential representations of EEG signals while combining graph convolutional neural networks and long short-term memory networks to recognize emotions from EEG signals. Experimental results on DEAP and SEED datasets show that the proposed model achieved the promising emotion classification performance compared with the state-of-the-art methods.

Multiview Feature Fusion Attention Convolutional Recurrent Neural Networks for EEG-Based Emotion Recognition

Emotion recognition is essential for computers to understand human emotions. Traditional EEG emotion recognition methods have significant limitations. To improve the accuracy of EEG emotion recognition, we propose a multiview feature fusion attention convolutional recurrent neural network (multi-aCRNN) model. Multi-aCRNN combines CNN, GRU, and attention mechanisms to fuse features from multiple perspectives deeply. Specifically, multiscale CNN can unite elements in the frequency and spatial domains through the convolution of different scales. The role of the attention mechanism is to weigh the frequency domain and spatial domain information of different periods to find more valuable temporal perspectives. Finally, the implicit feature representation is learned from the time domain through the bidirectional GRU to achieve the profound fusion of features from multiple perspectives in the time domain, frequency domain, and spatial domain. At the same time, for the noise problem, we use label smoothing to reduce the influence of label noise to achieve a better emotion recognition classification effect. Finally, the model is validated on the EEG data of 32 subjects on a public dataset (DEAP) by fivefold cross-validation. Multi-aCRNN achieves an average classification accuracy of 96.43% and 96.30% in arousal and valence classification tasks, respectively. In conclusion, multi-aCRNN can better integrate EEG features from different angles and provide better classification results for emotion recognition.

SUNET: Speaker-utterance interaction Graph Neural Network for Emotion Recognition in Conversations

Emotion Recognition in Conversations (ERC) can capture the speakers’ emotional changes in multiple rounds of conversation, so it has a wide range of applications. In recent years, Graph Neural Networks have been naturally used in ERC tasks due to their ability to capture complex non-Euclidian spatial features. However, how to model conversations easily and effectively to improve the effect of ERC in the complex interaction mode still needs to be explored. To this end, we propose a new approach to construct a speaker-utterance interactive heterogeneous network that effectively models context while taking into account the global characteristics of speakers. On this basis, we propose a graph neural network based on the speaker and the corresponding utterances interactions, which dynamically updates the representations of utterances and speakers according to the order in which the speakers talk. We formulate different update methods for utterance and speaker nodes to ensure that the particularity of the heterogeneous network is fully explored. We conduct extensive experiments on four ERC benchmark datasets, and our approach achieves an average 0.7% performance improvement over the most advanced methods, which validates the effectiveness of properly modeling speakers in ERC tasks.

Global-Local-Feature-Fused Driver Speech Emotion Detection for Intelligent Cockpit in Automated Driving

Affective interaction between the intelligent cockpit and humans is becoming an emerging topic full of opportunities. Robust recognition of the driver's emotions is the first step for affective interaction, and the intelligent cockpit recognizes emotions through the driver's speech, which has a wide range of technical application potential. In this paper, we first proposed a multi-feature fusion parallel structure speech emotion recognition network, which complementarily fuses the global acoustic features and local spectral features of the entire speech. Second, we designed and conducted the speech data collection under the driver's emotion and established the driver's speech emotion (SpeechEmo) dataset in the dynamic driving environment including 40 participants. Finally, the proposed model was validated on the SpeechEmo and public datasets, and quantitative analysis was carried out. It was found that the proposed model achieved advanced recognition performance, and the ablation experiments verified the importance of different components of the model. The proposed model and dataset are beneficial to the realization of human-vehicle affective interaction in intelligent cockpits in the future toward a better human experience.

An octonion-based nonlinear echo state network for speech emotion recognition in Metaverse

While the Metaverse is becoming a popular trend and drawing much attention from academia, society, and businesses, processing cores used in its infrastructures need to be improved, particularly in terms of signal processing and pattern recognition. Accordingly, the speech emotion recognition (SER) method plays a crucial role in creating the Metaverse platforms more usable​ and enjoyable for its users. However, existing SER methods continue to be plagued by two significant problems in the online environment. The shortage of adequate engagement and customization between avatars and users is recognized as the first issue and the second problem is related to the complexity of SER problems in the Metaverse as we face people and their digital twins or avatars. This is why developing efficient machine learning (ML) techniques specified for hypercomplex signal processing is essential to enhance the impressiveness and tangibility of the Metaverse platforms. As a solution, echo state networks (ESNs), which are an ML powerful tool for SER, can be an appropriate technique to enhance the Metaverse’s foundations in this area. Nevertheless, ESNs have some technical issues restricting them from a precise and reliable analysis, especially in the aspect of high-dimensional data. The most significant limitation of these networks is the high memory consumption caused by their reservoir structure in face of high-dimensional signals. To solve all problems associated with ESNs and their application in the Metaverse, we have come up with a novel structure for ESNs empowered by octonion algebra called NO2GESNet. Octonion numbers have eight dimensions, compactly display high-dimensional data, and improve the network precision and performance in comparison to conventional ESNs. The proposed network also solves the weaknesses of the ESNs in the presentation of the higher-order statistics to the output layer by equipping it with a multidimensional bilinear filter. Three comprehensive scenarios to use the proposed network in the Metaverse have been designed and analyzed, not only do they show the accuracy and performance of the proposed approach, but also the ways how SER can be employed in the Metaverse platforms.

GLFANet: A global to local feature aggregation network for EEG emotion recognition

Recently, emotion recognition technology based on electroencephalogram (EEG) signals is widely used in areas such as human–computer interaction and disease diagnosis. Traditional deep learning models rarely focus on the topological features of EEG electrodes, and often focus only on the local features of EEG signals, which makes it difficult to enhance the effectiveness of emotion recognition. In order to improve the accuracy and robustness of EEG-based emotion recognition algorithms, we propose an EEG emotion recognition algorithm based on a global to local feature aggregation network (GLFANet). This algorithm firstly uses the spatial location of the channels of EEG signals and the frequency domain features of each channel to construct an undirected topological graph to represent the spatial connection relationship between channels. Then, the GLFANet can learn deeper features of the undirected topology graph for emotion recognition. GLFANet mainly consists of a global learner composed of multiple graph convolution blocks and a local learner composed of multiple convolution blocks, which can learn both global and local features of EEG signals. The experiment results show that the proposed algorithm achieves higher accuracy on DEAP, SEED and DREAMER contrasted to other advanced algorithms.

Research on Speech Emotion Recognition Analysis Based on Deep Learning

This paper combines two aspects of feature selection and building deep neural networks to carry out targeted research to improve recognition accuracy. Firstly, speech preprocessing techniques are introduced to extract the speech spectrogram and lay the foundation for building the speech emotion recognition network model study. The focus is on building a speech emotion recognition network model based on residual network improvement and comparing experiments with AlexNet model network and ResNet-18 network model.

Joint Syntax-Enhanced and Topic-Driven Graph Networks for Emotion Recognition in Multi-Speaker Conversations

Daily conversations contain rich emotional information, and identifying this emotional information has become a hot task in the field of natural language processing. The traditional dialogue sentiment analysis method studies one-to-one dialogues and cannot be effectively applied to multi-speaker dialogues. This paper focuses on the relationship between participants in a multi-speaker conversation and analyzes the influence of each speaker on the emotion of the whole conversation. We summarize the challenges of emotion recognition work in multi-speaker dialogue, focusing on the context-topic switching problem caused by multi-speaker dialogue due to its free flow of topics. For this challenge, this paper proposes a graph network that combines syntactic structure and topic information. A syntax module is designed to convert sentences into graphs, using edges to represent dependencies between words, solving the colloquial problem of daily conversations. We use graph convolutional networks to extract the implicit meaning of discourse. In addition, we focus on the impact of topic information on sentiment, so we design a topic module to optimize the topic extraction and classification of sentences by VAE. Then, we use the combination of attention mechanism and syntactic structure to strengthen the model’s ability to analyze sentences. In addition, the topic segmentation technology is adopted to solve the long-term dependencies problem, and a heterogeneous graph is used to model the dialogue. The nodes of the graph combine speaker information and utterance information. Aiming at the interaction relationship between the subject and the object of the dialogue, different edge types are used to represent different interaction relationships, and different weights are assigned to them. The experimental results of our work on multiple public datasets show that the new model outperforms several other alternative methods in sentiment label classification results. In the multi-person dialogue dataset, the classification accuracy is increased by more than 4%, which verifies the effectiveness of constructing heterogeneous dialogue graphs.

Bi-CapsNet: A Binary Capsule Network for EEG-based Emotion Recognition.

In recent years, deep learning has gained widespread attention in electroencephalogram (EEG)-based emotion recognition. However, deep learning methods are usually time-consuming with a large amount of memory usage, which obstructs their practical usage on resource-constrained devices. In this paper, we propose a binary capsule network (Bi-CapsNet) for EEG emotion recognition with low computational cost and memory usage. The Bi-CapsNet binarizes 32-bit weights and activations to 1 bit, and replaces floating-point operations with efficient bitwise operations. To address the issue of function discontinuity in backward propagation, we use a continuous function to approximate the binarization process. Two popular EEG emotion databases, namely, DEAP and DREAMER, are used for performance evaluation. In comparison to its full-precision counterpart, the Bi-CapsNet achieves a reduction on the computational cost and a reduction on the memory usage, while with only a 1% drop on the recognition accuracy. Compared to some state-of-the-art EEG emotion recognition methods, the proposed method obtains more competitive performance. In addition, the Bi-CapsNet is implemented on a mobile phone via an open-source binary inference framework named Bolt, and it achieves an ∼5× inference acceleration in comparison to its full-precision counterpart.

A 3D Tensor Representation of Speech and 3D Convolutional Neural Network for Emotion Recognition

A 3D Tensor Representation of Speech and 3D Convolutional Neural Network for Emotion Recognition