Emotion-related Information Research Articles

Hybrid CNN-BiLSTM architecture with multiple attention mechanisms to enhance speech emotion recognition

During recent years, the concept of attention in deep learning has been increasingly used to boost the performance of Speech Emotion Recognition (SER) models. However, these models for SER exhibit shortcomings in jointly emphasizing the time-frequency and dynamic sequential variations, often under-utilizing the rich contextual emotion-related information. We propose a hybrid deep learning model for SER using Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory Networks (BiLSTM) with multiple attention mechanisms. Our model utilizes features from the speech waveform viz. Mel spectrograms and Mel Frequency Cepstral Coefficients (MFCC), along with their time derivatives as input to the CNN and BiLSTM modules, respectively. A Time–Frequency Attention (TFA) mechanism, optimally incorporated into CNN, helps to selectively focus on emotion-related energy–time–frequency variations in Mel spectrograms. Attention-based BiLSTM uses MFCC and its time derivatives to identify the positional information of emotion for addressing the dynamic sequential variations. Finally, we fuse the attention-learned features from the CNN and BiLSTM modules and feed them to a Deep Neural Network (DNN) for SER. The experiments were carried out using three different datasets: Emo-DB and IEMOCAP, which are public datasets, and Amritaemo_Arabic; a private dataset. The hybrid model exhibited superior performance on both the public and private datasets, generating an average SER accuracy of 94.62%, 67.85%, and 95.80% with Emo-DB, IEMOCAP, and Amritaemo_Arabic datasets, respectively, effectively outperforming several state-of-the-art models.

AN EMOTION ANALYSIS METHOD THAT INTEGRATES EEG SIGNALS AND MUSIC THERAPY USING A TRANSFORMER MODEL

The need for sentiment analysis in the mental health field is increasing, and electroencephalogram (EEG) signals and music therapy have attracted extensive attention from researchers as breakthrough ideas. However, the existing methods still face the challenge of integrating temporal and spatial features when combining these two types, especially when considering the volume conduction differences among multichannel EEG signals and the different response speeds of subjects; moreover, the precision and accuracy of emotion analysis have yet to be improved. To solve this problem, we integrate the idea of top-k selection into the classic transformer model and construct a novel top-k sparse transformer model. This model captures emotion-related information in a finer way by selecting k data segments from an EEG signal with distinct signal features. However, this optimization process is not without its challenges, and we need to balance the selected k values to ensure that the important features are preserved while avoiding excessive information loss. Experiments conducted on the DEAP dataset demonstrate that our approach achieves significant improvements over other models. By enhancing the sensitivity of the model to the emotion-related information contained in EEG signals, our method achieves an overall emotion classification accuracy improvement and obtains satisfactory results when classifying different emotion dimensions. This study fills a research gap in the field of sentiment analysis involving EEG signals and music therapy, provides a novel and effective method, and is expected to lead to new ideas regarding the application of deep learning in sentiment analysis.

A novel ensemble model for emotion classification in social media text

Social media provides people with a platform to share their experiences and perspectives, and text is the most common way as either posts or comments. Much emotion-related information, such as mental state and attitude, can be revealed through texts. As a result, text-based emotion analysis plays an important role. This paper aims to propose a new classification model using the ensemble learning method, which can classify the emotions detected from the text into six classes, including joy, fear, surprise, love, sadness, and anger. Multiple base models are trained at the first stage, including traditional machine learning models (Multinomial Naive Bayes, SVM, and Decision Tree) and deep learning models (CNN, LSTM, and GRU). Then, a new ensemble model using the stacking method is developed. The stacking of deep learning models and SVM has achieved the best classification performance, where the accuracy and F1 score are 0.8875 and 0.8410, respectively. The evaluation metrics demonstrate the effectiveness and robustness of the new ensemble model for this emotion classification task.

EEG emotion recognition based on differential entropy feature matrix through 2D-CNN-LSTM network

Emotion recognition research has attracted great interest in various research fields, and electroencephalography (EEG) is considered a promising tool for extracting emotion-related information. However, traditional EEG-based emotion recognition methods ignore the spatial correlation between electrodes. To address this problem, this paper proposes an EEG-based emotion recognition method combining differential entropy feature matrix (DEFM) and 2D-CNN-LSTM. In this work, first, the one-dimensional EEG vector sequence is converted into a two-dimensional grid matrix sequence, which corresponds to the distribution of brain regions of the EEG electrode positions, and can better characterize the spatial correlation between the EEG signals of multiple adjacent electrodes. Then, the EEG signal is divided into equal time windows, and the differential entropy (DE) of each electrode in this time window is calculated, it is combined with a two-dimensional grid matrix and differential entropy to obtain a new data representation that can capture the spatiotemporal correlation of the EEG signal, which is called DEFM. Secondly, we use 2D-CNN-LSTM to accurately identify the emotional categories contained in the EEG signals and finally classify them through the fully connected layer. Experiments are conducted on the widely used DEAP dataset. Experimental results show that the method achieves an average classification accuracy of 91.92% and 92.31% for valence and arousal, respectively. The method performs outstandingly in emotion recognition. This method effectively combines the temporal and spatial correlation of EEG signals, improves the accuracy and robustness of EEG emotion recognition, and has broad application prospects in the field of emotion classification and recognition based on EEG signals.

ST-SCGNN: A Spatio-Temporal Self-Constructing Graph Neural Network for Cross-Subject EEG-Based Emotion Recognition and Consciousness Detection.

In this paper, a novel spatio-temporal self-constructing graph neural network (ST-SCGNN) is proposed for cross-subject emotion recognition and consciousness detection. For spatio-temporal feature generation, activation and connection pattern features are first extracted and then combined to leverage their complementary emotion-related information. Next, a self-constructing graph neural network with a spatio-temporal model is presented. Specifically, the graph structure of the neural network is dynamically updated by the self-constructing module of the input signal. Experiments based on the SEED and SEED-IV datasets showed that the model achieved average accuracies of 85.90% and 76.37%, respectively. Both values exceed the state-of-the-art metrics with the same protocol. In clinical besides, patients with disorders of consciousness (DOC) suffer severe brain injuries, and sufficient training data for EEG-based emotion recognition cannot be collected. Our proposed ST-SCGNN method for cross-subject emotion recognition was first attempted in training in ten healthy subjects and testing in eight patients with DOC. We found that two patients obtained accuracies significantly higher than chance level and showed similar neural patterns with healthy subjects. Covert consciousness and emotion-related abilities were thus demonstrated in these two patients. Our proposed ST-SCGNN for cross-subject emotion recognition could be a promising tool for consciousness detection in DOC patients.

Effects of congruent emotional contexts during encoding on recognition: An ERPs study.

Past research showed that emotional contexts can impair recognition memory for the target item. Given that item-context congruity may enhance recognition memory, the present study aims to examine the effect of the congruent emotional encoding contexts on recognition memory. Participants studied congruent word-picture pairs (e.g., the word "cow" - a picture describing a cow) and incongruent word-picture pairs (e.g., the word "cow" - a picture describing a goat) and, subsequently, were asked to report the nature of the picture (emotional or neutral). Behavioral results revealed that emotional contexts impaired source but not item recognition, with congruent word-context mitigating this impairment and enhancing item recognition. Neural results from ERPs and theta oscillations found the recollection process, as shown by the LPC old/new effect and theta oscillations, for both item and source recognition across emotional contexts, irrespective of congruity. Meanwhile, the familiarity process as indexed by the FN400 old/new effect was found only for item recognition in congruent emotional contexts. These findings suggest that the congruent relationship of item-context could mitigate the emotion-induced source memory impairment and enhance item memory, with neural results elucidating the memory processes involved in retrieval of emotional information. Specifically, while emotion-related information generally elicits the recollection-based memory process, only congruent emotional information elicits the familiarity-based process.

WavDepressionNet: Automatic Depression Level Prediction Via Raw Speech Signals

Physiological reports have confirmed that there are differences in speech signals between depressed and healthy individuals. Therefore, as an application in the field of affective computing, automatic depression level prediction through speech signals has received the attention of researchers, which often estimate the depression severity of individuals by the Fourier or Mel spectrograms of speech signals. However, some studies on speech emotion recognition suggest that directly modeling the raw speech signal is more helpful for extracting emotion-related information. Inspired by this fact, we develop a WavDepressionNet to model raw speech signals for the improvement of prediction accuracy. In our method, a representation block is proposed to find a set of basis vectors to construct the optimal transformation space and generate the transformation result (named Depression Feature Map, DFM) of speech signal for facilitating the perception of depression cues. We further propose an assessment block, which cannot only use the designed spatiotemporal self-calibration mechanism to calibrate the DFM and highlight the useful elements, but also aggregates the calibrated DFM across various temporal ranges with the dilated convolution. Experimental results on the AVEC 2013 and AVEC 2014 depression databases demonstrate the effectiveness of our approach over previous works.

Temporal Convolutional Networks and BERT-Based Multi-Label Emotion Analysis for Financial Forecasting

The use of deep learning in conjunction with models that extract emotion-related information from texts to predict financial time series is based on the assumption that what is said about a stock is correlated with the way that stock fluctuates. Given the above, in this work, a multivariate forecasting methodology incorporating temporal convolutional networks in combination with a BERT-based multi-label emotion classification procedure and correlation feature selection is proposed. The results from an extensive set of experiments, which included predictions of three different time frames and various multivariate ensemble schemes that capture 28 different types of emotion-relative information, are presented. It is shown that the proposed methodology exhibits universal predominance regarding aggregate performance over six different metrics, outperforming all the compared schemes, including a multitude of individual and ensemble methods, both in terms of aggregate average scores and Friedman rankings. Moreover, the results strongly indicate that the use of emotion-related features has beneficial effects on the derived forecasts.

A Primary task driven adaptive loss function for multi-task speech emotion recognition

Although Speech Emotion Recognition (SER) is becoming an active research area, the state-of-the-art performance is limited by the scarcity of emotional datasets. The introduction of Multi-Task Learning (MTL) which jointly trains multiple relative tasks, can utilize more emotion-related information within limited data, enhancing the SER performance but also increasing the training difficulty. In this paper, we propose a primary-task driven adaptive loss function named as PTDA-Loss to adaptively adjust the task weights in the loss function, aiming to more reasonably allocate the learning resources in the training process and further improve the SER performance. We observe that excessive concern with the auxiliary task can lead to insufficient learning resources for SER, not conducive to SER performance improvement. Inspired by this finding, in PTDA-Loss, we retain the advantage of SER in the whole training process. Meanwhile, the learning resources for the auxiliary task will be appropriately but not over compressed, ensuring the effective learning of useful information from the auxiliary task. To improve the model’s learning ability, furthermore, we first employ several dynamic convolutional layers as the backbone network. Regarding gender recognition and speaker recognition as the auxiliary task respectively, ablation experiments on the IEMOCAP, MSP-IMPROV and MELD datasets prove the rationality and effectiveness of PTDA-Loss. Additionally, comparative experiments demonstrate that our method achieves state-of-the-art performance on the three datasets, further enhancing the SER accuracy.

Semantic Alignment Network for Multi-Modal Emotion Recognition

Modality alignment can maintain the consistency of semantics in multi-modal emotion recognition tasks, ensuring that features from different modalities accurately represent the emotion-related information in an encoding space. However, current alignment models either focus only on the local fusion of different modal representations or lack a mining process for unimodal specificity information. We design a Semantic Alignment network based on Multi-Spatial learning (SAMS) for multi-modal emotion recognition, which achieves local and global alignment between modalities using high-level emotion representations of different modalities as supervisory signals. SAMS builds a multi-spatial learning framework for each modality, and constructs a self-modal interaction module under this framework based on cross-modal semantic learning. SAMS provides two learning spaces for each modality, one to detect the affective information for a specific modality, and the other to learn semantic knowledge from other modalities. Subsequently, the features of these two spaces are aligned in temporal and utterance levels by homologous encoding and different target constraints. Based on the alignment characteristics of these two spaces, a self-modal interaction is built to investigate the fusion representation by exploring the global correlation between the alignment features in unimodal multi-spatial learning. In experiments, our proposed model yields consistent improvements on two standard multi-modal benchmarks, and outperforms state-of-the-art approaches. The code of our SAMS is available at: https://github.com/xiaomi1024/code_SAMS.

Caregiver speech predicts the emergence of children's emotion vocabulary.

Learning about emotions is an important part of children's social and communicative development. How does children's emotion-related vocabulary emerge over development? How may emotion-related information in caregiver input support learning of emotion labels and other emotion-related words? This investigation examined language production and input among English-speaking toddlers (16-30 months) using two datasets: Wordbank (N=5520; 36% female, 38% male, and 26% unknown gender; 1% Asian, 4% Black, 2% Hispanic, 40% White, 2% others, and 50% unknown ethnicity; collected in North America; dates of data collection unknown) and Child Language Data Exchange System (N=587; 46% female, 44% male, 9% unknown gender, all unknown ethnicity; collected in North America and the UK; data collection dates, were available between 1962 and 2009). First, we show that toddlers develop the vocabulary to express increasingly wide ranges of emotional information during the first 2 years of life. Computational measures of word valence showed that emotion labels are embedded in a rich network of words with related valence. Second, we show that caregivers leverage these semantic connections in ways that may scaffold children's learning of emotion and mental state labels. This research suggests that young children use the dynamics of language input to construct emotion word meanings, and provides new techniques for defining the quality of infant-directed speech.

EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network

Given the rapid development of brain–computer interfaces, emotion identification based on EEG signals has emerged as a new study area with tremendous importance in recent years. EEG-based emotion recognition remains a challenging task in pattern recognition due to the complexity and diversity of the emotion signal, even though the deep learning models have significantly outperformed the conventional techniques in this area. In this paper, we propose an EEG emotion recognition model on the basis of the attention mechanism and a pre-trained convolution capsule network to recognize various emotions more effectively. This model employs coordinate attention to endow the input signal with relative spatial information and then maps the EEG signal to higher dimensional space, which enriches the emotion-related information in EEG. The pre-trained model, which excels at extracting features, is adopted as the feature extractor. Last but not least, a double-layer capsule network is constructed for emotion recognition to completely utilize the relative location information of EEG data. The subject-dependent and subject-independent experiments using DEAP datasets are individually conducted in this study, with the findings demonstrating that the suggested strategy presented excellent recognition accuracy and generalizability.

Multi-Channel Weight-Sharing Autoencoder Based on Cascade Multi-Head Attention for Multimodal Emotion Recognition

Multimodal Emotion Recognition is challenging because of the heterogeneity gap among different modalities. Due to the powerful ability of feature abstraction, Deep Neural Networks (DNNs) have exhibited significant success in bridging the heterogeneity gap in cross-modal retrieval and generation tasks. In this work, a DNNs-based Multi-channel Weight-sharing Autoencoder with Cascade Multi-head Attention (MCWSA-CMHA) is proposed to generically address the affective heterogeneity gap in MER. Specifically, multimodal heterogeneity features are extracted by multiple independent encoders, and then a scalable heterogeneous feature fusion module (CMHA) is realized by connecting multiple multi-head attention modules in series. The core of the proposed algorithm is to reduce the heterogeneity between the output features of different encoders through the unsupervised training of MCWSA, and then to model the affective interactions between different modal features through the supervised training of CMHA. Experimental results demonstrate that the proposed MCWSA-CMHA achieves outperformance on two publicly available datasets compared with the state-of-the-art techniques. In addition, visualization experiments and approximation experiments are used to verify the effectiveness of each module in the proposed algorithm, and the experimental results show that the proposed MCWSA-CMHA can mine more emotion-related information among multimodal features compared with other fusion methods.

Learning DenseNet features from EEG based spectrograms for subject independent emotion recognition

Subject-independent emotion recognition (SIER) using electroencephalogram (EEG) signals has always been a challenge among the biomedical research community. One of the major reasons behind this is the chaotic and non-stationary nature of EEG signals which may show varying signal characteristics for the same emotions for different individuals. Therefore, it is a daunting challenge for the researchers to enhance the emotion recognition performance of the existing SIER systems, which is still quite low. To address these challenges, in this work, we have proposed a novel deep learning-based approach to efficiently extract and classify emotion-related information from the 2D spectrograms obtained from the 1D EEG signals. To uncover the hidden deep features of EEG signals we proposed a Deep Convolution neural network for Emotion Recognition (DCERNet) with dense connections among layers. The proposed DCERNet is developed with a sequence of customizations over a pre-trained Densenet121 model integrated with softmax and Support Vector Machine (SVM) classifiers on top. The EEG signals are transformed to 2D spectrograms and fed to the proposed model for identifying emotions. To evaluate the performance of the proposed models, comprehensive simulations are conducted using two publicly available databases SJTU Emotion EEG Dataset (SEED) and Database for Emotion Analysis of Physiological Signals (DEAP). Experimental outcomes illustrate that the proposed DCERNet model boosts the emotion recognition accuracy by almost 8% compared to the state-of-the-art methods using DEAP database.

Semi-supervised emotion recognition in textual conversation via a context-augmented auxiliary training task

Recognizing emotions in textual conversations (ERC) is to identify the emotion of utterances by considering conversational context. Current supervise-based ERC methods require a large number of diverse conversations to train a model that leverages context effectively. However, the scarcity of annotated training data in most ERC corpora hinders their performance improvement. In this paper, we explore to use the easily accessible unlabeled data mixed with labeled data to help ERC models to improve performance. Considering collected unlabeled data may not share the same emotion class with the labeled set, we propose a semi-supervised ERC algorithm that leverages the unlabeled conversational data through a novel Context-augmented AUXIliary training Task (CAUXIT), which trains along with the original ERC task in a multi-task fashion. Our idea is to utilize CAUXIT to learn a better utterance feature representation on top of existing ERC models. Especially, CAUXIT is designed to selectively mask the utterance based on a class-based sampling strategy and use the context, i.e., the rest utterances, to predict its emotion-related information rather than the lexical information of itself, which enhances the network’s ability in making emotion inference through context and consequently improve the utterance feature representation. In addition to applying CAUXIT to unlabeled data, we also extend it to labeled data to further enrich the supervision signal. As shown in the experiments, applying CAUXIT on various ERC models achieves a significant improvement over the same network architectures trained on labeled data, which verifies our approach as an effective semi-supervised ERC framework.

Localizing emotions: Soundscape representations through Smartphone Use

The term “soundscape” refers to a well-defined field that acts as a source of auditory stimuli and whose characteristics are directly related to the listener’s position. The study of soundscapes entails the study of the interplay between the listener and sound, as well as the attribution of representation to auditory information. The investigation of representation is intimately related to the person and emotions, as well as the spatial and auditory aspects of the environment. Emotion is a way of understanding the listeners, their experiences, and the environment. The research described in this paper aimed to identify and investigate the representation of the soundscape through emotional response. The research took place in the city of Limassol, Cyprus. The participants consisted of four groups of 10-15 persons each. A mixed methodological approach was followed – both quantitative and qualitative methods were employed. For the geolocation of the emotional responses that arise as a result of the representation of the urban soundscape in question, a mobile app was developed for the mediated experience; it runs on on Android, it is titled Locomotion, and it provides participants with the ability to indicate their emotional state based on the dimensions of Russell’s (2003) circumplex model of affect. The emotion-related information provided by the participants is overlaid on a map of the city, so that the emotional significance of various units of the urban environment is readily visible.

The Pastoral Origin of Semiotically Functional Tonal Organization of Music.

This paper presents a new line of inquiry into when and how music as a semiotic system was born. Eleven principal expressive aspects of music each contains specific structural patterns whose configuration signifies a certain affective state. This distinguishes the tonal organization of music from the phonetic and prosodic organization of natural languages and animal communication. The question of music’s origin can therefore be answered by establishing the point in human history at which all eleven expressive aspects might have been abstracted from the instinct-driven primate calls and used to express human psycho-emotional states. Etic analysis of acoustic parameters is the prime means of cross-examination of the typical patterns of expression of the basic emotions in human music versus animal vocal communication. A new method of such analysis is proposed here. Formation of such expressive aspects as meter, tempo, melodic intervals, and articulation can be explained by the influence of bipedal locomotion, breathing cycle, and heartbeat, long before Homo sapiens. However, two aspects, rhythm and melodic contour, most crucial for music as we know it, lack proxies in the Paleolithic lifestyle. The available ethnographic and developmental data leads one to believe that rhythmic and directional patterns of melody became involved in conveying emotion-related information in the process of frequent switching from one call-type to another within the limited repertory of calls. Such calls are usually adopted for the ongoing caretaking of human youngsters and domestic animals. The efficacy of rhythm and pitch contour in affective communication must have been spontaneously discovered in new important cultural activities. The most likely scenario for music to have become fully semiotically functional and to have spread wide enough to avoid extinctions is the formation of cross-specific communication between humans and domesticated animals during the Neolithic demographic explosion and the subsequent cultural revolution. Changes in distance during such communication must have promoted the integration between different expressive aspects and generated the basic musical grammar. The model of such communication can be found in the surviving tradition of Scandinavian pastoral music - kulning. This article discusses the most likely ways in which such music evolved.

Cochleogram-based approach for detecting perceived emotions in music

Identifying perceived emotional content of music constitutes an important aspect of easy and efficient search, retrieval, and management of the media. One of the most promising use cases of music organization is an emotion-based playlist, where automatic music emotion recognition plays a significant role in providing emotion related information, which is otherwise, generally unavailable. Based on the importance of the auditory system in emotional recognition and processing, in this study, we propose a new cochleogram-based system for detecting the affective musical content. To effectively simulate the response of the human auditory periphery, the music audio signal is processed by a detailed biophysical cochlear model, thus obtaining an output that closely matches the characteristics of human hearing. In this proposed approach, based on the cochleogram images, which we construct directly from the response of the basilar membrane, a convolutional neural network (CNN) is used to extract the relevant music features. To validate the practical implications of the proposed approach with regard to its possible integration in different digital music libraries, an extensive study was conducted to evaluate the predictive performance of our approach in different aspects of music emotion recognition. The proposed approach was evaluated on publicly available 1000 songs database and the experimental results showed that it performed better in comparison with common musical features (such as tempo, mode, pitch, clarity, and perceptually motivated mel-frequency cepstral coefficients (MFCC)) as well as official ”MediaEval” challenge results on the same reference database. Our findings clearly show that the proposed approach can lead to better music emotion recognition performance and be used as part of a state-of-the-art music information retrieval system.

Linking emotions to behaviors through deep transfer learning.

Human behavior refers to the way humans act and interact. Understanding human behavior is a cornerstone of observational practice, especially in psychotherapy. An important cue of behavior analysis is the dynamical changes of emotions during the conversation. Domain experts integrate emotional information in a highly nonlinear manner; thus, it is challenging to explicitly quantify the relationship between emotions and behaviors. In this work, we employ deep transfer learning to analyze their inferential capacity and contextual importance. We first train a network to quantify emotions from acoustic signals and then use information from the emotion recognition network as features for behavior recognition. We treat this emotion-related information as behavioral primitives and further train higher level layers towards behavior quantification. Through our analysis, we find that emotion-related information is an important cue for behavior recognition. Further, we investigate the importance of emotional-context in the expression of behavior by constraining (or not) the neural networks’ contextual view of the data. This demonstrates that the sequence of emotions is critical in behavior expression. To achieve these frameworks we employ hybrid architectures of convolutional networks and recurrent networks to extract emotion-related behavior primitives and facilitate automatic behavior recognition from speech.

A meta-analysis of the possible behavioural and biological variables linking trait emotional intelligence to health

ABSTRACTTrait Emotional Intelligence (trait EI) is a constellation of correlated emotion-related traits that capture an individual’s typical way of processing emotion-related information and reacting in emotional situations. Numerous studies have shown that trait EI is a significant predictor of both subjective and objective health. This correlational meta-analysis (k = 106, N = 45,262) aims to explore the behavioural and biological variables that could account for these effects. It also aims to provide a roadmap for future research by identifying what should be studied (pinpointing dead-end roads and promising paths) and how (methodological improvements needed to draw stronger conclusions). The results revealed large associations of trait EI with social support, sleep quality, and hypothalamic–pituitary–adrenal axis activity in challenging situations as well as medium associations with dietary habits, physical activity, and substance use. Other candidate pathways have given rise to much less research. Based on both theoretical predictions and preliminary findings, the paper categorises these pathways as promising or not promising. Future research would benefit from using more diverse samples, measuring behavioural variables more objectively, controlling for personality, and systematically examining to what extent changes in EI (e.g., following training) lead to changes in behaviours and/or biological parameters.