Emotion Labels Research Articles

Deep Learning for Arabic Speech Recognition Using Convolutional Neural Networks

Extracting the speaker's emotional state from their speech has become an active research topic lately due to the demand for more human interactive applications. This field of research has noted significant advancement, especially in the English language, owing to the availability of massive speech-labeled corpora. However, the progress of analogous methodologies in the Arabic language is still in its infancy stages. In this paper, we present a Speech Recognition model for the Arabic language, proficient in discerning both the emotional state and gender of the speaker through voice analysis. Three primary emotion labels were selected: low, standard, and high levels of emotion. Various spectral features, such as the mel-frequency cepstral coefficient (MFCC), were extracted and tested to determine the optimal features. Furthermore, various Machine Learning models (SVM, KNN, and HMM) and Deep Learning models (LSTM and CNN) were evaluated for training. The results were compared between the five models using different extracted features, ultimately culminating in the selection of MFCC, root-mean-square (RMS), mel-scaled spectrogram, spectral, and zero-crossing rate as spectral features, and the CNN as a classification model. This selection yielded significant results, with an accuracy of 93% for emotion recognition and 99% for gender recognition.

Decoding Emotions: Intelligent visual perception for movie image classification using sustainable AI in entertainment computing

Affective computing researchers have been studying emotion recognition (ER) based on real-world facial photos and videos for a while now, and it’s a very popular area. With the sounds produced by head posture, facial deformation, and lighting fluctuation, ER is still difficult to execute in the wild. This research employs a machine learning model for sustainable artificial intelligence (AI) in entertainment computing to perform emotion decoding for movie picture categorization. Emotion labels are produced for every video sample by the suggested model, which takes video data as input. First, we employ a face identification and selection process based on the video data to identify the most consequential face areas. Here, facial expressions from films have been gathered as input pictures, which have then been processed for noise reduction and normalisation. Then, in order to analyse the facial expressions in this image, it was segmented using the fuzzy K-means equalisation clustering model. Convolutional adversarial U-net graph neural networks have been used to classify the studied pictures for emotion decoding. Several movie-based emotion datasets are subjected to experimental analysis in order to determine accuracy, precision, recall, F-1 score, RMSE, and AUC. With excellent classification accuracy, the suggested deep learning paradigm shows promise in identifying the emotional shifts in gamers. The suggested method achieved 97% accuracy, 96% precision, 92% recall, 85% F-1 score, 79% RMSE, and 86% AUC.

Yes, I am afraid of the sharks and also wild lions!: A multitask framework for enhancing dialogue generation via knowledge and emotion grounding

Current end-to-end neural conversation models inherently lack the capability to generate coherently engaging responses. Efforts to boost informativeness have an adversarial effect on emotional and factual accuracy, as validated by several sequence-based models. While these issues can be alleviated by access to emotion labels and background knowledge, there is no guarantee of relevance and informativeness in the generated responses. In real dialogue corpus, informative words like named entities, and words that carry specific emotions can often be infrequent and hard to model, and one primary challenge of the dialogue system is how to promote the model’s capability of generating high-quality responses with those informative words. Furthermore, earlier approaches depended on straightforward concatenation techniques that lacked robust representation capabilities in order to account for human emotions. To address this problem, we propose a novel multitask hierarchical encoder–decoder model, which can enhance the multi-turn dialogue response generation by incorporating external textual knowledge and relevant emotions. Experimental results on a benchmark dataset indicate that our model is superior over competitive baselines concerning both automatic and human evaluation.

Towards the development of an automated robotic storyteller: comparing approaches for emotional story annotation for non-verbal expression via body language

AbstractStorytelling is a long-established tradition and listening to stories is still a popular leisure activity. Caused by technization, storytelling media expands, e.g., to social robots acting as multi-modal storytellers, using different multimodal behaviours such as facial expressions or body postures. With the overarching goal to automate robotic storytelling, we have been annotating stories with emotion labels which the robot can use to automatically adapt its behavior. With it, three different approaches are compared in two studies in this paper: 1) manual labels by human annotators (MA), 2) software-based word-sensitive annotation using the Linguistic Inquiry and Word Count program (LIWC), and 3) a machine learning based approach (ML). In an online study showing videos of a storytelling robot, the annotations were validated, with LIWC and MA achieving the best, and ML the worst results. In a laboratory user study, the three versions of the story were compared regarding transportation and cognitive absorption, revealing no significant differences but a positive trend towards MA. On this empirical basis, the Automated Robotic Storyteller was implemented using manual annotations. Future iterations should include other robots and modalities, fewer emotion labels and their probabilities.

MES-CTNet: A Novel Capsule Transformer Network Base on a Multi-Domain Feature Map for Electroencephalogram-Based Emotion Recognition.

Emotion recognition using the electroencephalogram (EEG) has garnered significant attention within the realm of human-computer interaction due to the wealth of genuine emotional data stored in EEG signals. However, traditional emotion recognition methods are deficient in mining the connection between multi-domain features and fitting their advantages. In this paper, we propose a novel capsule Transformer network based on a multi-domain feature for EEG-based emotion recognition, referred to as MES-CTNet. The model's core consists of a multichannel capsule neural network(CapsNet) embedded with ECA (Efficient Channel Attention) and SE (Squeeze and Excitation) blocks and a Transformer-based temporal coding layer. Firstly, a multi-domain feature map is constructed by combining the space-frequency-time characteristics of the multi-domain features as inputs to the model. Then, the local emotion features are extracted from the multi-domain feature maps by the improved CapsNet. Finally, the Transformer-based temporal coding layer is utilized to globally perceive the emotion feature information of the continuous time slices to obtain a final emotion state. The paper fully experimented on two standard datasets with different emotion labels, the DEAP and SEED datasets. On the DEAP dataset, MES-CTNet achieved an average accuracy of 98.31% in the valence dimension and 98.28% in the arousal dimension; it achieved 94.91% for the cross-session task on the SEED dataset, demonstrating superior performance compared to traditional EEG emotion recognition methods. The MES-CTNet method, utilizing a multi-domain feature map as proposed herein, offers a broader observation perspective for EEG-based emotion recognition. It significantly enhances the classification recognition rate, thereby holding considerable theoretical and practical value in the EEG emotion recognition domain.

FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions

Although deep generative models have greatly improved one-shot video-driven talking head generation, few studies address fine-grained controllable facial expression editing, which is crucial for practical applications. Existing methods rely on a fixed set of predefined discrete emotion labels or simply copy expressions from input videos. This is limiting as expressions are complex, and methods using only emotion labels cannot generate fine-grained, accurate or mixed expressions. Generating talking head video with precise expressions is also difficult using 3D model-based approaches, as 3DMM only models facial movements and tends to produce deviations. In this paper, we propose a novel framework enabling fine-grained facial expression editing in talking face generation. Our goal is to achieve expression control by manipulating the intensities of individual facial Action Units (AUs) or groups. First, compared with existing methods which decouple the face into pose and expression, we propose a disentanglement scheme to isolates three components from the human face, namely, appearance, pose, and expression. Second, we propose to use input AUs to control muscle group intensities in the generated face, and integrate the AUs features with the disentangled expression latent code. Finally, we present a self-supervised training strategy with well-designed constraints. Experiments show our method achieves fine-grained expression control, produces high-quality talking head videos and outperforms baseline methods.

Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audio-visual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

From Coarse to Fine: A Distillation Method for Fine-Grained Emotion-Causal Span Pair Extraction in Conversation

We study the problem of extracting emotions and the causes behind these emotions in conversations. Existing methods either tackle them separately or jointly model them at the coarse-grained level of emotions (fewer emotion categories) and causes (utterance-level causes). In this work, we aim to jointly extract more fine-grained emotions and causes. We construct a fine-grained dataset FG-RECCON, includes 16 fine-grained emotion categories and span-level causes. To further improve the fine-grained extraction performance, we propose to utilize the casual discourse knowledge in a knowledge distillation way. Specifically, the teacher model learns to predict causal connective words between utterances, and then guides the student model in identifying both the fine-grained emotion labels and causal spans. Experimental results demonstrate that our distillation method achieves state-of-the-art performance on both RECCON and FG-RECCON dataset.

Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling

Conversational Speech Synthesis (CSS) aims to accurately express an utterance with the appropriate prosody and emotional inflection within a conversational setting. While recognising the significance of CSS task, the prior studies have not thoroughly investigated the emotional expressiveness problems due to the scarcity of emotional conversational datasets and the difficulty of stateful emotion modeling. In this paper, we propose a novel emotional CSS model, termed ECSS, that includes two main components: 1) to enhance emotion understanding, we introduce a heterogeneous graph-based emotional context modeling mechanism, which takes the multi-source dialogue history as input to model the dialogue context and learn the emotion cues from the context; 2) to achieve emotion rendering, we employ a contrastive learning-based emotion renderer module to infer the accurate emotion style for the target utterance. To address the issue of data scarcity, we meticulously create emotional labels in terms of category and intensity, and annotate additional emotional information on the existing conversational dataset (DailyTalk). Both objective and subjective evaluations suggest that our model outperforms the baseline models in understanding and rendering emotions. These evaluations also underscore the importance of comprehensive emotional annotations. Code and audio samples can be found at: https://github.com/walker-hyf/ECSS.

MM-TTS: Multi-Modal Prompt Based Style Transfer for Expressive Text-to-Speech Synthesis

The style transfer task in Text-to-Speech (TTS) refers to the process of transferring style information into text content to generate corresponding speech with a specific style. However, most existing style transfer approaches are either based on fixed emotional labels or reference speech clips, which cannot achieve flexible style transfer. Recently, some methods have adopted text descriptions to guide style transfer. In this paper, we propose a more flexible multi-modal and style controllable TTS framework named MM-TTS. It can utilize any modality as the prompt in unified multi-modal prompt space, including reference speech, emotional facial images, and text descriptions, to control the style of the generated speech in a system. The challenges of modeling such a multi-modal style controllable TTS mainly lie in two aspects: 1) aligning the multi-modal information into a unified style space to enable the input of arbitrary modality as the style prompt in a single system, and 2) efficiently transferring the unified style representation into the given text content, thereby empowering the ability to generate prompt style-related voice. To address these problems, we propose an aligned multi-modal prompt encoder that embeds different modalities into a unified style space, supporting style transfer for different modalities. Additionally, we present a new adaptive style transfer method named Style Adaptive Convolutions (SAConv) to achieve a better style representation. Furthermore, we design a Rectified Flow based Refiner to solve the problem of over-smoothing Mel-spectrogram and generate audio of higher fidelity. Since there is no public dataset for multi-modal TTS, we construct a dataset named MEAD-TTS, which is related to the field of expressive talking head. Our experiments on the MEAD-TTS dataset and out-of-domain datasets demonstrate that MM-TTS can achieve satisfactory results based on multi-modal prompts. The audio samples and constructed dataset are available at https://multimodal-tts.github.io.

ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment

The objective of stylized speech-driven facial animation is to create animations that encapsulate specific emotional expressions. Existing methods often depend on pre-established emotional labels or facial expression templates, which may limit the necessary flexibility for accurately conveying user intent. In this research, we introduce a technique that enables the control of arbitrary styles by leveraging natural language as emotion prompts. This technique presents benefits in terms of both flexibility and user-friendliness. To realize this objective, we initially construct a Text-Expression Alignment Dataset (TEAD), wherein each facial expression is paired with several prompt-like descriptions. We propose an innovative automatic annotation method, supported by CahtGPT, to expedite the dataset construction, thereby eliminating the substantial expense of manual annotation. Following this, we utilize TEAD to train a CLIP-based model, termed ExpCLIP, which encodes text and facial expressions into semantically aligned style embeddings. The embeddings are subsequently integrated into the facial animation generator to yield expressive and controllable facial animations. Given the limited diversity of facial emotions in existing speech-driven facial animation training data, we further introduce an effective Expression Prompt Augmentation (EPA) mechanism to enable the animation generator to support unprecedented richness in style control. Comprehensive experiments illustrate that our method accomplishes expressive facial animation generation and offers enhanced flexibility in effectively conveying the desired style.

A collaborative filtering method by fusion of facial information features

Personalized recommendation systems fundamentally assess user preferences as a reflection of their emotional responses to items. Traditional recommendation algorithms, focusing primarily on numerical processing, often overlook emotional factors, leading to reduced accuracy and limited application scenarios. This paper introduces a collaborative filtering recommendation method that integrates features of facial information, derived from emotions extracted from such data. Upon user authorization for camera usage, the system captures facial information features. Owing to the diversity in facial information, deep learning methods classify these features, employing the classification results as emotional labels. This approach calculates the similarity between emotional and item labels, reducing the ambiguity inherent in facial information features. The fusion process of facial information takes into account the user’s emotional state prior to item interaction, which might influence the emotions generated during the interaction. Variance is utilized to measure emotional fluctuations, thereby circumventing misjudgments caused by sustained non-interactive emotions. In selecting the nearest neighboring users, the method considers not only the similarity in user ratings but also in their emotional responses. Tests conducted using the Movielens dataset reveal that the proposed method, modeling facial features, more effectively aligns recommendations with user preferences and significantly enhances the algorithm’s performance.

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

Facial Emotional Recognition using Raspberry Pi

This facial emotion recognition project utilizes a Raspberry Pi for real-time analysis of facial expressions. Using a camera module, the system captures facial images, and a pre-trained deep learning model processes the data to classify emotions such as happiness, sadness, anger, etc. The Raspberry Pi's GPIO pins are employed to trigger specific actions or display emotion labels. This compact and efficient system demonstrates the integration of edge computing and machine learning for emotion analysis in a portable setup. Keywords- : Face, Expression, Raspberry Pi, Recognition, Feature Extraction.

A hierarchy of visual processing deficits in body dysmorphic disorder: a conceptual review and empirical investigation

ABSTRACT Objective Abnormal visual processing has been proposed as a mechanism underlying excessive focus on minor appearance flaws in body dysmorphic disorder (BDD). Existing BDD research has not differentiated the various stages of face processing (featural, first-order configural, holistic and second-order configural) that are required for higher-order processes such as emotion recognition. This study investigated a hierarchical visual processing model to examine the nature of abnormalities in face processing in BDD. Method Thirty BDD participants and 27 healthy controls completed the Navon task, a featural and configural face processing task and a facial emotion labelling task. Results BDD participants performed similarly to controls when processing global and local non-face stimuli on the Navon task, when detecting subtle changes in the features and spacing of a target face, and when labelling emotional faces. However, BDD participants displayed poorer performance when viewing inverted faces, indicating difficulties in configural processing. Conclusions The findings only partially support prior work. However, synthesis of results with previous findings indicates that heterogenous task methodologies may contribute to inconsistent findings. Recommendations are provided regarding the task parameters that appear most sensitive to abnormalities in BDD.

Affective analysis in machine learning using AMIGOS with Gaussian expectation-maximization model

<p>Investigating human subjects is the goal of predicting human emotions in the stock market. A significant number of psychological effects require (feelings) to be produced, directly releasing human emotions. The development of effect theory leads one to believe that one must be aware of one's sentiments and emotions to forecast one's behavior. The proposed line of inquiry focuses on developing a reliable model incorporating neurophysiological data into actual feelings. Any change in emotional affect will directly elicit a response in the body's physiological systems. This approach is named after the notion of Gaussian mixture models (GMM). The statistical reaction following data processing, quantitative findings on emotion labels, and coincidental responses with training samples all directly impact the outcomes that are accomplished. In terms of statistical parameters such as population mean and standard deviation, the suggested method is evaluated compared to a technique considered to be state-of-the-art. The proposed system determines an individual's emotional state after a minimum of 6 iterative learning using the Gaussian expectation-maximization (GEM) statistical model, in which the iterations tend to continue to zero error. Perhaps each of these improves predictions while simultaneously increasing the amount of value extracted.</p>

Multimodal Emotion Recognition Using Bi-LG-GCN for MELD Dataset

Emotion recognition using multimodal data is a widely adopted approach due to its potential to enhance human interactions and various applications. By leveraging multimodal data for emotion recognition, the quality of human interactions can be significantly improved. We present the Multimodal Emotion Lines Dataset (MELD) and a novel method for multimodal emotion recognition using a bi-lateral gradient graph neural network (Bi-LG-GNN) and feature extraction and pre-processing. The multimodal dataset uses fine-grained emotion labeling for textual, audio, and visual modalities. This work aims to identify affective computing states successfully concealed in the textual and audio data for emotion recognition and sentiment analysis. We use pre-processing techniques to improve the quality and consistency of the data to increase the dataset’s usefulness. The process also includes noise removal, normalization, and linguistic processing to deal with linguistic variances and background noise in the discourse. The Kernel Principal Component Analysis (K-PCA) is employed for feature extraction, aiming to derive valuable attributes from each modality and encode labels for array values. We propose a Bi-LG-GCN-based architecture explicitly tailored for multimodal emotion recognition, effectively fusing data from various modalities. The Bi-LG-GCN system takes each modality's feature-extracted and pre-processed representation as input to the generator network, generating realistic synthetic data samples that capture multimodal relationships. These generated synthetic data samples, reflecting multimodal relationships, serve as inputs to the discriminator network, which has been trained to distinguish genuine from synthetic data. With this approach, the model can learn discriminative features for emotion recognition and make accurate predictions regarding subsequent emotional states. Our method was evaluated on the MELD dataset, yielding notable results in terms of accuracy (80%), F1-score (81%), precision (81%), and recall (81%) when using the MELD dataset. The pre-processing and feature extraction steps enhance input representation quality and discrimination. Our Bi-LG-GCN-based approach, featuring multimodal data synthesis, outperforms contemporary techniques, thus demonstrating its practical utility.

Detecting multiple coexisting emotions from public emergency opinions

To detect multiple coexisting emotions from public emergency opinions, this article proposes a novel two-stage multiple coexisting emotion-detection model. First, the text semantic feature extracted through bidirectional encoder representation from transformers (BERT) and the emotion lexicon feature extracted through the emotion dictionary are fused. Then, the emotion subjectivity judgement and multiple coexisting emotion detection are performed in two separate stages. In the first stage, we introduce synthetic minority oversampling technique (SMOTE) to enhance the balance of data distribution and select the optimal classifier to recognise opinion texts with emotion. In the second stage, the label powerset (LP)-SMOTE is proposed to increase the number of the minority category samples, and multichannel emotion classifiers and the decision mechanism are employed to recognise different types of emotions and determine the final coexisting emotion labels. Finally, the Weibo data about coronavirus disease 2019 (COVID-19) are collected to verify the effectiveness of the proposed model. Experiment results indicate that the proposed model outperforms state-of-the-art models, with the F1_macro of 0.8532, the F1_micro of 0.8333, and the hamming loss of 0.0476. The emotion detection results are conducive to decision-making for public emergency departments.

Label superiority effect on emotional voices in young children

ABSTRACT Emotion labels can be helpful for creating emotion categories. Russell and Widen (2002) demonstrated the label superiority effect; that is, emotion labels produce a more precise categorization of emotional faces than the corresponding emotional faces. The current study aimed to test the label superiority effect on emotional voices and examined whether emotion labels produce a more precise categorization of emotional voices than the corresponding emotional voices. Four- to six-year-olds were shown emotion categories, represented by a box, and emotion labels (e.g., ‘happy’ label) or corresponding unlabelled vocal expressions were used to specify the target emotions. The children had to decide which of a group of test voices should go into the box and which should be left out. We presented happy and angry voices in Experiment 1 and angry and fearful voices in Experiment 2. The results in Experiment 2, but not in Experiment 1, showed that children were more likely to correctly choose the target emotion using the emotion label than the emotional voices, partially confirming the label superiority effect on emotional voices.

Personalization of Affective Models Using Classical Machine Learning: A Feasibility Study

Emotion recognition, a rapidly evolving domain in digital health, has witnessed significant transformations with the advent of personalized approaches and advanced machine learning (ML) techniques. These advancements have shifted the focus from traditional, generalized models to more individual-centric methodologies, underscoring the importance of understanding and catering to the unique emotional expressions of individuals. Our study delves into the concept of model personalization in emotion recognition, moving away from the one-size-fits-all approach. We conducted a series of experiments using the Emognition dataset, comprising physiological and video data of human subjects expressing various emotions, to investigate this personalized approach to affective computing. For the 10 individuals in the dataset with a sufficient representation of at least two ground truth emotion labels, we trained a personalized version of three classical ML models (k-nearest neighbors, random forests, and a dense neural network) on a set of 51 features extracted from each video frame. We ensured that all the frames used to train the models occurred earlier in the video than the frames used to test the model. We measured the importance of each facial feature for all the personalized models and observed differing ranked lists of the top features across the subjects, highlighting the need for model personalization. We then compared the personalized models against a generalized model trained using data from all 10 subjects. The mean F1 scores for the personalized models, specifically for the k-nearest neighbors, random forest, and dense neural network, were 90.48%, 92.66%, and 86.40%, respectively. In contrast, the mean F1 scores for the generic models, using the same ML techniques, were 88.55%, 91.78% and 80.42%, respectively, when trained on data from various human subjects and evaluated using the same test set. The personalized models outperformed the generalized models for 7 out of the 10 subjects. The PCA analyses on the remaining three subjects revealed relatively little facial configuration differences across the emotion labels within each subject, suggesting that personalized ML will fail when the variation among data points within a subject’s data is too low. This preliminary feasibility study demonstrates the potential as well as the ongoing challenges with implementing personalized models which predict highly subjective outcomes like emotion.