Emotion Recognition In Conversations Research Articles

Improving multimodal fusion with Main Modal Transformer for emotion recognition in conversation

Emotion recognition in conversation (ERC) is essential for developing empathic conversation systems. In conversation, emotions can exist in multiple modalities, i.e., audio, text, and visual. Due to the inherent characteristics of each modality, it is not easy for the model to use all modalities effectively when fusing modal information. However, existing approaches consider the same representation ability of each modality, resulting in unsatisfactory fusion across modalities. Therefore, we consider different modalities with different representation abilities, propose the concept of the main modal, i.e., the modal with stronger representation ability after feature extraction, and then propose the method of Main Modal Transformer (MMTr) to improve the effect of multimodal fusion. The method preserves the integrity of the main modal features and enhances the representation of weak modalities by using multihead attention to learn the information interactions between modalities. In addition, we designed a new emotional cue extractor that extracts emotional cues from two levels (the speaker’s self-context and the contextual context in conversation) to enrich the conversation information obtained by each modal. Extensive experiments on two benchmark datasets validate the effectiveness and superiority of our model.

Context- and Sentiment-Aware Networks for Emotion Recognition in Conversation

Emotion recognition in conversation (ERC) has promising potential in many fields, such as recommendation systems, man–machine interaction, and medical care. In contrast to other emotion identification tasks, conversation is essentially a process of dynamic interaction in which people often convey emotional messages relying on context and common-sense knowledge. In this article, we propose a context- and sentiment-aware framework, termed Sentic GAT, to solve this challenge. In Sentic GAT, common-sense knowledge is dynamically represented by the context- and sentiment-aware graph attention mechanism based on sentimental consistency, and context information is captured by the dialogue transformer (DT) with hierarchical multihead attention (HMAT), where HMAT is used to obtain the dependency of historical utterances on themselves and other utterances for better context representation. Additionally, we explore a contrastive loss to discriminate context-free and context-sensitive utterances in emotion identification to enhance context representation in straightforward conversations that directly express ideas. The experimental results show that context and sentimental information can promote the representation of common-sense knowledge, and the intra- and inter-dependency of contextual utterances effectively improve the performance of Sentic GAT. Moreover, our Sentic GAT using emotional intensity outperforms the most advanced model on the tested datasets.

Adapted Dynamic Memory Network for Emotion Recognition in Conversation

In this article, we address Emotion Recognition in Conversation (ERC) where conversational data are presented in a multimodal setting. Psychological evidence shows that self and inter-speaker influence are two central factors to emotion dynamics in conversation. State-of-the-art models do not effectively synthesise these two factors. Therefore, we propose an Adapted Dynamic Memory Network (A-DMN) where self and inter-speaker influences are modelled individually and further synthesised oriented towards the current utterance. Specifically, we model the dependency of the constituent utterances in a dialogue video using a global RNN to capture inter-speaker influence. Likewise, each speaker is assigned an RNN to capture their self influence. Afterwards, an Episodic Memory Module is devised to extract contexts for self and inter-speaker influence and synthesise them to update the memory. This process repeats itself for multiple passes until a refined representation is obtained and used for final prediction. Additionally, we explore cross-modal fusion in the context of multimodal ERC, and propose a convolution-based method which proves effective in extracting local interactions and computationally efficient. Extensive experiments demonstrate that A-DMN outperforms the state-of-the-art models on benchmark datasets.

Context- and Knowledge-Aware Graph Convolutional Network for Multimodal Emotion Recognition

This work proposes an approach for emotion recognition in conversation that leverages context modeling, knowledge enrichment, and multimodal (text and audio) learning based on a graph convolutional network (GCN). We first construct two distinctive graphs for modeling the contextual interaction and knowledge dynamic. We then introduce an affective lexicon into knowledge graph building to enrich the emotional polarity of each concept, that is the related knowledge of each token in an utterance. Then, we achieve a balance between the context and the affect-enriched knowledge by incorporating them into the new adjacency matrix construction of the GCN architecture, and teach them jointly with multiple modalities to effectively structure the semantics-sensitive and knowledge-sensitive contextual dependence of each conversation. Our model outperforms the state-of-the-art benchmarks by over 22.6% and 11% relative error reduction in terms of weighted-F1 on the IEMOCAP and MELD databases, respectively, demonstrating the superiority of our method in emotion recognition.

Hybrid Curriculum Learning for Emotion Recognition in Conversation

Emotion recognition in conversation (ERC) aims to detect the emotion label for each utterance. Motivated by recent studies which have proven that feeding training examples in a meaningful order rather than considering them randomly can boost the performance of models, we propose an ERC-oriented hybrid curriculum learning framework. Our framework consists of two curricula: (1) conversation-level curriculum (CC); and (2) utterance-level curriculum (UC). In CC, we construct a difficulty measurer based on ``emotion shift'' frequency within a conversation, then the conversations are scheduled in an ``easy to hard" schema according to the difficulty score returned by the difficulty measurer. For UC, it is implemented from an emotion-similarity perspective, which progressively strengthens the model’s ability in identifying the confusing emotions. With the proposed model-agnostic hybrid curriculum learning strategy, we observe significant performance boosts over a wide range of existing ERC models and we are able to achieve new state-of-the-art results on four public ERC datasets.

Is Discourse Role Important for Emotion Recognition in Conversation?

A conversation is a sequence of utterances, where each utterance plays a specific discourse role while expressing a particular emotion. This paper proposes a novel method to exploit latent discourse role information of an utterance to determine the emotion it conveys in a conversation. Specifically, we use a variant of the Variational-Autoencoder (VAE) to model the context-aware latent discourse roles of each utterance in an unsupervised way. The latent discourse role representation further equips the utterance representation with a salient clue for more accurate emotion recognition. Our experiments show that our proposed method beats the best-reported performances on three public Emotion Recognition in Conversation datasets. This proves that the discourse role information of an utterance plays an important role in the emotion recognition task, which no previous work has studied.

Domain Adversarial Network for Cross-Domain Emotion Recognition in Conversation

Emotion Recognition in Conversation (ERC) aims to recognize the emotion for each utterance in a conversation automatically. Due to the difficulty of collecting and labeling, this task lacks the dataset corpora available on a large scale. This increases the difficulty of finishing the supervised training required by large-scale neural networks. Introducing the large-scale generative conversational dataset can assist with modeling dialogue. However, the spatial distribution of feature vectors in the source and target domains is inconsistent after introducing the external dataset. To alleviate the problem, we propose a Domain Adversarial Network for Cross-Domain Emotion Recognition in Conversation (DAN-CDERC) model, consisting of domain adversarial and emotion recognition models. The domain adversarial model consists of the encoders, a generator and a domain discriminator. First, the encoders and generator learn contextual features from a large-scale source dataset. The discriminator performs domain adaptation by discriminating the domain to make the feature space of the source and target domain consistent, so as to obtain domain invariant features. Then DAN-CDERC transfers the learned domain invariant dialogue context knowledge from the domain adversarial model to the emotion recognition model to assist in modeling the dialogue context. Due to the use of a domain adversarial network, DAN-CDERC obtains dialogue-level contextual information that is domain invariant, thereby reducing the negative impact of inconsistency in domain space. Empirical studies illustrate that the proposed model outperforms the baseline models on three benchmark emotion recognition datasets.

Few-shot emotion recognition in conversation with sequential prototypical networks

Detecting emotions in a conversational context benefits several industrial cases such as customer service, user appraisal from speech recognition, and so on. However, in most cases, research data differ from real data due to them being private, confidential, or difficult to label. In this work we present ProtoSeq, an adaptation of the Prototypical Networks to enable dealing with sequences in a few-shot learning way, reducing the need for labeling confidential data.

Emotion recognition in conversations with emotion shift detection based on multi-task learning

Emotion recognition in conversations (ERC) has attracted increasing attention from the research community for its wide applications. For ERC, the main challenge is how to effectively utilize the conversational context based on the adequate analysis of each utterance. Current research ignores to use the emotion shift information when modeling the conversational context which tends to result in recognition performance inadequacy. We believe that employing emotion shift as explicit guidance would help to further improve the performance of ERC. Therefore, we propose a multi-task learning model ESD–ERC, which comprises the auxiliary task of Emotion Shift Detection (ESD) and the main task of ERC. The model exploits a shared BERT-based encoder to extract the unified emotion semantic representations, obtains the emotion shift representations through ESD based on Bi-directional Long Short-Term Memory Neural Network and feeds the emotion semantic representations concatenated with the emotion shift representations into the context level Transformer with positional encoding for ERC. The comparative experiment results show that our model outperforms the state-of-the-art models on two different datasets, validating our idea about emotion shift. In addition, we verify the effectiveness of each component of ESD–ERC by ablation experiments and explain the significance of ESD by case study.

Emotion Recognition with Conversational Generation Transfer

Emotion recognition in conversation is one of the essential tasks of natural language processing. However, this task’s annotation data is insufficient since such data is hard to collect and annotate. Meanwhile, there is large-scale data for conversational generation, and this data does not need annotation manually. But, whether the vector space between different datasets is similar will be a problem. Therefore, we utilize a same dataset to train the conversational generator and the classifier, and transfer knowledge between them. In particular, we propose an Emotion Recognition with Conversational Generation Transfer (ERCGT) framework to model the interaction among utterances by transfer learning. First, we train a conversational generator. In the second step, a transfer learning model is used to transfer the knowledge of generator to the emotion recognition model. Empirical studies illustrate the effectiveness of the proposed framework over several strong baselines on three benchmark emotion classification datasets.

Discovering emotion and reasoning its flip in multi-party conversations using masked memory network and transformer

Efficient discovery of a speaker’s emotional states in a multi-party conversation is significant to design human-like conversational agents. During a conversation, the cognitive state of a speaker often alters due to certain past utterances, which may lead to a flip in their emotional state. Therefore, discovering the reasons (triggers) behind the speaker’s emotion-flip during a conversation is essential to explain the emotion labels of individual utterances. In this paper, along with addressing the task of emotion recognition in conversations (ERC), we introduce a novel task – Emotion-Flip Reasoning (EFR), that aims to identify past utterances which have triggered one’s emotional state to flip at a certain time. We propose a masked memory network to address the former and a Transformer-based network for the latter task. To this end, we consider MELD, a benchmark emotion recognition dataset in multi-party conversations for the task of ERC, and augment it with new ground-truth labels for EFR. An extensive comparison with five state-of-the-art models suggests improved performances of our models for both the tasks. We further present anecdotal evidence and both qualitative and quantitative error analyses to support the superiority of our models compared to the baselines.

LR-GCN: Latent Relation-Aware Graph Convolutional Network for Conversational Emotion Recognition

As an intersection of artificial intelligence and human communication analysis, Emotion Recognition in Conversation (ERC) has attracted much research attention in recent years. Existing studies, however, are limited in adequately exploiting latent relations among the constituent utterances. In this paper, we address this issue by proposing a novel approach named Latent Relation-Aware Graph Convolutional Network (LR-GCN), where both speaker dependency of the interlocutors is leveraged and latent correlations among the utterances are captured for ERC. Specifically, we first establish a graph model to incorporate the context information and speaker dependency of the conversation. Afterward, the multi-head attention mechanism is introduced to explore the latent correlations among the utterances and generate a set of all-linked graphs. Here, aiming to simultaneously exploit the original modeled speaker dependency and the explored correlation information, we introduce a dense connection layer to capture more structural information of the generated graphs. Through a multi-branch graph network, we achieve a unified representation of each utterance for final prediction. Detailed evaluations on two benchmark datasets demonstrate LR-GCN outperforms the state-of-the-art approaches.

Deep Graph-Recurrent Model for Emotion Recognition in Conversation Using Fully Connected Directed Acyclic

Deep Graph-Recurrent Model for Emotion Recognition in Conversation Using Fully Connected Directed Acyclic

M2seq2seq: A Multi-Task Sequence to Sequence Learning Model for Multi-Modal Sarcasm, Sentiment and Emotion Recognition in Conversation

M2seq2seq: A Multi-Task Sequence to Sequence Learning Model for Multi-Modal Sarcasm, Sentiment and Emotion Recognition in Conversation

Korean Drama Scene Transcript Dataset for Emotion Recognition in Conversations

Understanding emotions in conversation is a challenging task as the sentences often have an implied meaning which is not generally understood in isolation. Efficient use of contextual information is important for emotion recognition in conversations. Many of the published datasets provide contextual information for situations such as text-based online messaging, chatbots, and movie dialogues. However, such dialogue-based datasets are collected by selecting the ideal conversational situations and thus do not include many variations in dialogue length and number of participants. Therefore, such datasets may not be applicable for emotion recognition in text-based movie transcripts, where scenes contain variations in the number of speakers and length of the spoken sentences. We present a conversation dataset based on the Korean TV show transcripts for analysis of the emotions in presence of scene context. Korean Drama Scene Transcript dataset for Emotion Recognition (KD-EmoR) is a text-based conversation dataset. We analyze three classes of complex emotions namely euphoria, dysphoria, and neutral in the scenes of TV Drama to build a publicly available dataset for further research. We developed a context-aware deep learning model to classify the emotions utilizing speaker-level context and scene context and achieved an F1 score of 0.63 on the proposed dataset.

AdCOFE: Advanced Contextual Feature Extraction in conversations for emotion classification.

Emotion recognition in conversations is an important step in various virtual chatbots which require opinion-based feedback, like in social media threads, online support, and many more applications. Current emotion recognition in conversations models face issues like: (a) loss of contextual information in between two dialogues of a conversation, (b) failure to give appropriate importance to significant tokens in each utterance, (c) inability to pass on the emotional information from previous utterances. The proposed model of Advanced Contextual Feature Extraction (AdCOFE) addresses these issues by performing unique feature extraction using knowledge graphs, sentiment lexicons and phrases of natural language at all levels (word and position embedding) of the utterances. Experiments on emotion recognition in conversations datasets show that AdCOFE is beneficial in capturing emotions in conversations.

A multi-view network for real-time emotion recognition in conversations

Real-time emotion recognition in conversations (RTERC), the task of using the historical context to identify the emotion of a query utterance in a conversation, is important for opinion mining and building empathetic machines. Existing works mainly focus on obtaining each utterance representation separately and then utilizing utterance-level features to model the emotion representation of the query. These approaches treat each utterance as a unit and capture the utterance-level dependencies in the context, but ignore the word-level dependencies among different utterances. In this paper, we propose a multi-view network (MVN) to explore the emotion representation of a query from two different views, i.e., word- and utterance-level views. For the word-level view, MVN takes the context and query as word sequences and then models the word-level dependencies among utterances. For the utterance-level view, MVN extracts each utterance representation separately and then models the utterance-level dependencies in the context. Experimental results on two public emotion conversation datasets show that the proposed model outperforms the state-of-the-art baselines.

Multimodal emotion recognition with hierarchical memory networks

Emotion recognition in conversations is crucial as there is an urgent need to improve the overall experience of human-computer interactions. A promising improvement in this field is to develop a model that can effectively extract adequate contexts of a test utterance. We introduce a novel model, termed hierarchical memory networks (HMN), to address the issues of recognizing utterance level emotions. HMN divides the contexts into different aspects and employs different step lengths to represent the weights of these aspects. To model the self dependencies, HMN takes independent local memory networks to model these aspects. Further, to capture the interpersonal dependencies, HMN employs global memory networks to integrate the local outputs into global storages. Such storages can generate contextual summaries and help to find the emotional dependent utterance that is most relevant to the test utterance. With an attention-based multi-hops scheme, these storages are then merged with the test utterance using an addition operation in the iterations. Experiments on the IEMOCAP dataset show our model outperforms the compared methods with accuracy improvement.

DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition

This paper presents our pioneering effort for emotion recognition in conversation (ERC) with pre-trained language models. Unlike regular documents, conversational utterances appear alternately from different parties and are usually organized as hierarchical structures in previous work. Such structures are not conducive to the application of pre-trained language models such as XLNet. To address this issue, we propose an all-in-one XLNet model, namely DialogXL, with enhanced memory to store longer historical context and dialog-aware self-attention to deal with the multi-party structures. Specifically, we first modify the recurrence mechanism of XLNet from segment-level to utterance-level in order to better model the conversational data. Second, we introduce dialog-aware self-attention in replacement of the vanilla self-attention in XLNet to capture useful intra- and inter-speaker dependencies. Extensive experiments are conducted on four ERC benchmarks with mainstream models presented for comparison. The experimental results show that the proposed model outperforms the baselines on all the datasets. Several other experiments such as ablation study and error analysis are also conducted and the results confirm the role of the critical modules of DialogXL.

DECN: Dialogical emotion correction network for conversational emotion recognition

Emotion recognition in conversation (ERC) is an important research topic in artificial intelligence. Different from the emotion estimation in individual utterances, ERC requires proper handling of human interactions in conversations. Several approaches have been proposed for ERC and achieved promising results. In this paper, we propose a correction model for previous approaches, called “Dialogical Emotion Correction Network (DECN)”. This model aims to automatically correct some errors made by emotion recognition strategies and further improve the recognition performance. Specifically, DECN employs a graphical network to model human interactions in conversations. To further utilize the contextual information, DECN also employs the multi-head attention based bi-directional GRU component. Since DECN is a correction model for ERC, it can be easily integrated with any emotion recognition strategy. Experimental results on the IEMOCAP and MELD datasets verify the effectiveness of our proposed method. DECN can improve the performance of emotion recognition strategies with few parameters and low computational complexity.