MRSLN: A Multimodal Residual Speaker-LSTM Network to alleviate the over-smoothing issue for Emotion Recognition in Conversation

Abstract

Multimodal Emotion Recognition in Conversation (ERC) plays a significant role in the field of human–computer intelligent interaction since it enables computers to perceive and infer the emotions expressed by the individuals, thereby intelligently responding to them. Most of current ERC methods pay more attention to modeling the complex interaction between different modalities. However, the features extracted by their unimodal networks are over-smoothed and may contain insufficient intra-speaker contextual information, which results in suboptimal results. In this paper, we focus on the unimodal learning and propose a simple late fusion framework named Multimodal Residual Speaker-LSTM Network (MRSLN), which uses speaker information to directly model inter-speaker and intra-speaker dependency, rather than fuse it into the learned features. MRSLN uses the speaker-LSTM consisting of the inter-speaker LSTM, intra-speaker LSTM, and the residual network between the input and output of the inter-speaker LSTM. Our proposed method can alleviate the issue of over-smoothing in deep Long Short Term Memory (LSTM) network and also incorporate additional intra-speaker contextual information. Extensive experiments conducted on IEMOCAP and MELD datasets demonstrate that MRSLN effectively captures inter-speaker and intra-speaker information and outperforms currently complex state-of-the-art (SOTA) models in efficiency and classification performance.