Language Reinforced Superposition Multimodal Fusion for Sentiment Analysis

Abstract

Nowadays, BERT has been effective in context extraction and widely utilized in multimodal fusion. However, many multimodal learning systems with BERT consider different modalities equally important, which ignores the huge capability difference between the pre-trained models and others. Moreover, equal multimodal fusion might introduce noise and limit the performance of these systems. To this end, we propose a superposition multimodal fusion framework to strengthen the importance of language modality with nonverbal information transmission. With the superposition strategy, language information will fuse with nonverbal information at different stages, which allows the model to fully learn language modality information. Besides, to ensure that language information is the main content of the fusion, we propose a comparison loss to help the reinforced cross-modal attention module better transfer nonverbal information to language features. Extensive experiments are conducted to compare with baselines and the results demonstrate that our method achieves the superior performance.