Multi-modal sarcasm detection based on Multi-Channel Enhanced Fusion model

Abstract

The voluminous quantity of data accessible on social media platforms offers insight into the sentiment disposition of individual users, where multi-modal sarcasm detection is often confounding. Existing sarcasm detection methods use different information fusion methods to combine information from different modalities but ignore hidden information within modalities and inconsistent information between modalities. Discovering the implicit information within the modalities and strengthening the information interaction between modalities is still an important challenge. In this paper, we propose a Multi-Channel Enhanced Fusion (MCEF) model for cross-modal sarcasm detection to maximize the information extraction between different modalities. Specifically, text extracted from images acts as a new modality in the front-end fusion models to augment the utilization of image semantic information. Then, we propose a novel bipolar semantic attention mechanism to uncover the inconsistencies among different modal features. Furthermore, a decision-level fusion strategy from a new perspective is devised based on four models to achieve multi-channel fusion, each with a distinct focus, to leverage their advantages and mitigate the limitations. Extensive experiments demonstrate that our model surpasses current state-of-the-art models in multi-modal sarcasm detection.