Multi-modal sarcasm detection based on emotion perception and cross-modality attention fusion

Abstract

Sarcasm is a rhetorical device commonly used in social media and is prevalent on some social platforms, such as Twitter and Reddit, to dismiss, criticize or ridicule people or events using metaphors and exaggeration. With the rapid growth of social media and internet technology, the way people express their emotions and feelings is not limited to text. Therefore, a multi-modal sarcasm detection task is crucial to understanding people’s real feelings and beliefs. However, most existing models often use implicit fusion and do not significantly align the emotions between modalities explicitly, neglecting the significant role of emotional words in sarcasm detection. In this paper, a model was proposed based on emotion perception and cross-modality attention fusion for multi-modal sarcasm detection. Specifically, an external emotional knowledge was introduced for emotional information enhancement. In addition, the dual-channel BERT-based module and cross-modality interaction fusion were proposed based on an attention mechanism. The experimental results on a public multi-modal sarcasm detection dataset based on Twitter demonstrate the superiority of the proposed model.