Adding compact parameter blocks to multimodal transformers to detect harmful memes

Abstract

Harmful content serves as a potent tool for spreading destructive ideologies and creating divisions among people, often exploiting factors like race, religion, and socioeconomic status. Social media platforms are especially conducive to such content, mainly through memes. Hence, the development of tools that can detect and eliminate harmful memes before they reach a wide audience and cause harm is a very important issue. However, this task represents a critical challenge, as memes can take various forms, typically combining images with text. Recent research has focused on building multimodal machine learning models, especially transformers-based models, to uncover harmful content within memes. Although these models have shown effectiveness, they often require substantial computational resources for training and incorporate additional information besides the textual and visual components of the memes, making them complex and challenging to reproduce. This study introduces a novel approach to enhance the performance of established multimodal transformer models without needing extra information. It works by adding a specific module called Compact Parameter Blocks (CPB) into the encoder segments of these models. This module enhances the quality of input data, leading to improvements in the model’s attention mechanism and in its overall regularization. This results in better performance across all relevant metrics. In this paper, we use this approach with two variations of multimodal models — VisualBERT (Visual Bidirectional Encoder Representations from Transformers) and VilBERT (Vision and Language Bidirectional Encoder Representations from Transformers) — both built on canonical transformers to identify harmful content in memes across four different datasets. The experimental results demonstrated that this method outperforms several robust and complex approaches.