Is Multi-Modal Necessarily Better? Robustness Evaluation of Multi-Modal Fake News Detection

Abstract

The proliferation of fake news and its serious negative social influence push fake news detection methods to become necessary tools for web managers. Meanwhile, the multi-media nature of social media makes multi-modal fake news detection popular for its ability to capture more modal features than uni-modal detection methods. However, current literature on multi-modal detection is more likely to pursue the detection accuracy but ignore the robustness (the detection ability in the case of abnormality and malicious attack) of the detector. To address this problem, we propose a comprehensive robustness evaluation of multi-modal fake news detectors. In this work, we simulate the attack methods of malicious users and developers, i.e., posting fake news and injecting backdoors. Specifically, we evaluate multi-modal detectors with five adversarial and two backdoor attack methods. Experiment results imply that: (1) The detection performance of the state-of-the-art detectors degrades significantly under adversarial attacks, e.g., BDANN's detection accuracy on malicious news drops by 47% compared to normal, even worse than general detectors (Att-RNN); (2) Most multimodal detectors are more vulnerable to visual modality than textual modality; (3) Backdoor attacks on popular events news severely degrade detectors (accuracy dropped by an average of 20%); (4) These detectors degrade more (another 2% reduction in accuracy) when subjected to multi-modal attacks; (5) Defense methods will improve the robustness of multi-modal detectors, but cannot fully resist the effects of malicious attacks.