Escaping the neutralization effect of modality features fusion in multimodal Fake News Detection

Abstract

Fake news spreads at unprecedented speeds through online social media, raising many concerns and negative impacts on a variety of domains. To control this issue, Fake News Detection (FND) naturally becomes the chief task while multimodal FND has recently attracted more attention due to the fast-growing multimodal content in online social media. Commonly, the existing multimodal FND methods directly fuse the unimodal features into the multimodal features before detecting the veracity, however, such methods result in the neutralization effect problem caused by the veracity conflict between different modalities. To alleviate this problem, we propose a novel multimodal FND method, namely MultImodal fake NEws detectoR with Unimodal Veracity Signals (Miner-uvs). The basic idea of Miner-uvs is to incorporate an unimodal veracity classifier that can predict the veracity labels of each modality, and then use the unimodal veracity labels to weigh the unimodal features, so as to achieve more discriminative multimodal features. Because the unimodal veracity labels are unknown, we formulate the unimodal veracity classification as the Positive and Unlabeled (PU) learning problem; and solve it by using a variational PU learning method with a contrastive multimodal alignment regularization. To evaluate the effectiveness of Miner-uvs, we compare it with dozens of existing multimodal FND methods across benchmark datasets. Experimental results demonstrate that Miner-uvs consistently outperforms the existing competitors.