Re-Think Before You Share: A Comprehensive Study on Prioritizing Check-Worthy Claims

Abstract

The massive amount of misinformation spreading on the internet on a daily basis has enormous negative impacts on societies. Therefore, we need systems to help fact-checkers to combat misinformation and to raise public awareness of this important problem. In this article, we propose a hybrid model which combines bidirectional encoder representations from transformer (BERT) model with various features to prioritize claims based on their check-worthiness. Features we use include domain-specific controversial topics (CT), word embeddings (WE), part-of-speech (POS) tags, and others. In addition, we explore various ways of increasing labeled data size to effectively train the models, such as increasing positive (IncPos) samples, active learning (AL), and utilizing labeled data in other languages. In our extensive experiments, we show that our model outperforms all state-of-the-art models in test collections of Conference and Labs of Evaluation Forum (CLEF) CheckThat! Lab (CTL) 2018 and 2019. In addition, when positive samples are increased in the training set, our model achieves the best mean average precision (MAP) score reported so far for the test collection of CTL 2020. Furthermore, we show that cross-lingual training is effective for prioritizing Arabic and Turkish claims, but not for English.