Abstract
There have been many efforts to detect rumors using various machine learning (ML) models, but there is still a lack of understanding of their performance against different rumor topics and available features, resulting in a significant performance degrade against completely new and unseen (unknown) rumors. To address this issue, we investigate the relationship between ML models, features, and rumor topics to select the best rumor detection model under specific conditions using 13 different ML models. Our experiment results demonstrate that there is no clear winner among the ML models in all different rumor topics with respect to the detection performance. To overcome this problem, a possible way is to use an ensemble of ML models. Although previous work presents an improved detection of rumors using ensemble solutions (ES), their evaluation did not consider detecting unknown rumors. Further, they did not present nor evaluate the configuration of the ES to ensure that it indeed performs better than using a single ML model. Based on these observations, we propose to evaluate the use of an ES by examining their unknown rumor detection performance compared with single ML models but as well as different configurations of the ESes. Our experimental results using real-world datasets found that an ES of Random Forest, XGBoost and Multilayer perceptron overall produced the best F1 score of 0.79 for detecting unknown rumors, a significant improvement compared with a single best ML model which only achieved a 0.58 F1 score. We also showed that not all ESes are the same, with significantly degraded detection and large variations in performance when different ML models are used to construct the ES. Hence, it is infeasible to rely on any single ML model-based rumor detector. Finally, our solution also performed better than other recent detectors, such as eventAI and NileTMRG that performed similar to using a single ML model - making it a much more attractive solution to detect unknown rumors in practice.
Highlights
The spread of rumors is still prevalent today
We found that using a well-formulated ensemble solutions (ES) (e.g., ES combining Random Forest, XGBoost, and Multilayer perceptron denoted as RXM-ES) achieved 0.79 F1-score, compared to 0.58 using a single machine learning (ML) model and 0.62 using random ES model for detecting unknown rumors
Top-performing ML models are shown in Tables 3 and 4 for known and unknown rumors, respectively (The best model and its F1 score are highlighted in bold font style)
Summary
The spread of rumors is still prevalent today. To address this issue, various machine learning (ML) models have been proposed to detect rumors [1], [2]. We analyze the relationship between the rumor topics, features and ML models to understand the factors that affect their rumor detection performances, for new and previously unforeseen rumors (here, a rumor event is an instance of the rumor topic, and we will denote new and unforeseen ones as unknown rumors) To achieve this goal, we conduct several experiments using two publicly available datasets: (1) PHEME [6], and (2) RumourEval2019 (RE2019)1 [7]. The RXMES achieved an average F1 score of 0.62, compared with 0.51 using a single ML model in our 10-fold cross-validation using the RE2019 dataset These results indicate that not all ESes could detect unknown rumors the same, and requires a careful selection of ML models to construct the ES (such as RXM-ES in our experiment).
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call