Abstract
The identification of bots is an important and complicated task. The bot classifier "Botometer" was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer’s diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n = 4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies in social science using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed for identifying bots.
Highlights
Identifying bots on social media platforms like Twitter is a task that is as important as it is challenging
Even though many new bot detection methods are developed every year—outperforming Botometer in some cases—[15, 16, 18], we focus on Botometer as it is by far the most used tool in studies analyzing bots on Twitter [12], especially in social science [5, 7, 10]
The nearer the Receiver Operating Characteristics (ROC) curve (Fig 2) is to the upper-left corner and the more space is under the curve, the better Botometer can distinguish between bots and humans
Summary
Identifying bots (here understood as fully automated accounts) on social media platforms like Twitter is a task that is as important as it is challenging. Being able to estimate the authenticity of a Twitter discourse by detecting malicious botnets or identifying how many “real” (i.e. human) followers a politician has, is an important baseline for computational social scientists. This is relevant in the context of political communication but especially so when it comes to the detection of disinformation campaigns and strengthening a platform’s security. The False positive problem of automatic bot detection in social science research learning based classifiers [3]. The trained classifier Botometer [3, 4] has established itself in the social sciences to estimate the number of bots within a given dataset
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
