Bots are the new fraud: A post-hoc exploration of statistical methods to identify bot-generated responses in a corrupt data set

Abstract

Bots may be increasingly capable of emulating individual human behaviors in online survey research. Although effective methods for bot-detection exist, it is unclear whether or not these methods translate for use outside simulation. There are complexities in practical application that complicate the use of statistical methods, as the true number of bot-generated responses must be inferred. This paper presents a preliminary translational model that combines a manual flagging procedure and statistical methods for bot-detection (Mahalanobis distance and person-total correlation). Cases (n = 1306) from a real, bot-corrupted data were sorted via a manual flagging procedure as either “likely bot” (n = 402) or “likely human” (n = 904). Statistical methods for bot-detection were applied. Findings revealed significantly greater Mahalanobis distances among individual responses in the “likely human” group, compared to the “likely bot” group (p < 0.001; r = 0.32); no significant differences were found with person-total-correlation. For both groups, internal consistency was strongly aligned with published human data. Results suggest bots may be increasingly capable of replicating nuanced individual response behaviors in survey data. Authors explore the potential uses and limitations of the proposed model, while emphasizing the importance of improving the translational capacity of bot-detection methods.