A real time anti-spamming system in crowdsourcing platform

Abstract

Crowdsourcing platforms such as Amazon Mechanical Turk, CrowdFlower and Clickworker have rapidly gained the popularity as they allow outsourcing tasks to a group of people through an open call. By combining the power of human intelligence tasks can be solved in cheap cost and short completion time in crowdsourcing platforms. However quality control still remains a challenge since spammers in crowdsourcing platforms may submit unreliable answers. To identify the spammer involved in crowdsourcing projects, this paper analyzes labelers' behavioral characteristics (e.g., response time) in the Baidu Crowdsourcing Platform (BCP). By doing so, we gain insight into the complex interactions between the labelers and the projects conducted in crowdsourcing platforms. Based on the above analysis results, a probabilistic model is furthermore constructed to obtain objective measures of labeler reliability. In particular, our approach attempts to combine the labeler behavioral characteristics to identify spammer. Furthermore, we develop a real-time anti-spamming system so as to identify spammers while projects are conducted in BCP. Finally, we implement comprehensive experiments to evaluate our approach on the real projects that have been published in BCP. The experimental results and performance comparisons with baseline models show that our method can identify spammers timely and effectively.