Reliable Multiple-choice Iterative Algorithm for Crowdsourcing Systems

Abstract

The appearance of web-based crowdsourcing systems gives a promising solution to exploiting the wisdom of crowds efficiently in a short time with a relatively low budget. Despite their efficiency, crowdsourcing systems have an inherent problem in that responses from workers can be unreliable since workers are low-paid and have low responsibility. Although simple majority voting can be a solution, various research studies have sought to aggregate noisy responses to obtain greater reliability in results through effective techniques such as Expectation-Maximization (EM) based algorithms. While EM-based algorithms get the limelight in crowdsourcing systems due to their useful inference techniques, Karger et al. made a significant breakthrough by proposing a novel iterative algorithm based on the idea of low-rank matrix approximations and the message passing technique. They showed that the performance of their iterative algorithm is order-optimal, which outperforms majority voting and EM-based algorithms. However, their algorithm is not always applicable in practice since it can only be applied to binary-choice questions. Recently, they devised an inference algorithm for multi-class labeling, which splits each task into a bunch of binary-choice questions and exploits their existing algorithm. However, it has difficulty in combining into real crowdsourcing systems since it overexploits redundancy in that each split question should be queried in multiple times to obtain reliable results. In this paper, we design an iterative algorithm to infer true answers for multiple-choice questions, which can be directly applied to real crowdsourcing systems. Our algorithm can also be applicable to short-answer questions as well. We analyze the performance of our algorithm, and prove that the error bound decays exponentially. Through extensive experiments, we verify that our algorithm outperforms majority voting and EM-based algorithm in accuracy.