Theoretical guarantee for crowdsourcing learning with unsure option

Abstract

Crowdsourcing learning, in which labels are collected from multiple workers through crowdsourcing platforms, has attracted much attention during the past decade. This learning paradigm would reduce the labeling cost since crowdsourcing workers may be non-expert and hence less costly. On the other hand, crowdsourcing learning algorithms also suffer from being misled by incorrect labels introduced by imperfect workers. To control such risks, recently, it has been suggested to provide workers an additional unsure option during the labeling process. Although the benefits of the unsure option have been empirically demonstrated, theoretical analysis is still limited.In this article, a theoretical analysis of crowdsourcing learning with the unsure option is presented. Specifically, an upper bound of minimally sufficient number of crowd labels required for learning a probably approximately correct (PAC) classification model with and without the unsure option are given respectively. Next, a condition under which providing (or not providing) an unsure option to workers is derived. Then, the theoretical results are extended to guide non-identical label options (with or without unsure options) to different workers. Last, several useful applications are proposed based on theoretical results.