An optimization approach for worker selection in crowdsourcing systems

Abstract

Crowdsourcing has attracted considerable attention in recent years. A large amount of labelled data can be obtained efficiently and cheaply from the crowdsourcing platform. Therefore, providing a complete, feasible and efficient paradigm for marking unlabelled data in crowdsourcing is in demand. Since the quality of crowd workers directly influences the quality of the labelled data, an optimization model subjecting to quality constraints is built in this paper, aiming to minimize cost function designed based on worker quality. Cost-controlled and quality-assured worker selection algorithms are proposed to solve crowd-labelling problems for static and dynamic crowdsourcing systems separately. Numerical examples based on collected data validate the feasibility of the proposed algorithms. Through extensive simulations, it is demonstrated the proposed method can keep a trade-off between the total cost and the accuracy of label inference which is assured via worker selection. In addition, the proposed approach can also adjust the parameters flexibly according to requesters’ demand for the accuracy or budget requirements, which shows its practicality.