An online statistical quality control framework for performance management in crowdsourcing

Abstract

The big research topic has grown rapidly for the past decade due to the advent of the data deluge. Recent advancements in the literature leverage human computing power known as crowdsourcing to manage and harness big for various applications. However, human involvement in the completion of crowdsourcing tasks is an error-prone process that affects the overall performance of the crowd. Thus, controlling the quality of workers is an essential step for crowdsourcing systems, which due to unavailability of ground-truth for any task at hand becomes increasingly challenging. To propose a solution to this problem, in this study, we propose OSQC (Online Statistical Quality Control Framework) for managing the performance of workers in crowdsourcing. OSQC ascertains the worker's performance by using a statistical model and then leverages the traditional statistical control techniques to decide whether to retain a worker for crowdsourcing or to evict him. We evaluate our proposed framework on a real dataset and demonstrate how OSQC assists crowdsourcing to maintain its accuracy.