Utilizing Human Intelligence in a Crowdsourcing Marketplace for Big Data Processing

Abstract

Crowd sourcing emerging as a distributed problem-solving model can dispatch tasks to a large number of human workers. The workers' human intelligence can be utilized to execute tasks which cannot be efficiently performed by computer-based systems. A crowd sourcing environment supports various data processing tasks, for example, data collection, data annotation, data validation, data classification, and natural language processing. In particular, the crowd sourcing environment can be applied for big data processing as well. In this paper, we propose Human-Intelligence-as-a-Service (HIaaS) which is a cloud computing service model utilizing human intelligence in a crowd sourcing marketplace to process big data. We also propose an optimization model based on stochastic programming for provisioning human workers in a HIaaS-based marketplace. Numerical studies are performed to evaluate the optimization model. A video rating system handling a big data set is an illustrative example analyzed in the studies. The results show that the model can efficiently reduce the cost to provision workers for processing a big data job.