Effective Result Inference for Context-Sensitive Tasks in Crowdsourcing

Abstract

Effective result inference is an important crowdsourcing topic as workers may return incorrect results. Existing inference methods assign each task to multiple workers and aggregate the results from these workers to infer the final answer. However, these methods are rather ineffective for context-sensitive tasks (\(\mathtt{CSTs}\)), e.g., handwriting recognition, due to the following reasons. First, each \(\mathtt{CST}\) is rather hard and workers usually cannot correctly answer a whole \(\mathtt{CST}\). Thus a task-level inference strategy cannot achieve high-quality results. Second, a \(\mathtt{CST}\) should not be divided into multiple subtasks because the subtasks are correlated with each other under certain contexts. So a subtask-level inference strategy cannot achieve high-quality results as it neglects the correlation between subtasks. Thus it calls for an effective result inference method for \(\mathtt{CSTs}\). To address this challenge, this paper proposes a smart assembly model (\(\mathtt{SAM}\)), which can assemble workers’ complementary answers in the granularity of subtasks without losing the context information. Furthermore, we devise an iterative decision model based on the partially observable Markov decision process, which can decide whether we need to ask more workers to get better results. Experimental results show that our method outperforms state-of-the-art approaches.