Context-aware result inference in crowdsourcing

Abstract

Many result inference methods have been proposed to address the quality-control problem in crowdsourcing. However, existing methods are ineffective for context-sensitive tasks (CSTs), e.g., handwriting recognition, translation, speech transcription, where context correlation within a task cannot be ignored for two reasons. Firstly, it is ineffective to crowdsource a whole CST (e.g., recognizing handwritten texts) and use task-level inference methods to infer the answer, because it is rather hard to correctly complete a whole complicated task. Secondly, although a CST is composed of a set of atomic subtasks (e.g., recognizing a handwritten word), it is unsuitable to split it into multiple subtasks and adopt a subtask-level inference algorithm to infer the result, because this will lose the context correlation (e.g., phrases) among subtasks and increase the difficulty to complete a task. Thus it calls for a new approach to handling CSTs. In this work, we study the result inference problem for CSTs and propose a context-aware inference algorithm. We design an inference algorithm by incorporating the context information. Furthermore, we introduce an iterative method to improve the quality. The results of experiments on real-world CSTs demonstrated the superiority of our approach compared with the state-of-the-art methods.