Workforce-efficient consensus in crowdsourced transcription of biocollections information

Abstract

Crowdsourcing can be a cost-effective method for tackling the problem of digitizing historical biocollections data, and a number of crowdsourcing platforms have been developed to facilitate interaction with the public and to design simple “Human Intelligence Tasks”. However, the problem of reaching consensus on the response of the crowd is still challenging for tasks for which a simple majority vote is inadequate. This paper (a) describes the challenges faced when trying to reach consensus on data transcribed by different workers, (b) offers consensus algorithms for textual data, (c) implements a consensus-based controller to assign a dynamic number of workers per task and per field of a task, (d) proposes the use of clustering to further eliminate redundant work and (e) proposes enhancements of future crowdsourcing task assignments in order to minimize the need for complex consensus algorithms. Experiments using the proposed algorithms show multifold increase in the ability to reach consensus when compared to majority voting using exact string matching. In addition, the workforce controller is able to decrease the crowdsourcing cost per task and per task field by 37% and 50%, respectively, when compared to a strategy that uses a fixed number of workers. The accuracy of clustering is also good and it has the potential to increase the quality of tasks that can be clustered.