Abstract
Crowdsourcing works by distributing many small tasks to large numbers of workers, yet the true potential of crowdsourcing lies in workers doing more than performing simple tasks—they can apply their experience and creativity to provide new and unexpected information to the crowdsourcer. One such case is when workers not only answer a crowdsourcer’s questions but also contribute new questions for subsequent crowd analysis, leading to a growing set of questions. This growth creates an inherent bias for early questions since a question introduced earlier by a worker can be answered by more subsequent workers than a question introduced later. Here we study how to perform efficient crowdsourcing with such growing question sets. By modeling question sets as networks of interrelated questions, we introduce algorithms to help curtail the growth bias by efficiently distributing workers between exploring new questions and addressing current questions. Experiments and simulations demonstrate that these algorithms can efficiently explore an unbounded set of questions without losing confidence in crowd answers.
Highlights
Crowdsourcing has emerged as a powerful new paradigm for accomplishing work by using modern communications technology to direct large numbers of people who are available to complete tasks to others who need large amounts of work to be completed [1,2,3,4]
We focus on cases, such as the synonym proposal task (SPT), where questions have binary answers, e.g, when workers are asked whether or not a link between two items should exist
Synonymy proposal is a good test application for the question sampling algorithms we study because workers can understand the question and are capable of proposing new questions
Summary
Crowdsourcing has emerged as a powerful new paradigm for accomplishing work by using modern communications technology to direct large numbers of people who are available to complete tasks (workers) to others who need large amounts of work to be completed (crowdsourcers) [1,2,3,4]. Crowdsourcing often focuses on tasks that are easy for humans to solve, but may be difficult for a computer. Parsing human written text can be a difficult task and optical character recognition systems may be unable to identify all scanned words [5,6,7]. The reCAPTCHA [8] system takes scanned images of text which were difficult for computers to recognize and hands them off to Internet workers for recognition. By having many people individually solve quick and easy tasks, reCAPTCHA is able over time to transcribe massive quantities of text. Crowdsourcing in general is especially important as a new vehicle for addressing problems of social good [9,10,11]
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
