Design and evaluation of crowdsourcing platforms based on users’ confidence judgments

Abstract

Crowdsourcing deals with solving problems by assigning them to a large number of non-experts called crowd using their spare time. In these systems, the final answer to the question is determined by summing up the votes obtained from the community. The popularity of these systems has increased by facilitating access for community members through mobile phones and the Internet. One of the issues raised in crowdsourcing is how to choose people and how to collect answers. Usually, users are separated based on their performance in a pre-test. Designing the pre-test for performance calculation is challenging; The pre-test questions should be selected to assess characteristics in individuals that are relevant to the main questions. One of the ways to increase the accuracy of crowdsourcing systems is by considering individuals’ cognitive characteristics and decision-making models to form a crowd and improve the estimation of their answer accuracy to questions. People can estimate the correctness of their responses while making a decision. The accuracy of this estimate is determined by a quantity called metacognition ability. Metacoginition is referred to the case where the confidence level is considered along with the answer to increase the accuracy of the solution. In this paper, by both mathematical and experimental analysis, we would answer the following question: Is it possible to improve the performance of a crowdsourcing system by understanding individuals’ metacognition and recording and utilizing users’ confidence in their answers?