Design and Analysis of Incentive and Reputation Mechanisms for Online Crowdsourcing Systems

Abstract

Today, online crowdsourcing services like Amazon Mechanical Turk, UpWork, and Yahoo! Answers are gaining in popularity. For such online services, it is important to attract “workers” to provide high-quality solutions to the “tasks” outsourced by “requesters.” The challenge is that workers have different skill sets and can provide different amounts of effort. In this article, we design a class of incentive and reputation mechanisms to solicit high-quality solutions from workers. Our incentive mechanism allows multiple workers to solve a task, splits the reward among workers based on requester evaluations of the solution quality, and guarantees that high-skilled workers provide high-quality solutions. However, our incentive mechanism suffers the potential risk that a requester will eventually collects low-quality solutions due to fundamental limitations in task assigning accuracy. Our reputation mechanism ensures that low-skilled workers do not provide low-quality solutions by tracking workers’ historical contributions and penalizing those workers having poor reputations. We show that by coupling our reputation mechanism with our incentive mechanism, a requester can collect at least one high-quality solution. We present an optimization framework to select parameters for our reputation mechanism. We show that there is a trade-off between system efficiency (i.e., the number of tasks that can be solved for a given reward) and revenue (i.e., the amount of transaction fees), and we present the optimal trade-off curve between system efficiency and revenue. We demonstrate the applicability and effectiveness of our mechanisms through experiments using a real-world dataset from UpWork. We infer model parameters from this data, use them to determine proper rewards, and select the parameters of our incentive and reputation mechanisms for UpWork. Experimental results show that our incentive and reputation mechanisms achieve 98.82% of the maximum system efficiency while only sacrificing 4% of revenue.