Crowdsourcing Contests Research Articles

Selection Process of Crowdsourcing Contests and the Roles of Incentives and Solvers’ Experience

Selection Process of Crowdsourcing Contests and the Roles of Incentives and Solvers’ Experience

Analyzing Participant Behaviors in Design Crowdsourcing Contests Using Causal Inference on Field Data

Crowdsourcing is the practice of getting ideas and solving problems using a large number of people on the Internet. It is gaining popularity for activities in the engineering design process ranging from concept generation to design evaluation. The outcomes of crowdsourcing contests depend on the decisions and actions of participants, which in turn depend on the nature of the problem and the contest. For effective use of crowdsourcing within engineering design, it is necessary to understand how the outcomes of crowdsourcing contests are affected by sponsor-related, contest-related, problem-related, and individual-related factors. To address this need, we employ existing game-theoretic models, empirical studies, and field data in a synergistic way using the theory of causal inference. The results suggest that participants' decisions to participate are negatively influenced by higher task complexity and lower reputation of sponsors. However, they are positively influenced by the number of prizes and higher allocation to prizes at higher levels. That is, an amount of money on any following prize generates higher participation than the same amount of money on the first prize. The contributions of the paper are: (a) a causal graph that encodes relationships among factors affecting crowdsourcing contests, derived from game-theoretic models and empirical studies, and (b) a quantification of the causal effects of these factors on the outcomes of GrabCAD, Cambridge, MA contests. The implications of these results on the design of future design crowdsourcing contests are discussed.

Ethical Concerns of and Risk Mitigation Strategies for Crowdsourcing Contests and Innovation Challenges: Scoping Review.

BackgroundCrowdsourcing contests (also called innovation challenges, innovation contests, and inducement prize contests) can be used to solicit multisectoral feedback on health programs and design public health campaigns. They consist of organizing a steering committee, soliciting contributions, engaging the community, judging contributions, recognizing a subset of contributors, and sharing with the community.ObjectiveThis scoping review describes crowdsourcing contests by stage, examines ethical problems at each stage, and proposes potential ways of mitigating risk.MethodsOur analysis was anchored in the specific example of a crowdsourcing contest that our team organized to solicit videos promoting condom use in China. The purpose of this contest was to create compelling 1-min videos to promote condom use. We used a scoping review to examine the existing ethical literature on crowdsourcing to help identify and frame ethical concerns at each stage.ResultsCrowdsourcing has a group of individuals solve a problem and then share the solution with the public. Crowdsourcing contests provide an opportunity for community engagement at each stage: organizing, soliciting, promoting, judging, recognizing, and sharing. Crowdsourcing poses several ethical concerns: organizing—potential for excluding community voices; soliciting—potential for overly narrow participation; promoting—potential for divulging confidential information; judging—potential for biased evaluation; recognizing—potential for insufficient recognition of the finalist; and sharing—potential for the solution to not be implemented or widely disseminated.ConclusionsCrowdsourcing contests can be effective and engaging public health tools but also introduce potential ethical problems. We present methods for the responsible conduct of crowdsourcing contests.

Know When to Run: Recommendations in Crowdsourcing Contests

Crowdsourcing contests have emerged as an innovative way for firms to solve business problems by acquiring ideas from participants external to the firm. As the number of participants on crowdsourcing contest platforms has increased, so has the number of tasks that are open at any time. This has made it difficult for solvers to identify tasks in which to participate. We present a framework to recommend tasks to solvers who wish to participate in crowdsourcing contests. The existence of competition among solvers is an important and unique aspect of this environment, and our framework considers the competition a solver would face in each open task. As winning a task depends on performance, we identify a theory of performance and reinforce it with theories from learning, motivation, and tournaments. This augmented theory of performance guides us to variables specific to crowdsourcing contests that could impact a solver’s winning probability. We use these variables as input into various probability prediction models adapted to our context, and make recommendations based on the probability or the expected payoff of the solver winning an open task. We validate our framework using data from a real crowdsourcing platform. The recommender system is shown to have the potential of improving the success rates of solvers across all abilities. Recommendations have to be made for open tasks and we find that the relative rankings of tasks at similar stages of their time lines remain remarkably consistent when the tasks close. Further, we show that deploying such a system should benefit not only the solvers, but also the seekers and the platform itself.

Crowdsourcing and community engagement: a qualitative analysis of the 2BeatHIV contest

Background: As HIV cure research advances, it is important to engage local communities. Crowdsourcing may be an effective, bottom-up approach. Crowdsourcing contests elicit public contributions to solve problems and celebrate finalists. We examine the development of a crowdsourcing contest to understand public perspectives about HIV cure research.Methods: We used flyers, emails, online advertisement and phone calls to recruit a convenience sample of community members to participate in focus-group discussions. Participants developed a contest name, logo and hashtag. Qualitative analysis identified emergent themes in the focus group transcripts.Results: Seventy-one people participated in four focus groups. Emergent themes for HIV cure engagement included: (1) emphasising collective approaches to HIV cure; (2) dispelling myths to spur discussion; (3) using HIV cure as motivation for participation; and (4) using creative community engagement.Conclusion: Crowdsourcing contests may be useful for engaging local communities, developing culturally tailored awareness campaign messaging, and encouraging the public to learn more about HIV cure research.

Reversed loss aversion in crowdsourcing contest

Crowdsourcing contest is an effective means for firms to outsource tasks online to a large group of solvers in order to obtain creative solutions. This study investigates loss aversion of solvers in crowdsourcing contests. An experiment was conducted, and reversed loss aversion was identified for solvers, suggesting that solvers experience more happiness from the gains when they win the contest than the pain from the equivalent losses when they fail. The results also suggested that solvers experience higher reversed loss aversion for ideation contests than for expertise-based contests. We then investigated the effects of reversed loss aversion from a game theory perspective. The solutions showed that solvers’ effort level is greater with reversed loss aversion, while the optimal reward for the contest remains the same. In light of our findings, sponsors should conduct contests to solve ideation problems in which the solvers are loss averse reversed and will input more effort. Diversified business models could be developed by crowdsourcing platforms to match solvers and different crowdsourcing tasks.

Innovation and Crowdsourcing Contests

In an innovation contest, an organizer seeks solutions to an innovation-related problem from a group of independent agents. Agents, who can be heterogeneous in their ability levels, exert efforts to improve their solutions, and their solution qualities are uncertain due to the innovation and evaluation processes. In this chapter, we present a general model framework that captures main features of a contest, and encompasses several existing models in the literature. Using this framework, we analyze two important decisions of the organizer: a set of awards that will be distributed to agents and whether to restrict entry to a contest or to run an open contest. We provide a taxonomy of contest literature, and discuss past and current research on innovation contests as well as a set of exciting future research directions.

What Would an HIV Cure Mean to You? Qualitative Analysis from a Crowdsourcing Contest in Guangzhou, China.

Much HIV cure social science research has focused on high-income countries. Local key population perspectives, especially from people living with HIV (PLHIV), are needed in low- and middle-income countries. We organized an open contest soliciting responses from key populations, including PLHIV, about what a cure would mean in their lives. Tailored in-person events and social media were used to engage PLHIV, men who have sex with men (MSM), people who inject drugs, and local residents. We received 471 contributions over 4 months. Our thematic analysis showed that many people perceived that a cure would sterilize HIV and bring about new life for PLHIV. Many individuals believed a cure would decrease PLHIV discrimination and many MSM perceived a cure would decrease MSM discrimination. Some participants noted that a cure could help improve interpersonal relations, particularly with families and partners. Many individuals envisioned HIV cure as a panacea to bring about social stability. Some participants also anticipated changes in attitudes toward sex that may result in increased condomless sex. Our findings suggest a continued need for careful management of patient expectations and community engagement.

Wisdom of the Crowds: Crowd-Based Development of a Logo for a Conference Using a Crowdsourcing Contest.

Crowdsourcing methods have been widely used in business, but rarely in sexual health research. We evaluated a crowdsourced logo contest as part of an international human immunodeficiency virus conference. A logo crowdsourcing contest was conducted for the 20th International Acquired Immunodeficiency Syndrome Conference. Crowdsourcing has a group of individuals solve a task, often as part of an open contest. Semistructured interviews were conducted with key informants including contest organizers, contest contributors, and conference attendees. Interviews were transcribed verbatim and analysed using a content analysis approach. In total, 22 interviews were conducted with 3 contest organizers, 7 contest contributors, and 12 conference attendees. All individuals reported that the crowdsourced logo provided benefits beyond branding the conference, including creating a shared sense of purpose among diverse conference participants and explicitly demonstrating the strong community orientation of the conference. Conference organizers and attendees all reported deeper engagement because of the story of the Tanzanian artist who won the contest. Most conference attendees (11/12) preferred the crowdsourced logo compared with the previous company-organized logo, and all (22/22) supported the logo contest continuing. Implementing a logo contest was simple and relatively inexpensive. Stakeholders identified several ways to enhance crowdsourcing logo contest methods including wider promotion of the contest to encourage broader participation, greater transparency in the selection process, and a different prize structure which acknowledges the contribution of more contestants. We found that a crowdsourcing contest helped engage local and global communities in the lead-up before and during an international conference. Similar participatory events may be useful for sexual health conferences and research projects.

Effective Prize Structure for Simple Crowdsourcing Contests with Participation Costs

This paper studies the use of a multi-prize compensation scheme for "simple" contests where participation is costly and the quality of participants' contributions is a priori uncertain at the time they make their decision related to participating in the contest. The equilibrium analysis provided enables demonstrating not only that a multi-prize structure is often beneficial but also that in some cases the principal's expected profit is maximized when offering a second prize greater than the first prize. This may seem somehow counter-intuitive especially given that the principal's profit is only influenced by the quality of the best submission rather than the aggregate of submissions. Special emphasis is placed on the case where the contestants are a priori homogeneous which is often the case in real-life, whenever the contestants are basically a priori alike and the quality of their submissions is determined subjectively by some referee. Here, we manage to prove that a multi-prize structure is dominated by a winner-takes-all scheme, suggesting that the benefit in the multi-prize contest scheme fully derives from the heterogeneity between prospective contestants. Finally, we show that there is a class of settings where the use of the multi-prize crowdsourcing contest model enables achieving the performance of the fully cooperative model (which is an upper bound for the performance in any type of contest), and that for settings of this class the optimal prize allocation can be extracted through a set of linear equations.

Simultaneous vs. Sequential Crowdsourcing Contests

In a crowdsourcing contest, innovation is outsourced by a firm to an open crowd that compete in generating innovative solutions. Given that the projects typically consist of multiple attributes, how should the firm optimally design a crowdsourcing contest for such a project? We consider two alternative mechanisms. One is a simultaneous contest, where the best solution is chosen from the aggregate solutions simultaneously submitted by all contestants. The other is multiple sequential sub-contests, with each dedicated to one attribute and the contestants asked to build upon the best work in progress from previous sub-contests. It is intuitive that the sequential contest has the advantage of potentially creating a cooperative final solution contributed by different contestants. However, somewhat surprisingly, we show that the sequential contest may reduce the incentive for the crowd to exert effort. The comparison of the expected best performances in the two contests depends on the project's characteristics. For example, if the project is relatively difficult (resp., easy), the sequential (resp., simultaneous) contest is optimal. If the number of contestants is large enough (resp., small enough), the sequential (resp., simultaneous) contest is optimal. When the firm optimally allocates the total prize to sub-contests, if the difficulty is sufficiently different across attributes, the sequential contest performs better by motivating contestants to make more efforts than the simultaneous contest. Otherwise, the simultaneous contest may perform better. We also consider the expected average performance as an alternative criterion and extend the model to account for contestants with heterogeneous cost functions and starting points.

The Role of Feedback in Dynamic Crowdsourcing Contests: A Structural Empirical Analysis

In this paper, we empirically examine the impact of performance feedback on the outcome of crowdsourcing contests. We develop a dynamic structural model to capture the economic processes that drive contest participants’ behavior, and estimate the model using a rich data set collected from a major online crowdsourcing design platform. The model captures key features of the crowdsourcing context, including a large participant pool, entries by new participants throughout the contest, exploitation (revision of previous submissions) and exploration (radically novel submissions) behaviors by contest incumbents, and the participants’ strategic choice among these entry, exploration, and exploitation decisions in a dynamic game. We find that the cost associated with exploratory actions is higher than the cost associated with exploitative actions. High-performers prefer the exploitative strategy, while low-performers tend to make fewer follow-up submissions and prefer the exploratory strategy. Using counter-factual simulations, we compare the outcome of crowdsourcing contests under alternative feedback disclosure policies and award levels. Our simulation results suggest that the full feedback policy (providing feedback throughout the contest) may not be optimal. The late feedback policy (providing feedback only in the second half of the contest) leads to a better overall contest outcome.

Participation recommendation system for crowdsourcing contests

We propose a novel participation recommendation approach for crowdsourcing contests including probabilistic modeling of contest participation and winner determination. Our method estimates the winning and participation probability of each worker and offers ranked lists of recommended contests. Since there is only one winner in most contests, standard recommendation techniques fail to estimate the accurate winning probability using only the extremely sparse winning information of completed contests. Our solution is to utilize contest participation information and features of workers and contests as auxiliary information. We use the concept of a transfer learning method for matrices and a feature-based matrix factorization method. Experiments conducted using real crowdsourcing contest datasets show that the use of auxiliary information is crucial for improving the performance of contest recommendation, and also reveal several important common skills.

Optimal crowdsourcing contests

We study the design and approximation of optimal crowdsourcing contests. Crowdsourcing contests can be modeled as all-pay auctions because entrants must exert effort up-front to enter. Unlike all-pay auctions where a usual design objective would be to maximize revenue, in crowdsourcing contests, the principal only benefits from the submission with the highest quality. We give a theory for optimal crowdsourcing contests that mirrors the theory of optimal auction design: the optimal crowdsourcing contest is a virtual valuation optimizer (the virtual valuation function depends on the distribution of contestant skills and the number of contestants). We also compare crowdsourcing contests with more conventional means of procurement. In this comparison, crowdsourcing contests are relatively disadvantaged because the effort of losing contestants is wasted. We show that the total wasted effort is at most the maximum effort which implies that crowdsourcing contests are a 2-approximation to an idealized model of conventional procurement.

Crowdsourcing contests: understanding the effect of competitors’ participation history on their performance

Crowdsourcing contests have become increasingly important and prevalent with the ubiquity of the Internet. Designing efficient crowdsourcing contests is not possible without the deep understanding of the factors affecting individuals’ continuous participation and their performance. Prior studies have mainly focused on identifying the effect of task-specific, environment-specific, organisation-specific, and individual-specific factors on individuals’ performance in crowdsourcing contests. And to our knowledge, there are no or very few studies on evaluating the effect of individuals’ participation history on their performance. This paper aims to address this research gap using a data set from TopCoder. This study derives competitors’ participation history factors, such as participation frequency, participation recency, winning frequency, winning recency, tenure, and last performance to construct models depicting effects of these factors on competitors’ performance in online crowdsourcing contests. The research findings demonstrate that most of competitors’ participation history factors have significant effect on their performance. This paper also indicates that competitors’ participation frequency and winning frequency moderate the relationship between last performance and performance, and relationship between tenure and performance positively. On the other hand, individuals’ participation recency and winning recency moderate relationship between last performance and performance negatively, but have no significant effect on the relationship between tenure and performance.

A Division Strategy for Achieving Efficient Crowdsourcing Contest

A Division Strategy for Achieving Efficient Crowdsourcing Contest

Winner-Take-All Crowdsourcing Contests with Stochastic Production

We study winner-take-all contests for crowdsourcing procurement in a model of costly effort and stochastic production. The principal announces a prize value P, agents simultaneously select a level of costly effort to exert towards production, yielding stochastic quality results, and then the agent who produces the highest quality good is paid P by the principal. We derive conditions on the probabilistic mapping from effort to quality under which this contest paradigm yields efficient equilibrium outcomes, and demonstrate that the conditions are satisfied in a range of canonical settings.