Crowdsourcing Contests Research Articles

Expertise Diversity in Online Crowdsourcing Contests: Impacts of Creative Stars on Team Performance

Expertise Diversity in Online Crowdsourcing Contests: Impacts of Creative Stars on Team Performance

Understanding the Effect of Environment- and Organization-Specific Factors on Sustained Crowdsourcing Contest Participation

ABSTRACT Crowdsourcing contests are vital for organizational problem-solving and decision-making. Continued participation in these contests ensures platform success, yet environment- and organization-specific factors are underexplored. This study addresses this gap by examining how team ties, organization exposure, and experience with the organization impact sustained participation in crowdsourcing contests. An empirical study of 9,596 individuals across 138 Kaggle.com contests reveals these factors significantly influence ongoing participation. To foster collaboration and long-term engagement, platforms should highlight information about sponsoring organizations. These findings enhance understanding of leveraging environment- and organization-specific factors for sustained crowdsourcing success.

When Do Marketing Ideation Crowdsourcing Contests Create Shareholder Value? The Effect of Contest Design and Marketing Resource Factors

Firms often use crowdsourcing contests to develop marketing ideas and solutions. Despite the prevalence—and unique aspects—of marketing ideation crowdsourcing contests (MICCs), there has been little examination of these contests’ shareholder wealth implications. Adopting a signaling perspective, the authors conduct an event study of 508 MICC announcements and find that they are associated with higher returns, but also higher idiosyncratic risk, indicating that investors hold a mixed view of such contests. Further, the authors consider how contest design factors and firm marketing resources may signal the cultivation of intellectual and relational market-based assets to shape their stock market impact, providing firms guidance to better design their MICCs. Specifically, they find that returns are enhanced when firms use professional (vs. general public) contests, specifically scoped contests, and contests using crowd judging (vs. expert panels), and for firms with stronger marketing capabilities. However, brand factors have mixed effects on returns, with a brand's relevant stature having a positive effect and its energized differentiation having a negative effect on returns. Product MICCs and generally scoped contests heighten the negative effects on risk, whereas marketing resources have no impact. Results offer implications for practitioners, including the finding that many MICC design choices commonly used in practice (i.e., general public contests and expert panels) are viewed less favorably by investors.

An Examination of the Dynamics of Crowdsourcing Contests: Role of Feedback Type

As more businesses are turning to crowdsourcing platforms for solutions to business problems, determining how to manage the sourcing contests based on their objectives has become critically important. Aside from static design parameters, such as the reward, a lever organizations can use to dynamically steer contests toward desirable goals is the feedback offered to contestants during the contest. In this study, first, using the psychology literature on the theory of feedback intervention, we classify feedback into two types: outcome and process. Second, using data from almost 12,000 design contests, we empirically examine the effects of the two types of feedback on the convergence and diversity of submissions following feedback interventions. We find that process feedback, providing goal-oriented information to contestants, fosters convergent thinking, leading to submissions that are similar. Outcome feedback, on the other hand, encourages divergent thinking, producing a greater variety of solutions to a problem. Furthermore, the effects are strengthened when the feedback is provided earlier in the contest rather than later. Based on our findings, we offer insights on how practitioners can strategically use an appropriate form of feedback to either generate greater diversity of solutions or efficient convergence to an acceptable solution.

Engaging Youth and Young Adults in the COVID-19 Pandemic Response via the "It's Our Turn" Crowdsourcing Contest.

As the coronavirus disease (COVID-19) pandemic continued to progress into 2021, appeals were made to take a stronger focus on the perceptions and practices of youth and young adults (YYAs) regarding COVID-19 mitigation, as well as the impact of mitigation strategies on the overall wellbeing of YYAs. In this paper, we describe our efforts to increase YYA engagement in Arizona's COVID-19 response by pairing embedded values from youth participatory action research (YPAR) with a crowdsourcing challenge contest design. The research protocol and implementation are described, followed by a thematic analysis of YYA-led messaging portrayed in 23 contest submissions and reflections formed by 223 community voters after viewing contest submissions. The authors conclude that a YYA-led crowdsourcing contest presented an opportunity to (a.) investigate the perceptions and behaviors of YYAs and their networks regarding the COVID-19 pandemic and mitigation efforts and (b.) amplify the voices of YYAs in the pandemic response. Perhaps even more importantly, this approach also offered insight into the exacerbated impact of the pandemic on YYA mental health and wellbeing, and the utility of YPAR in raising awareness of these effects among the contexts and social networks of YYAs.

Being a Solo Endeavor or Team Worker in Crowdsourcing Contests? It is a Long-term Decision You Need to Make

Workers in crowdsourcing are evolving from one-off, independent micro-workers to on-demand collaborators with a long-term orientation. They were expected to collaborate as transient teams to solve more complex, non-trivial tasks. However, collaboration as a team may not be as prevalent as possible, given the lack of support for synchronous collaboration and the "competition, collaboration but transient" nature of crowdsourcing. Aiming at unfolding how individuals collaborate as a transient team and how such teamwork can affect an individual's long-term success, this study investigates the individuals' collaborations on Kaggle, a crowdsourcing contest platform for data analysis. The analysis reveals a growing trend of collaborating as a transient team, which is influenced by contest designs like complexity and reward. However, compared with working independently, the surplus of teamwork in a contest varies over time. Furthermore, the teamwork experience is beneficial for individuals in the short term and long term. Our study distinguishes the team-related intellectual capital and solo-related intellectual capital, and finds a path dependency effect for the individual to work solely or collectively. These findings allow us to contribute insights into the collaborative strategies for crowd workers, contest designers, and platform operators like Kaggle.

Role of motivations, self-regulations, and perceived competitive intensity in solvers’ continuance intention in crowdsourcing contests

ABSTRACT Crowdsourcing contest has emerged as an innovative way to source ideas and solutions from online public in which only the solvers who make the winning bids receive payoffs. The uncertainty of the payoffs, arising from the competitive attribute of these contests, gives rise to the necessity to understand what sustains solvers’ participation. This study extends self-determination theory (SDT) to the crowdsourcing contest setting to investigate how controlled and autonomous motivation influence solver continuance intention through self-regulations (i.e. metacognitive strategies and emotion control) and when the effect of different motivation types on self-regulations are more or less prevalent. The results indicate that metacognitive strategies do mediate the relationship between motivation type and continuance intention. Meanwhile, as perceived competitive intensity increases, the positive effect of controlled motivation on metacognitive strategies is stronger but its negative effect on emotion control is also exacerbated. In addition, autonomous motivation promotes solvers’ metacognitive strategies and emotion control despite the varying degrees of perceived competitive intensity. These findings offer precious managerial insights into solvers’ continuance intention in crowdsourcing contests.

Recombinant Innovations: What Differentiates Experienced Designers in Open Crowdsourcing Contests?

Prior studies of online crowdsourcing platforms have found that experienced designers are more likely to win in crowdsourcing contests. However, what gives experienced designers an edge in these contests is not well understood. Our study seeks to understand what differentiates experienced designers from other participants, with a particular focus on how they leverage information in open design contests. We use a large-scale empirical analysis employing deep-learning algorithms and find that, experienced designers are more adept at integrating information from several prior highly-rated submissions from other designers within a contest, while less-experienced designers are more likely to excessively imitate individual prior highly-rated submissions. We also find that experienced designers whose submissions are closer in similarity to a synthesized image of several highly-rated prior submissions, are more likely to win. Our results are consistent with prior work on recombinant innovations which finds that a majority of innovations happen by a synthesis or recombination of prior innovations, and that designers, as they gain experience, learn optimal recombination strategies. Our findings provide new insights into the winning strategies of experienced designers in crowdsourcing platforms and have implications for the design of such markets.

Monetary Rewards and Self-Selection in Design Crowdsourcing Contests Managing Participation, Contribution Appropriateness, and Winning Trade-Offs

Monetary Rewards and Self-Selection in Design Crowdsourcing Contests Managing Participation, Contribution Appropriateness, and Winning Trade-Offs

The Role of Feedback in Dynamic Crowdsourcing Contests: A Structural Empirical Analysis

In this paper, we empirically examine the impact of performance feedback on the outcome of crowdsourcing contests. We develop a dynamic structural model to capture the economic processes that drive contest participants’ behavior and estimate the model using a detailed data set about real online logo design contests. Our rich model captures key features of the crowdsourcing context, including a large participant pool; entries by new participants throughout the contest; exploitation (revision of previous submissions) and exploration (radically novel submissions) behaviors by contest incumbents; and the participants’ strategic choice among these entry, exploration, and exploitation decisions in a dynamic game. Using counterfactual simulations, we compare the outcome of crowdsourcing contests under alternative feedback disclosure policies and award levels. Our simulation results suggest that, despite its prevalence on many platforms, the full feedback policy (providing feedback throughout the contest) may not be optimal. The late feedback policy (providing feedback only in the second half of the contest) leads to a better overall contest outcome. This paper was accepted by Gabriel Weintraub, revenue management and market analytics department.

Competing Tasks and Task Quality: An Empirical Study of Crowdsourcing Contests

An outcome of the rising popularity of crowdsourcing contest platforms has been an increase in the number of tasks available on a platform at the same time. These concurrent tasks compete for the attention of the same pool of solvers. There is some evidence that an increase in the number of competing tasks can reduce the number of solvers participating in a focal task. However, it is not clear exactly how competing tasks influence the quality of tasks, as represented by the best solutions they receive. Using data collected from a popular crowdsourcing platform, we investigate the effects of competing tasks and uncover important underlying mechanisms governing the relationship between competing tasks and task quality. Through analyses at the task, solver, and submission levels, we establish that competing tasks can influence task quality in two ways. First, the number of solvers in each task decreases when there are more competing tasks, and this reduction negatively impacts task quality. Second, the availability of a greater number of competing tasks leads to solvers participating in more tasks simultaneously. This multitasking in turn allows solvers to learn from other tasks, generate better solutions, and improve task quality. Interestingly, we find that these opposing forces offset each other, and that the overall solution quality of the focal task is not affected, despite the fact that an increase in the number of competing tasks reduces the number of solvers participating it.

Attracting High‐Quality Contestants to Contest in the Context of Crowdsourcing Contest Platform

Contests have long existed to source ideas and solutions. The emergence of crowdsourcing contest platforms greatly reduces the costs and broadens the reach of running contests by facilitating access to a large pool of labor, making contests a cost‐efficient approach to procure ideas/solutions even for employers of limited resources. As a result, the use of contests surges and there is a flux of small‐to‐medium size contests to these platforms. This study focuses on these new waves of small‐to‐medium‐size contests on crowdsourcing contest platforms and extends the literature in several ways: First, prior literature focuses extensively on contest prize, we also consider the effect of contest duration, which plays salient role on contest platforms. Our findings provide evidence that contest duration affects both the number and the quality of contestants attracted to a contest. Second, prior literature assumes contests exist independently and that the quality distribution of contestants is exogenous; accordingly, the number of contestants can be regarded as a proxy of contest performance. We show that contest design parameters have direct effects on the quality distribution of contestants. Specifically, increasing contest duration, while attracting more contestants, results in a downward shift on the quality distribution of the contestants attracted to the contest. Third, recognizing that each contest does not exist independently on crowdsourcing contest platforms, we also incorporate the number of competing contests in our model. Our results provide practical recommendations on how to set up successful contests and attract high‐quality contestants to a contest on a crowdsourcing contest platform.

Designing and Driving Crowdsourcing Contests in Large Public Service Organizations

Overview: When designed and driven efficiently, crowdsourcing can leverage the power of collective intelligence and yield innovative solutions. To date, the crowdsourcing literature has focused on exemplary corporate initiatives and less on crowdsourcing contests in public service organizations (PSOs), which have a diverse ecosystem. Existing literature has only sparsely studied the design aspect of crowdsourcing as a process. We explored crowdsourcing contests hosted by two large PSOs, Deutsche Bahn and Indian Railways, from a process perspective. We created a six-stage framework for crowdsourcing contests that other PSOs can use. We highlight the need for effective internal and external marketing to enhance the effectiveness of crowdsourcing in PSOs. With structured efforts, crowdsourcing contests can help PSOs cocreate impactful solutions by seamlessly blending internal and external knowledge and efforts.

Joint vs. Separate Crowdsourcing Contests

In a crowdsourcing contest, innovation is outsourced by a firm to an open crowd that competes in generating innovative solutions. Given that the projects typically consist of multiple attributes, how should the firm optimally design a crowdsourcing contest for such a project? We consider two alternative mechanisms. One is a joint contest, where the best solution is chosen from the joint solutions across attributes submitted by all contestants. The other is multiple separate parallel subcontests, with each dedicated to one attribute of the project. It is intuitive that the separate contest has the advantage of potentially creating a “cooperative” final solution contributed by different contestants. However, somewhat surprisingly, we show that the separate contest may reduce the incentive for the crowd to exert effort, resulting in the joint contest becoming the optimal scheme. The comparison of the expected best performances in the two contests depends on the project’s characteristics. For example, if contestants’ performances have a sufficiently high (respectively, low) level of randomness, the separate (respectively, joint) contest is optimal. If the number of contestants is large (respectively, small) enough, the separate (respectively, joint) contest is optimal. Moreover, we find that when the prize is endogenized, the optimal amount of the prize in the joint contest is no less than that in the separate contest. Finally, we extend the model to account for contestants with heterogeneous types. This paper was accepted by Gad Allon, operations management.

Social mechanisms in crowdsourcing contests: a literature review

ABSTRACT Crowdsourcing contests allow organisations to engage with an external workforce. Over the years, the phenomenon has attracted considerable research interest. In the present review, we synthesise the crowdsourcing contest literature by adopting the social mechanism lens. We begin by observing that stakeholders in crowdsourcing contests range from individuals (solvers) to large-scale organisations (seekers). Given that such vastly different entities interact during a crowdsourcing contest, it is expected that their behaviour, too, can have a varying range of predictors, such as individual and organisational factors. However, prior reviews on Crowdsourcing contests and crowdsourcing, in general, haven't explored the phenomenon's multi-layered nature. In addressing this gap, we synthesise 127 scholarly articles and identify underlying social mechanisms that explain key behavioural outcomes of seekers and solvers. Our review makes two specific contributions. First, we determine three distinct tensions that emerge from the key design decisions that might be at odds with the central principle of crowdsourcing contests: broadcast search for solutions from a long-tail of solvers. Second, we provide three recommendations for future research that, we believe, could provide a richer understanding of the seeker and solver behaviour.

Research Mentorship Crowdsourcing Contest: Toward Global Health Good

As one of the six consortia funded through the NIH Fogarty International Center, the University of North Carolina at Chapel Hill, Johns Hopkins University, Morehouse School of Medicine, and Tulane University (UJMT) Fogarty Global Health Fellows Program provides postdoctoral trainees and doctoral students support and training for yearlong research attachments at selected low- and middle-income countries. To understand the current impact of this global health research training opportunity, the UJMT consortium conducted a crowdsourcing contest to gather creative messages to promote and improve the impact of the program in partnership with Social Entrepreneurship to Spur Health. Between January and March 2019, the contest received a total of 47 submissions from 14 countries; 44 were found eligible. After judging, 19 (41.2%) submissions received a mean score of 7 or higher. We examined both textual and descriptive submissions for emergent themes and identified a range of facilitators who can be used to further improve the training programs; examples include exceptional mentorship, acquisition of research skills, career development, personal development, and multisite training opportunities. In conclusion, the crowdsourcing mentorship contest demonstrated the feasibility and acceptability of leveraging existing research networks for community engagement and how useful information can be effectively collected to highlight the effectiveness of a program and expand the reach.

The 4 youth by youth HIV self-testing crowdsourcing contest: A qualitative evaluation.

Crowdsourcing, a participatory approach to solicit ideas from a large group of diverse individuals, provides an opportunity to nurture youth participation in HIV self-testing service design. The objective of this study was to describe the responses to a crowdsourcing contest aimed at soliciting ideas on how to promote HIV self-testing (HIVST) among young people in Nigeria. The "4 Youth by Youth" HIV Self-Testing Crowdsourcing contest was an online and off-line contest that asked Nigerian youth (10-24 years old) for ideas in response to the following: How might we promote HIVST among young people in Nigeria? All data were collected and analyzed between October 2018, and June 2019. Ideas and perceptions generated from the crowdsourcing contest were qualitatively analyzed using thematic content analysis. Specifically, four reviewers analyzed whether the ideas generated were desirable (appealing to young people), feasible (easy to implement) and impactful (will significantly influence HIVST uptake among young people). A total of 903 entries were received in response to the contest prompt. Participants submitted entries in various forms: online form (39.7%), offline Dropbox (44.6%), email (6.1%) and WhatsApp (9.7%). Of the total entries, 85% (n = 769/903) entries were eligible and were scored as having either high, moderate or low level of feasibility, impact and desirability, on a 3-point Likert scale. A significant portion of the entries were given a score of 3 for feasibility (4.9%), desirability (7.1%), impact (3.0%) or a total overall score of 7 or more (8.2%). The three main themes that emerged from the entries include:1) Peer-to-peer distribution and leveraging on existing infrastructures 2) Youth-Oriented Branding of the HIVST Kit 3) Mobile platforms and social media technology. The "4 Youth by Youth" Self-Testing contest engaged a broad audience of young people to generate ideas and perspectives on how to promote HIVST. This process informed the development of youth innovated implementation strategies to increase uptake of HIVST among adolescents and youth at risk for HIV.

Crowd-Sourcing for Data Science and Quantifiable Challenges: Optimal Contest Design

We study the optimal design of a crowd-sourcing contest in settings where the output (from the contestants) is quantifiable -- for example, a data science challenge. This setting is in contrast to settings where the output is only qualitative and cannot be quantified in an objective manner -- for example, when the goal of the contest is to design a logo. The rapidly growing literature on the design of crowd-sourcing contests focuses largely on ordinal contests -- these are contests where contestants' outputs are ranked by the organizer and awards are based on the relative ranks. Such contests are ideally suited for the latter setting, where output is qualitative. For our setting (quantitative output), it is possible to design contests where awards are based on the actual outputs and not on their ranking alone -- thus, our space of contest designs includes ordinal contests but is significantly larger. We derive an easy-to-implement contest design for this setting and establish its optimality.

Research on Factors Affecting Solvers’ Participation Time in Online Crowdsourcing Contests

A crowdsourcing contest is one of the most popular modes of crowdsourcing and is also an important tool for an enterprise to implement open innovation. The solvers’ active participation is one of the major reasons for the success of crowdsourcing contests. Research on solvers’ participation behavior is helpful in understanding the sustainability and incentives of solvers’ participation in the online crowdsourcing platform. So, how to attract more solvers to participate and put in more effort is the focus of researchers. In this regard, previous studies mainly used the submission quantity to measure solvers’ participation behavior and lacked an effective measure on the degree of participation effort expended by a solver. For the first time, we use solvers’ participation time as a dependent variable to measure their effort in a crowdsourcing contest. Thus, we incorporate participation time into the solver’s participation research. With the data from Taskcn.com, we analyze how participation time is affected four key factors including task design, task description, task process, and environment, respectively. We found that, first, for task design, higher task rewards will attract solvers to invest more time in the participation process and the relationship between participation time and task duration is inverted U-shaped. Second, for task description, the length of the task description has a negative impact on participation time and the task description attachment will positively influence the participation time. Third, for the task process, communication and supplementary explanations in a crowdsourcing process positively affect participation time. Fourth, for environmental factors, the task density of the crowdsourcing platform and the market price of all crowdsourcing contests have respectively negative and positive effects on participation time.

Managing the Crowds: The Effect of Prize Guarantees and In-Process Feedback on Participation in Crowdsourcing Contests

Crowdsourcing contests are contests by which organizations tap into the wisdom of crowds by outsourcing tasks to large groups of people on the Internet. In an online environment often characterized by anonymity and lack of trust, there are inherent uncertainties for participants of such contests. This study focuses on crowdsourcing contests with winner-take-all prizes. During these contests, submissions are made sequentially and contest hosts can provide public in-process feedback to the submissions as soon as they are received. Drawing on the uncertainty literature, we examine how the use of prize guarantees (guaranteeing that a winner will be picked and paid) and in-process feedback (numeric ratings to individual designs and public textual comments during the contest) can help attract more submissions by influencing the various uncertainties faced by the contestants. We find that guaranteeing the prize increases submissions. The volume of in-process feedback (both numeric reviews and textual comments) has a positive effect on the number of submissions, and such an effect is bigger in contests without prize guarantees. In addition, providing highly positive or extremely negative feedback discourages overall future submissions, and the negative effect of highly positive feedback is mitigated in guaranteed contests.