Crowd Labor Research Articles

Platform Rhetoric and Fan Labour as the Building Blocks of <em>LEGO Ideas</em>

Platform Rhetoric and Fan Labour as the Building Blocks of <em>LEGO Ideas</em>

Pterodactyl: Two-Step Redaction of Images for Robust Face Deidentification

Redacting faces in images is trivial when the number of faces is small and the annotator is trusted. For large batches, automated face detection has been the only currently viable solution, yet even the best ML-based solutions have error rates that would be unacceptable for sensitive applications. Crowd-based face detection/redaction systems exist, yet the process and the cost make them not feasible. We present Pterodactyl, a system for detecting (and redacting) faces at scale. It uses the AdaptiveFocus filter, which splits the image into smaller regions and uses machine learning to select a median filter for each region to hide the facial identities in the image while simultaneously allowing those faces to be detectable by crowd workers. The filter uses a convolutional neural network trained on images associated with the median filter level that allows detection and prevents identification. This filter allows Pterodactyl to achieve human-level detection with just 14% crowd labor as another recent crowd-based face detection/redaction system (IntoFocus). Our evaluation found that the redaction accuracy was higher than a commercial machine-based application and on par with IntoFocus while requiring 86% less crowd work (number of comparable tasks).

Voter experience and ballot language framing effects: Evidence from a survey experiment

AbstractObjectiveDiscover the extent to which habitual voting may alter the effects of framing in the language of ballot measures.MethodsOnline survey was conducted using Amazon Mechanical Turk crowd labor service. Participants were presented with two ballot questions with two separate sets of language, randomized so that each participant had an equal chance of seeing either version. Logistic regression was used to analyze responses.ResultsSupport for one measure was drastically reduced with the change in question framing; for the other, the results were inconclusive. None of the models exhibited any evidence that habitual voting mitigated framing effects.ConclusionExperienced voters are no more immune to framing effects than inexperienced voters.

Crowd Labor Markets as Platform for Group Decision and Negotiation Research: A Comparison to Laboratory Experiments

Crowd labor markets such as Amazon Mechanical Turk (MTurk) have emerged as popular platforms where researchers can relatively inexpensively and easily run web-based experiments. Some work even suggests that MTurk can be used to run large-scale field experiments in which groups of participants interact synchronously in real-time such as electronic markets. Besides technical issues, several methodological questions arise and lead to the question of how results from MTurk and laboratory experiments compare. Our data shows comparable results between MTurk and a standard lab setting with student subjects in a controlled environment when running rather simple individual decision tasks. However, our data shows stark differences in results between the experimental settings for a rather complex market experiment. Each experimental setting—lab and MTurk—has its own benefits and drawbacks; which of the two settings is better suited for a specific experiment depends on the theory or artifact to be tested. We discuss potential causes for differences (language understanding, education, cognition and context) that we cannot control for and provide guidance for the selection of the appropriate setting for an experiment. In any case, researchers studying complex artifacts like group decisions or markets should not prematurely adopt MTurk based on extant literature regarding comparable results across experimental settings for rather simple tasks.

Value Sensitive Design for Peer Production Systems: Mediating Social Interactions

Advances in information and communication technologies (ICT) have fueled the rise of the peer production community, a type of socio-technical system that self-organizes to deliver products or services created in a distributed manner. These novel modes of production, examples of which include such successful projects as Wikipedia, GNU/Linux, or Project Gutenberg, defy the expectations of standard organizations - they open rather than enclose knowledge and they motivate contribution through social rather than monetary incentives. Optimists reveling in these new possibilities for collaboration forecast that peer production will take over standard companies and reduce existing inequalities thanks to its ability to unleash the innovative potential of individuals and social groups in an unprecedented manner. Yet, what is called "peer production" often entails a variety of social structures and activities that do not have much in common. Some of these - crowd labor for example - have little potential to change the existing patterns of hierarchical, closed organizations. We argue here that what distinguishes those peer production systems that have the potential to revolutionize economies from other ICT mediated collaborations are the values that such systems embody, pursue, and promote (such as openness, equality, or freedom).

Straight from the Source? Media Framing of Creative Crowd Labor and Resultant Ethical Concerns

Increasing numbers of marketers are turning to the crowd—members of the public engaged with brands via the Internet—to develop marketing and advertising campaigns. Some marketers use social media to connect directly with customers, while others use crowdsourcing agencies to harness the power of crowd labor. As more members of the public become aware of creative crowdsourcing, they look to the media to understand more about it. As a result, it is important to examine how the media currently frame creative crowdsourcing to the public, particularly when numerous ethical issues about crowdsourcing have been identified (e.g. work is devalued as crowdworkers earn little money and no benefits for their work). This study examines media coverage of creative crowdsourcing to examine how benefits and challenges are presented. Informed by a framework developed by Swain (Understanding and communicating science: new agendas in communication, Routledge, London, pp 209–232, 2009), results indicate that most media coverage focuses solely on benefits to the industry: benefits to the ‘crowd’ are rarely discussed, yet drawbacks to the crowd are mentioned more frequently than drawbacks for the industry. This provides a skewed vision of what creative crowdsourcing is and may affect participation in the practice.

Adaptive Performance Optimization over Crowd Labor Channels

We describe a system which monitors the performance of labor channels within a crowdsourcing platform in an online manner. This allows us to automatically determine if and when to switch between labor channels in order to improve overall performance of crowd tasks.

Hybrid curation of gene-mutation relations combining automated extraction and crowdsourcing.

Background: This article describes capture of biological information using a hybrid approach that combines natural language processing to extract biological entities and crowdsourcing with annotators recruited via Amazon Mechanical Turk to judge correctness of candidate biological relations. These techniques were applied to extract gene– mutation relations from biomedical abstracts with the goal of supporting production scale capture of gene–mutation–disease findings as an open source resource for personalized medicine. Results: The hybrid system could be configured to provide good performance for gene–mutation extraction (precision ∼82%; recall ∼70% against an expert-generated gold standard) at a cost of $0.76 per abstract. This demonstrates that crowd labor platforms such as Amazon Mechanical Turk can be used to recruit quality annotators, even in an application requiring subject matter expertise; aggregated Turker judgments for gene–mutation relations exceeded 90% accuracy. Over half of the precision errors were due to mismatches against the gold standard hidden from annotator view (e.g. incorrect EntrezGene identifier or incorrect mutation position extracted), or incomplete task instructions (e.g. the need to exclude nonhuman mutations). Conclusions: The hybrid curation model provides a readily scalable cost-effective approach to curation, particularly if coupled with expert human review to filter precision errors. We plan to generalize the framework and make it available as open source software.Database URL: http://www.mitre.org/publications/technical-papers/hybrid-curation-of-gene-mutation-relations-combining-automated

Crowdsourcing Multi-Label Classification for Taxonomy Creation

Recent work has introduced CASCADE, an algorithm for creating a globally-consistent taxonomy by crowdsourcing microwork from many individuals, each of whom may see only a tiny fraction of the data (Chilton et al. 2013). While CASCADE needs only unskilled labor and produces taxonomies whose quality approaches that of human experts, it uses significantly more labor than experts. This paper presents DELUGE, an improved workflow that produces taxonomies with comparable quality using significantly less crowd labor. Specifically, our method for crowdsourcing multi-label classification optimizes CASCADE’s most costly step (categorization) using less than 10% of the labor required by the original approach. DELUGE’s savings come from the use of decision theory and machine learning, which allow it to pose microtasks that aim to maximize information gain.

Crowdsourcing 101

Crowdsourcing is a problem-solving approach that taps the knowledge, energy, and creativity of a global, online community. Like marketing, crowdsourcing originated and continues to evolve in the private sector. Health communicators and social marketers can use crowdsourcing across the research-development-dissemination process. This article provides an overview of crowdsourcing and how it can be used to speed up and enhance social marketing and health communication. Nielsen's 1/9/90% model is presented as a guide for engaging appropriate crowds for tasks throughout the development process. The four Fs that motivate online contributors--Fun, Feeling good (fulfillment), Fame, and Fortune--are also presented as ways of incentivizing crowd engagement and matching the incentive to the task at hand. Crowdsourcing resources, such as curating agencies, websites, and crowd labor markets, can be tremendous force multipliers. If done strategically, crowdsourcing has the promise of giving well-researched and creative social marketing results for less money and in less time than traditional methods.

Putting “Human Crowds” in the Loop of Bibliography Evaluation: A Collaborative Working Environment for CSCW Publications

The current impact of financial crisis on societal and scientific frameworks has raised the need to harvest and evaluate vast volumes of data in a socially-mediated interaction context to reduce knowledge gaps and accelerate innovation at a global scale. Existing mechanisms are inefficient for a single human to classify and transform data into knowledge patterns from a large number of publications. This time-consuming and computationally difficult activity requires a substantial cognitive effort grounded on scientific metrics and theoretical foundations to produce quality metadata through different knowledge representations. This paper reports on a work in progress community self-organizing bibliographic information system for semantic analytics focused on what scientific research data mean, and how they can be best interpreted through a division of intellectual labor among social, computer, and citizen scientists. Such a crowd labor ecosystem should be not restricted to traditional bibliometric approaches, attempting to uncover patterns and trends in publication data sets whilst intellectual connections can be examined to demonstrate how a scientific field is conceptually, intellectually, and socially structured.

Analyzing Crowd Labor and Designing Incentives for Humans in the Loop

Organizations increasingly use crowdsourcing techniques to solve problems outside the reach of pure automatic computation. People's participation and willingness to contribute are thus critical issues that organizations and software designers must take into account when developing crowdsourcing solutions. Many studies examine motivations and incentives, often analyzing specific crowdsourcing initiatives or groups of contributors. Building on this evidence, and on findings in organizational theory and experimental economics, the authors' framework helps developers and organizations analyze a concrete situation and identify adequate sets of incentives before designing crowdsourcing platforms.

Enabling trust in crowd labor relations through identity sharing

While online Crowdsourcing marketplaces provide a powerful avenue for facilitating new forms of information- driven micro-labor, their practical value is significantly reduced by worker “spam” and employer fraud. We hypothesize anonymity of parties is a major source of these problems, and we thus propose a human-centric solution: encourage employers and workers to voluntarily de- anonymize in order to reap a potential benefit of more productive and profitable labor interactions. To facilitate voluntary identity sharing, we have built a prototype identity management application allowing individuals to associate their crowdsourcing worker/employer identities to their public profiles on social network sites. By providing a vehicle for identity sharing, the prototype provides the foundation for a future user study of employers and workers engaged in known-identity crowd labor relationships.