Crowdwork Platforms Research Articles

Tragedy of the Commons in Crowd Work-Based Research

Academic scholars have leveraged crowd work platforms such as MTurk to conduct research and collect data, but the data quality crisis in crowd work has been an alarming phenomenon recently. Though prior studies have discussed data quality and validity issues in crowd work via surveys and experiments, they kind of neglected to explore the scholars’ and particularly the IRB's ethical concerns and the related policies in various ethical guidelines for crowd work-based research in these respects. In this study, we interviewed 17 scholars from six disciplines and 15 IRB directors and analysts in the U.S. and analyzed 28 research guidance documents to fill these gaps. We identified common themes among our interviewees and documents but also discovered distinctive and even opposing views regarding the approval rate, rejection, and internal/external research validity. Based on the findings, we discussed a potential Tragedy of the Commons regarding data quality deterioration and the disciplinary differences regarding validity in crowd work-based research. We further explored the origin of the data quality and validity issues in crowd work-based research. We advocated the IRB's ethical concerns in crowd work-based research be heard and respected further and be reflected in the ethical guidance for crowd work-based research. Finally, we proposed our research implications, limits, and future work.

Stories About Crowdwork – Analysis of the Self-Representation of Crowdwork Platforms on the Internet

Crowdwork is often characterised by low incomes and insecure employment conditions. Nevertheless, the business models of crowdwork platforms require a sufficiently large number of crowdworkers. Like other markets, the market for crowdwork is socially constituted, and platform providers try to influence market activities by creating certain images of crowdwork. For platform providers, it is crucial to build a narrative that makes working on the platform attractive to potential crowdworkers. This article examines how platforms present themselves to crowdworkers. Therefore, the self-representation of German-language platforms is analysed. It can be shown that the narratives used and the stories told about crowdwork differ from type to type. Only some of the platforms describe crowdwork as a form of ‘real’ employment. These platforms obviously address (solo-) self-employed people, particularly those working in the IT sector or the creative industries. But even these platforms mainly describe crowdwork as an additional source of income. Especially when earning potential is low, a lot of platforms try to point out other advantages of crowdwork.

La Independiente: an AI-enhanced Platform Co-Designed with Latin-American Crowd-Workers

Crowd-work has increased significantly in recent years, particularly among women from Latin America. However, the specific needs and characteristics of this workforce have not been studied nearly enough. For this reason, we have conducted a series of surveys, questionnaires, and design sessions directly with Latin-American users of crowd-working platforms. Our aim was to create a system to empower crowd-workers with AI enhanced tools for their day-to-day tasks. As a result, we created a customized platform, "La Independiente", and two web plugins. This project is unique in that it leverages gender perspective methodologies, AI powered-systems, and public policy analysis to design smart tools that are both professionally useful and culturally relevant.

Crowdworking: Nurturing Expert-Centric Absorptive Capacity

Organizations increasingly engage with external communities for value generation through an ever-growing multitude of digital services. Absorptive capacity, or the organizational capability to identify, assimilate, and apply new knowledge for commercial ends, is a key determinant of how organizations successfully generate value from external sources of knowledge and sustain a competitive advantage. Crowdworking—a novel form of digitally mediated work—allows organizations to hire on-demand highly skilled external experts to leverage their knowledge, skills, and networks. The approach of integrating crowdworking into organizations is increasingly gaining traction among large corporations seeking to harness the knowledge in external communities for value generation. Building on an in-depth embedded case study in a large organization that relies on two established crowdwork platforms, we explore and shed light on how the organization developed its crowdworking-related absorptive capacity to generate value from external experts. The paper offers new insights into the prevailing modus operandi related to harnessing external knowledge in today’s organizations.

The problem with annotation. Human labour and outsourcing between France and Madagascar

Artificial intelligence advancements have reignited job displacement debates that focus on how the use of artificial intelligence affects labour, without considering how the production of this technology influences labour division. The generalisation of machine learning has created an increased demand for outsourced data workers. Outsourcing companies and crowdwork platforms are both used to generate, annotate, and enrich data. This data tasks are performed by workers from low-income countries, who often earn poverty wages. As with traditional outsourcing, workers must integrate complex multinational subcontracting networks. In this article, we examine how France outsources artificial intelligence-related tasks to workers in the African island nation of Madagascar. For our study, we interviewed 26 data workers, eight employees of French start-ups, and conducted secondary research on two artificial intelligence systems – a canteen checkout terminal and an algorithm to detect shoplifters in stores. The data collected allowed us to reconstruct an end-to-end artificial intelligence production value chain, revealing the need for data classification and artificial intelligence problematisation. Commercial artificial intelligence, therefore, does not displace employment by automating service jobs. Rather, by delocalising labour into the Global South, it lengthens the externalisation chain.

Flexible working and its relations with work-life conflict and well-being among crowdworkers in Germany.

New forms of digital work such as crowdwork promise to increase potentials for flexible working. As crowdworkers can work highly autonomous and from anywhere at any time via the internet, they may be able to make work and private demands more compatible and avoid role conflicts between the life spheres, and in turn, experience enhanced well-being. This research asked whether crowdworkers benefit from flexible working potentials. It investigated the association of the use of different dimensions of flexible working (flex-time, flex-place, and task autonomy) in crowdwork with work-life conflict and well-being. Moreover, it explored whether the relationship between flexible working and well-being was mediated by work-life conflict. The study drew upon a sample of 470 crowdworkers from a cross-sectional self-assessment survey on four different German crowdwork platforms. Structural equation modeling was conducted to estimate the effects of flexible working on work-life conflict, somatic health symptoms, and life satisfaction. Indirect effects were estimated to test the mediation hypothesis. The findings suggest that if crowdworkers use temporal and task flexibility, they experience fewer work-life conflict, fewer somatic health symptoms, and greater life satisfaction. This does not apply to flex-place. The relationship between flexible working and well-being is partly mediated by work-life conflict experiences. This study supports that crowdwork offers a high potential for working highly autonomous and flexibly in time and place. It is associated with lower levels of work-life conflict and enhanced well-being, but the benefits of this flexibility are not universally included in crowdwork.

The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk

The Unintended Consequences of Automated Scripts in Crowdwork Platforms: A Simulation Study in MTurk

Insurance demand experiments: Comparing crowdworking to the lab

AbstractWe analyze an insurance demand experiment conducted in two different settings: in‐person at a university laboratory and online using a crowdworking platform. Subject demographics differ across the samples, but average insurance demand is similar. However, choice patterns suggest online subjects are less cognitively engaged—they have more variation in their demand and react less to changes in exogenous factors of the insurance situation. Applying data quality filters does not lead to more comparable demand patterns between the samples. Additionally, while online subjects pass comprehension questions at the same rate as in‐person subjects, they show more random behavior in other questions. We find that online subjects are more likely to engage in “coarse thinking,” choosing from a reduced set of options. Our results justify caution in using crowdsourced subjects for insurance demand experiments. We outline some best practices which may help improve data quality from experiments conducted via crowdworking platforms.

Hourly Wages in Crowdworking: A Meta-Analysis

In the past decade, crowdworking on online labor market platforms has become an important source of income for a growing number of people worldwide. This development has led to increasing political and scholarly interest in the wages people can earn on such platforms. This study extends the literature, which is often based on a single platform, region, or category of crowdworking, through a meta-analysis of prevalent hourly wages. After a systematic literature search, the paper considers 22 primary empirical studies, including 105 wages and 76,765 data points from 22 platforms, eight different countries, and 10 years. It is found that, on average, microtasks results in an hourly wage of less than $6. This wage is significantly lower than the mean wage of online freelancers, which is roughly three times higher when not factoring in unpaid work. Hourly wages accounting for unpaid work, such as searching for tasks and communicating with requesters, tend to be significantly lower than wages not considering unpaid work. Legislators and researchers evaluating wages in crowdworking need to be aware of this bias when assessing hourly wages, given that the majority of literature does not account for the effect of unpaid work time on crowdworking wages. To foster the comparability of different research results, the article suggests that scholars consider a wage correction factor to account for unpaid work. Finally, researchers should be aware that remuneration and work processes on crowdworking platforms can systematically affect the data collection method and inclusion of unpaid work.

Niewidzialni pracownicy, czyli kto stoi za sztuczną inteligencją

Invisible workers—who is behind the artificial intelligence? The article is dedicated to microworkers—highly precarious, often “invisible”workers whose labour is pivotal for the development of artificial intelligence (AI). It examines how crowdwork platforms such as Amazon Mechanical Turk, Appen and Clickworker organise the work of globally dispersed workers annotating data. It argues that, more often than not, this kind of work is performed in a highly structured, yet intransparent way, and is subject to substantial algorithmic control. This gives rise to the question of the employment classification of formally independent microworkers. The paper discusses this issue in light of the proposed Directive on the improved working conditions of platform workers.

Women in the gig economy: feminising ‘digital labour’

This article explores the gendered dynamics of labouring on digital labour platforms and gives voice to women gig workers. Millions of women worldwide find work through digital labour platforms, yet remain largely invisible within the expansive digital labour research agenda. The analysis is built from original interviews with 49 women in the UK using a range of popular remote crowdwork platforms (including PeoplePerHour, Upwork, TaskRabbit, Freelancer) to access desk-based, white-collar gig work from home. The article makes three original contributions. First, it widens the analytical focus of the digital labour research agenda to recognise the role of workers’ gender identities and uneven household gender divisions of care in shaping the operation and outcomes of digital labour platforms, in ways that remain ‘hidden in the cloud’. Second, in contrast to widespread celebratory claims that platforms disrupt stubborn gender labour market inequalities, the analysis identifies significant gendered constraints on women’s algorithmic visibilities and abilities to compete for gig work online, alongside multiple health and safety issues among women gig workers undocumented in previous research. Third, in response to these new insights, and based on calls from women gig workers themselves, it sets out a series of new directions for extending this urgent multidisciplinary research agenda.

Inside the Black Box: The Evolution of Algorithmic Management in Micro-task Crowdworking Platforms

Inside the Black Box: The Evolution of Algorithmic Management in Micro-task Crowdworking Platforms

Workplace learning in Crowdwork Questionnaire (WLCQ): Measuring self‐regulated learning and skill development in online platform work

AbstractThis paper addresses learning and development processes in online platform work. Specifically, it proposes a new instrument to survey and analyze self‐regulated workplace learning in crowdwork, a type of online platform labor in which a global pool of workers are matched with clients through digital platforms to carry out remunerated tasks. Although workplace learning practices of workers in traditional, organisationally embedded jobs have been studied extensively, the findings cannot be transferred to describe and explain learning behaviors within crowdwork, which is fundamentally different from traditional work in that it is autonomous, radically distributed, and no organisationally provided training opportunities exist in crowdwork. To advance the understanding of workplace learning in crowdwork we reviewed the literature on workplace learning, platform work, and self‐regulated learning to develop the Workplace Learning in Crowdwork Questionnaire, which we subsequently validated with 992 crowdworkers from six European countries on three crowdwork platforms. The instrument can be used by researchers to study the nature of (self‐directed) workplace learning in online labor platforms. It can also be used by crowdworkers to self‐reflect on their learning and development.

The original sin of crowd work for human subjects research

PurposeAcademic scholars have leveraged crowd work platforms such as Amazon Mechanical Turk for human subjects research for almost two decades. However, few scholars have reflected or questioned this mode of academic research. This paper aims to examine three fundamental problems of crowd work and elaborates on their lasting effects on impacting the validity and quality of human subjects research on crowd work.Design/methodology/approach`A critical analysis is conducted on the characteristics of crowd work, and three fundamental problems of crowd work since its origin were identified, namely, the position of “Human-as-a-service,” the confusion of terminology and crowd work platforms’ abdication of responsibilities.FindingsThis paper explains that the three identified fundamental problems of crowd work render at least two lasting problems in crowd work-based research: first, the negligence of the teleological difference between crowd work and academic research; second, the ontological schism between scholars and institutional review boards (IRBs) in their ethical concerns and practices.Originality/valueThis paper critiques the foundation of crowd work-based research that has become growingly popular, extolled and taken for granted. Such a critique is deficient in literature and may seem a bit peculiar. However, we hold that it is time to take research ethics seriously in crowd work because we need to introspect and question ourselves as scholars: What is our motive or ethical stance in using crowd work for human subjects research? Is it for advancing scientific knowledge, promoting crowd workers’ welfare, or predominantly for benefiting ourselves from the fast, cheap and “good” data via crowd work?

Gender and precarity in platform work: Old inequalities in the new world of work

AbstractPlatform work creates a work model that is both a curse and a blessing for vulnerable labour market segments. Based on research on female precarity, the article expects that remote platform work—so‐called crowdwork—could especially attract women who need to combine income and care responsibilities. This article investigates whether women experience more precarity on crowdwork platforms than men, and why their risks differ. It analyses data from a quantitative survey with crowdworkers in Germany and the United States. The results indicate higher precarity risks for women due to care work, which are also indirectly mediated via the employment status. The higher commodification of labour and weaker social infrastructure lead to generally greater precarity risks for platform workers in the United States. The high differences between women and men in Germany underline the gendered nature of labour market dualization and precarization as well as the traditional division of housework. Policy measures should address both platform work and these structural inequalities.

Рынок труда в цифровой экономике – перспективы регулирования

The article examines the social problems that arise due to the lack of legal mechanisms for regulating social and labor relations in a digitalizing labor market. The author emphasizes that the main result of the formation of the digital economy was the organizational restructuring of the labor market. The consequence of this is the transformation of traditional enterprises that hire staff on the terms of an employment contract into crowd-working platforms that “serve” an unlimited number of employers and provide them with the “services” of workers for the implementation of certain technological operations. The resulting relations are increasingly fragmented and atomized in comparison with the traditional social and labor relations. The legal form of exercising the employees’ right to work is changing – from hired labor into self-employment. As a result of the ongoing transformations, labor relations are deprived of a social component based on the joint responsibility of employers and employees for the social well-being of the latter. Thus, the main legal and at the same time socioeconomic problem in the context of digitalization of the labor market is the “disappearance” (in the legal sense) of the subject that is responsible for the payment of insurance social contributions of digital (platform) workers. This is one of the main social challenges of the digital economy: it is becoming more difficult to fill social funds in the same volumes, to provide insurance for the employed against major social risks, which ultimately threatens the implementation of the social function of the state. The author emphasizes that the deterioration of the social situation is by no means due to the legal or technological complexity of creating effective mechanisms for regulating the labor market in the context of digitalization, it is an objective phenomenon. The crisis that has developed in the world economy prevents the resolution of legal conflicts arising in the field of employment. The problems of regulation of labor relations and the formation of mechanisms for the implementation of social policy in the context of a digitalizing labor market must be perceived only as separate “mosaic pieces” in a single canvas of social problems of modern society in the period of its transformation. When solving this problem, we can talk about choosing from two possible options: either about the return of relations in the sphere of labor to the field of social relations in the legal sense, or about recreating the state system of social protection of the population, taking into account the possibilities of its customization of the functions of public administration, which are given by the universal digitalization.

Between automation and gamification: forms of labour control on crowdwork platforms

Algorithmic management is a core concept to analyse labour control on online labour platforms. It runs the risk, however, of oversimplifying the existing variety and complexity of control forms. In order to provide a basis for further research, this article develops a typology of labour control forms within crowdwork and discusses how they influence perceptions of working conditions. It identifies the two most relevant forms of labour control in crowdwork: direct control mainly takes the form of automated output control, while indirect control aiming at creating motivation and commitment is mainly exerted through ranking and reputation systems (gamification). The article shows that these forms of control and their combination are linked with very different ways of how platform workers perceive working conditions on platforms. In addition, the analysis shows significant differences regarding the perception of working conditions between those who work on platforms in addition to a regular employment as opposed to those who are self-employed and rely more strongly, if not fully, on their income from platform work. The analysis is based on qualitative and quantitative research on crowdwork platforms. In particular, it builds on an online survey conducted with 1,131 crowdworkers active on different types of platforms.

I Want to, but First I Need to

Career development is vital for ensuring a happy and productive workforce, and for maintaining relevance in a rapidly changing economy shaped by technological progress. Yet career development is largely ignored in crowdwork. Crowdwork platforms like Amazon Mechanical Turk (AMT) do not support crowdworkers in reskilling and changing careers. In this paper, we study the career goals of AMT workers and the challenges they face in trying to transition out of crowdwork and into high-skill jobs offline or into specialized freelance work. We performed a qualitative study in which we surveyed 20 AMT workers and interviewed 6 of them about their career goals, how they are currently pursuing them, and the challenges they have faced. We found that crowdworkers aspire to transition out of AMT but face challenges due to lack of career guidance, and limited time and financial resources. Drawing on literature in career studies and organization science, we discuss how crowdworkers' challenges are further aggravated by the enviornment on AMT, and provide implications for future research and design that may better support crowdworkers in making a career change.

The Gender Pay Gap in the Platform Economy: Comparing the Importance of Market and Organisational Dynamics on Two German Crowdworking Platforms

The rise of the platform economy has brought about crowdwork as a new form of flexible work where individuals solve specific problems or provide specific services or products in exchange for payment via online platforms. Survey data for crowdworkers in Germany collected by the 'Digital Future' collaborative research unit are used to compare gender inequalities in hourly pay among crowdworkers sampled from a marketplace platform and a micro-task platform. The results reveal that fathers earn higher hourly pay than mothers and childless women and men, but only on the marketplace platform. These differences can partly be explained by fathers being better positioned in the overall labour market and fathers' investment patterns in crowdwork, with longer seniority on the platform and quick task performance, which results in higher hourly pay. Investments in crowdwork and overall labour market positioning are only of modest importance on the micro-task platform. This points to different organisational settings and inequality regimes on the two platforms under study.

Justice expectations in crowd and platform-mediated work

Crowdwork conducted via digital platforms is a young form of work, but a growing part of the gig economy. Typical for crowdwork is low pay, volatile income streams and no social security benefits. Also, crowdworkers have few possibilities for social comparison or negotiation because they work outside of company organisations. This article examines the question of whether these conditions mean that crowdworkers’ expectations about justice in crowdwork arrangements differ in comparison to their expectations regarding justice in conventional employment relationships. This question is addressed empirically on the basis of 36 qualitative interviews and a survey of 230 crowdworkers. The justice expectations of crowdworkers involved in different types of crowdworking platforms in German-speaking countries were examined. In our sample, crowd work typically serves to supplement – not replace – conventional employment. This explorative research shows that crowdworkers use similar standards of justice regarding work performance in their evaluations of work mediated via crowdwork platforms and conventional employment. It shows that crowdworkers perceive injustices in four specific areas: planning insecurity, lack of transparency in performance evaluation, lack of clarity in task briefings and low remuneration. These areas correspond to the theoretical dimensions of distributive and procedural justice on Colquitt’s (2001) organisational justice scale. These findings have implications for future efforts to regulate crowdwork.