Crowdsourcing Platform Research Articles

"Guilds" as Worker Empowerment and Control in a Chinese Data Work Platform

Data work plays a fundamental role in the development of algorithmic systems and the AI industry. It is often performed in business process outsourcing (BPO) companies and crowdsourcing platforms, involving a global and distributed workforce as well as networks of collaborative actors. Previous work on community building among data workers centers organization and mutual support or focuses on the structuring and instrumentalization of crowdworker groups for complicated projects. We add to these lines of research by focusing on a specific form of community building encouraged and facilitated by platforms in China: guilds. Based on ethnographic work on a Chinese crowdsourcing platform and 14 semi-structured interviews with data workers, our findings show that guilds are a form of both worker empowerment and control. With this work, we add a nuanced empirical case to the interconnection of BPOs, online communities and crowdsourcing platforms in the current data production sector in China, thus expanding previous investigations on global perspectives of data production. We discuss guilds in relation to individual workers and highlight their effects on data work, including efficient coordination, enhanced standardization, and flattened power structure.

Determining the Effect of Feedback Quality on User Engagement on Online Idea Crowdsourcing Platforms Using an AI model

The success of idea crowdsourcing platforms relies on fostering a collaborative environment that encourages active user participation, measured by the quality and quantity of contributions and interactions. However, understanding the impact of peer interactions on user engagement and the innovation process remains challenging. While previous studies have focused on sentiment analysis, the contextual interpretation of peer feedback and its effects on engagement and innovation have yet to be explored. To address this knowledge gap, we propose a feature-based AI model that categorizes peer feedback based on its quality and polarity. Our model achieves 96% accuracy in identifying feedback quality (constructive vs. non-constructive and toxic vs. non-toxic) and feedback polarity (negative vs. positive). This contextual feedback categorization provides a foundation for quantitatively analyzing the impacts of feedback on user engagement and the innovation process. Our results show that positive and constructive peer feedback has a significantly positive effect, while toxic and negative peer feedback has a significantly negative impact on the process of idea generation, idea evaluation, and idea selection. Our results also indicate that inexperienced users are more susceptible to toxic feedback than experienced users. Based on these findings, we suggest a recommendation and reward system for incentivizing constructive feedback and preventing toxic feedback.

The multi-objective task assignment scheme for software crowdsourcing platforms involving new workers

The multi-objective task assignment scheme for software crowdsourcing platforms involving new workers

A Preliminary Investigation of a Conceptual Model Describing the Associations Between Childhood Maltreatment and Alcohol Use Problems

Background/Objectives: Childhood maltreatment has been linked to numerous adverse outcomes in adulthood, including problem substance use. However, not all individuals exposed to childhood maltreatment develop substance use problems, indicating the role of other factors in influencing this outcome. Past work suggests that adverse early life experiences, including childhood maltreatment, lead to neurobiological changes in frontolimbic functions that, in turn, result in altered stress and reward responses, heightened impulsivity, affect dysregulation, and, ultimately, increased risk for maladaptive behaviors such as substance use. The aim of this preliminary investigation using cross-sectional data was to test associations between these factors in the relationship between childhood maltreatment and alcohol use problems in a sample of emerging adults. Methods: Emerging adults (18–30 years old) who identified as regular drinkers (i.e., drinking at least 2–4 times in the past month) were recruited from a crowd-sourcing platform (Prolific) as well as community samples. Participants completed online standardized questionnaires assessing reward sensitivity and responsiveness, impulsivity, emotion regulation, and alcohol consequences. Results: Path analyses demonstrated good fit for the data (SRMR = 0.057, RMSEA = 0.096, 90% CI [0.055, 0.142], CFI = 0.957). Childhood maltreatment was associated with reward responsiveness (β = −0.026, Z = −4.222, p < 0.001) and emotion dysregulation (β = 0.669, Z = 9.633, p < 0.001), which in turn was associated with urgency and, subsequently, alcohol consequences (β = 0.758, Z = 7.870, p < 0.001). Conclusions: Although these findings are preliminary, the current study is one of the first to test a comprehensive model addressing the relationship between childhood maltreatment and alcohol use problems. The findings have the potential to inform treatment strategies that target motivation and goal-directed action for reducing and managing consequences associated with childhood maltreatment. Future research should test the model using longitudinal data to address the limitations of a cross-sectional study and assess temporal associations between constructs.

Unveiling the Inter-Related Preferences of Crowdworkers: Implications for Personalized and Flexible Platform Design

Crowdsourcing platforms have traditionally been designed with a focus on workstation interfaces, restricting the flexibility that crowdworkers need. Recognizing this limitation and the need for more adaptable platforms, prior research has highlighted the diverse work processes of crowdworkers, influenced by factors such as device type and work stage. However, these variables have largely been studied in isolation. Our study is the first to explore the interconnected variabilities among these factors within the crowdwork community. Through a survey involving 150 Amazon Mechanical Turk crowdworkers, we uncovered three distinct groups characterized by their interrelated variabilities in key work aspects. The largest group exhibits a reliance on traditional devices, showing limited interest in integrating smartphones and tablets into their work routines. The second-largest group also primarily uses traditional devices but expresses a desire for supportive tools and scripts that enhance productivity across all devices, particularly smartphones and tablets. The smallest group actively uses and strongly prefers non-workstation devices, especially smartphones and tablets, for their crowdworking activities. We translate our findings into design insights for platform developers, discussing the implications for creating more personalized, flexible, and efficient crowdsourcing environments. Additionally, we highlight the unique work practices of these crowdworker clusters, offering a contrast to those of more traditional and established worker groups.

Brief online suicide risk assessment of adults does not affect state mood, even in the context of elevated suicidality self-stigma, suicidal ideation and psychological distress.

The current study aimed to assess whether online suicide risk assessment affects state mood and is the first to examine if suicide-related self-stigma or coping related to suicidal ideation are predictors of mood change. The Australian participants (N = 661, Mage = 34.9, SD = 12.3, 57.1% female), recruited through a crowd-sourcing platform, completed a visual analogue mood measure before and after the Suicidal Ideation Attributes Scale (SIDAS), an assessment tool. Followed by a modified version of the Internalised Stigma Scale, the Brief COPE and DASS-21. State mood did not change from pre- to post-suicide risk assessment in the overall sample, t(662) = -.16, p = .868, d = -.01. Contrary to hypotheses, neither self-stigma nor coping were related to mood change following exposure to the SIDAS. The multiple regression model was not significant, F(9,643) = 1.16, p = .31., nor was any single predictor including gender, current Suicide risk β = -.04, t = -.80 or psychological distress β = -.09, t = -1.76, p = .08. These findings suggest that online exposure to a suicide risk tool is unlikely to be iatrogenic in relation to state mood, even in the context of elevated self-stigma, suicidal ideation and psychological distress.

DERCo: A Dataset for Human Behaviour in Reading Comprehension Using EEG

This paper introduces the DERCo (Dublin EEG-based Reading Experiment Corpus), a language resource combining electroencephalography (EEG) and next-word prediction data obtained from participants reading narrative texts. The dataset comprises behavioral data collected from 500 participants recruited through the Amazon Mechanical Turk online crowd-sourcing platform, along with EEG recordings from 22 healthy adult native English speakers. The online experiment was designed to examine the context-based word prediction by a large sample of participants, while the EEG-based experiment was developed to extend the validation of behavioral next-word predictability. Online participants were instructed to predict upcoming words and complete entire stories. Cloze probabilities were then calculated for each word so that this predictability measure could be used to support various analyses pertaining to semantic context effects in the EEG recordings. EEG-based analyses revealed significant differences between high and low predictable words, demonstrating one important type of potential analysis that necessitates close integration of these two datasets. This material is a valuable resource for researchers in neurolinguistics due to the word-level EEG recordings in context.

Media Literacy Interventions Improve Resilience to Misinformation: A Meta-Analytic Investigation of Overall Effect and Moderating Factors

The widespread dissemination of misinformation has become a global concern. A recommended solution is to improve people’s ability to discern true from false information through appropriate media literacy education programs. This meta-analysis quantitatively synthesized the results of 49 experimental studies ( N = 81,155) that examined the efficacy of media literacy interventions in mitigating misinformation. This study finds that media literacy interventions generally improve resilience to misinformation ( d = 0.60). Specifically, the interventions reduce belief in misinformation ( d = 0.27), improve misinformation discernment ( d = 0.76), and decrease misinformation sharing ( d = 1.04). Moreover, media literacy interventions have stronger effects (1) when multiple sessions rather than a single session are implemented, (2) in high (vs. low) uncertainty avoidance cultures, and (3) among college students than among adults recruited from online crowdsourcing platforms (e.g., Amazon Mechanical Turk). These findings enrich our understanding of inoculation theory and provide valuable guidance for the design of future media literacy intervention programs.

Informal Caregiver Burden and Potential Consequences

Providing care to an adult loved one is often described as a burden, but it can also promote flourishing. The present study examined the extent to which state hope and emotion regulation strategies of positive reappraisal and emotional suppression might account for the relation of burden on both burnout and flourishing in a sample of 162 informal caregivers (54.8% women; Mage = 42.92, SDage = 13.22). It was hypothesized that state hope would mediate the relationship between caregiver burden and both outcomes, with state hope being positively related to flourishing and negatively related to burnout. It was also hypothesized that positive reappraisal and emotional suppression would mediate the relationship between caregivers’ burden and the outcomes, with positive reappraisal predicting greater flourishing and emotional suppression predicting greater burnout, as well as less flourishing. Participants were recruited from the Prolific crowd sourcing platform and had been pre-screened as informal and unpaid caregivers of an adult. They completed a battery of questionnaires that were all self-report in nature for the cross-sectional design. Results indicate that only state hope mediated the relationship between caregiver burden and the outcomes of burnout and flourishing. Contrary to what was hypothesized, emotion regulation strategy was not a mechanism in these relationships. State hope could be a potential avenue for intervention for informal caregivers in that it may impact caregiver burden as well as burnout and flourishing.

Dynamic Operations of a Mobile Charging Crowdsourcing Platform

This paper investigates the operation of a novel electric vehicles (EVs) charging service mode, that is, crowdsourced mobile charging service for EVs, whereby a crowdsourcing platform is established to arrange suppliers (crowdsourced chargers) to deliver charging service to customers’ electric vehicles (parked EVs) at low-battery levels. From the platform operator’s perspective, we aim to determine the optimal operation strategies for mobile charging crowdsourcing platforms to achieve specific objectives. A mathematical modeling framework is developed to capture the interactions among supply, demand, and service operations in the crowdsourced mobile charging market. To design an efficient solution method to solve the formulated model, we first analyze the model properties by rigorously proving that a crucial variable set for operating the mobile charging crowdsourcing system includes charging price, commission control, and period-specific aggregate demand control. Besides, we provide both an equivalent condition and a necessary condition for checking the feasibility of these crucial variables. On top of this, we construct a search tree according to the operation periods in a day to solve the optimal operation strategies, wherein a nondominated principle is adopted as an accelerating technique in the searching process. The solution obtained from the proposed solution algorithm is proved to be sufficiently close to the actual global optimal solutions of the formulated model up to the resolution of the discretization scheme adopted. Numerical examples provide evidence verifying the model’s validity and the solution method’s efficiency. Overall, the research outcome of this work can offer service operators structured and valuable guidelines for operating mobile charging crowdsourcing platforms. Funding: This work was supported by the Singapore Ministry of Education [Grant RG124/21]. Supplemental Material: The online appendices are available at https://doi.org/10.1287/trsc.2023.0126 .

The Dark Side of Gamification: Unintended Consequences of Designing Gamified Crowdsourcing Platforms

The Dark Side of Gamification: Unintended Consequences of Designing Gamified Crowdsourcing Platforms

Describing functional low back pain disability in the American population: a crowd-sourced evaluation of the Oswestry disability index

BACKGROUND CONTEXTThe Oswestry disability index (ODI) is widely utilized as a patient reported outcome (PRO) tool to assess patients presenting with low back pain (LBP) and following thoracolumbar spine surgery. No primary study has calculated the baseline range of ODI values in the diverse American population. Establishing age-adjusted normative values for ODI in the American population is crucial for assessing the utility of treatment strategies. PURPOSEThe purpose of this study is to describe the baseline range of functional low back disability as measured by the ODI in an American population. STUDY DESIGN/SETTINGCross-Sectional Observational Study. PATIENT SAMPLEA total of 1214 participants were recruited from the United States in January 2024 using a combination of the Connect and PrimePanel platforms by CloudResearch to complete a survey administered on a RedCap online database. The survey consisted of 10 demographic questions and the 10 ODI survey questions. The distribution of the survey was designed to obtain approximately 100 respondents in each of the following age groups: 18–29, 30 to 39, 40 to 49, 50 to 59, 60 to 69, 70 to 79, and 80 to 89. The distribution of the sample was similarly designed to match the United States Census racial data with 78.1% White, 13.9% Black, and 7.9% other. OUTCOME MEASURESOswestry disability index (ODI). METHODSA crowd-sourcing platform called Cloudresearch was used to collect a representative sample of the US population by answering questions of the Oswestry disability questionnaire (ODQ), a 10-question survey. RESULTSThe final sample size was 797 participants including 386 (48.4%) males and 411 (51.6%) females; 169 participants were excluded that did not complete the survey and an additional 248 were excluded for failing attention check questions. The overall mean ODI score for the combined age groups was 14.35 (95% CI [13.33, 15.37]). The mean ODI scores increased with age, with the highest mean ODI in ages 70 to 79 at 18.0 (95% CI [14.76, 21.24]). Female participants reported higher mean ODI scores than their male counterparts in the 18 to 29 age group (p=.01), 50 to 59 age group (p=.01), and 60 to 69 age group (p=.02). Additionally, a weak positive correlation was found between Body Mass Index (BMI) and ODI scores (r = 0.22, p<.001). CONCLUSIONOur findings demonstrate a clear trend of increased disability with age. This study describes the baseline range of functional low back pain disability in the United States population. By defining these parameters, healthcare professionals can better tailor age and sex-specific interventions to manage disability in the aging US population, ultimately improving patient care and both operative and nonoperative treatment plans for LBP-related thoracolumbar pathology.

Crowdsourced Comparative Judgement for Evaluating Learner Texts: How Reliable are Judges Recruited from an Online Crowdsourcing Platform?

Abstract Recent studies of proficiency measurement and reporting practices in applied linguists have revealed widespread use of unsatisfactory practices such as the use of proxy measures of proficiency in place of explicit tests. Learner corpus research is one specific area affected by this problem: few learner corpora contain reliable, valid evaluations of text proficiency. This has led to calls for the development of new L2 writing proficiency measures for use in research contexts. Answering this call, a recent study by Paquot et al. (2022) generated assessments of learner corpus texts using a community-driven approach in which judges, recruited from the linguistic community, conducted assessments using comparative judgement. Although the approach generated reliable assessments, its practical use is limited because linguists are not always available to contribute to data collections. This paper, therefore, explores an alternative approach, in which judges are recruited through a crowdsourcing platform. We find that assessments generated in this way can reach near identical levels of reliability and concurrent validity to those produced by members of the linguistic community.

Comparing two crowdsourcing platforms: assessing their potential for mapping Antarctica

sGeographical data acquisition in Antarctic regions is challenging due to the lack of human habitation, harsh environmental conditions, and limited accessibility. This research explores and evaluates the capability of two crowdsourcing platforms in mapping facilities across Antarctic regions. The study presents crowdsourcing projects related to polar regions in the literature. The methodology section outlines the data acquisition techniques employed by Flickr and Happywhale, and the spatial evaluation methods applied to the collected data. In the implementation and results section, the spatiotemporal potential of the data obtained from the two identified crowdsourced platforms is assessed, and the results based on spatial statistical methods are compared. In the discussion and conclusion section, the contribution of the two identified crowdsourced platforms to mapping activities is evaluated in terms of spatial, temporal, and content differences. This study reveals that Happywhale offers data with higher spatial consistency, considering seasonal representation and spatial autocorrelation. Additionally, content restrictions and reliance on GPS enhance spatial accuracy in Happywhale. At the same time, the liberation of data production leads to lower quality but increased quantity, diversity, and spatial coverage, as observed in Flickr. By comparing two crowdsourced platforms, this study enhances data acquisition and evaluation potential in Antarctica.

Derailment is associated with increased depressed mood at lower levels of stress: An experimental approach

BackgroundAs an emergent developmental construct, derailment – an individual difference hallmarked by a temporal discordance of the self – has been positively associated with depressive features, though less is known about the role of stress in this relationship. The aim of the present study was to explore the interaction between derailment and subjective stress on depressed mood. Methods265 adults recruited through Prolific – a crowd sourcing platform were first tasked with completing baseline measures, then were randomly assigned to a low stress or high stress condition. We utilized an adapted version of the Online Trier Stress Test to manipulate stress levels. Next, depressed mood and post-test perceived stress were measured. We analyzed data through STATA using a two-tailed independent samples t-test and multiple linear regression. ResultsFindings suggested the experimental manipulation increased subjective stress within the high stress condition compared to the low stress condition (Mdiff = 36.14, SE = 4.4, t = 8.21, p < .001). Further, we found the interaction between derailment and stress conditions was associated with increased depressed mood (β = −0.11, SE = 0.05, p = .038, R2 = 0.5). Notably, we found significant between-condition differences in depressed mood for those who endorsed either low (d = 0.22, SE = 0.07, p = .003) or moderate levels of derailment (d = 0.11, SE = 0.05, p = .033), but not for those who endorsed high levels of derailment (d = 0.01, SE = 0.07, p = .92). LimitationsThis study utilized self-report measures, which may be susceptible to biased responses. ConclusionsDerailment may play a role in the persistence of depressed mood regardless of varying contexts of stress.

Expecting to integrate additional information improves averaging of experience

Humans learn both directly, from own experience, and via social communication, from the experience of others. They also often integrate these two sources of knowledge to make predictions and choices. We hypothesized that when faced with the need to integrate communicated information into personal experience, people would represent the average of experienced exemplars with greater accuracy. In two experiments, Mturk users estimated the mean of consecutively and rapidly presented number sequences that represented bonuses ostensibly paid by different providers on a crowdsource platform. Participants who expected integrating these values with verbal information about possible change in bonuses were more accurate in extracting the means of the values compared to participants who did not have such expectation. While our study focused on socially communicated information, the observed effect may potentially extend to other forms of information integration. We suggest that expected integration of experience with additional information facilitates an abstract representation of personal experiences.

Research on the Acceptance of Crowdsourced Translation Platform Technologies by Translators

In the current era marked by rapid digitalization and globalization, crowdsourced translation platforms have emerged as key players in the global translation market, prized for their flexibility and efficiency. This new model of translation services not only provides a means to address large-scale translation demands but also introduces challenges and shifts in conventional translation workflows. This study investigates translators’ acceptance of paid crowdsourced translation platform technologies, with the aim of identifying the primary factors that influence technology acceptance and understanding how these factors impact translators’ work attitudes and behaviors. Through a mixed-methods approach that integrates surveys and in-depth interviews, the study gathers data from 300 translators with varied experiences and backgrounds. Utilizing the Technology Acceptance Model (TAM), the research reveals the significant effects of perceived usefulness and perceived ease of use on translators’ acceptance of technology. It is particularly noteworthy that translators show a high sensitivity to their perceptions of the platforms’ efficiency and potential changes in their income. Furthermore, job satisfaction emerges as a key factor in determining translators’ willingness to continue using crowdsourced platforms. This research not only provides insights into how to enhance the relationship between crowdsourced translation platforms and translators but also offers valuable recommendations for the advancement of translation education and the professional development of translators.

Why Do Online Crowdsourcing Platform (OCP) Visitors Recommend Idea Competition to Others?

Why Do Online Crowdsourcing Platform (OCP) Visitors Recommend Idea Competition to Others?

Matching Tasks and Workers under Known Arrival Distributions: Online Task Assignment with Two-sided Arrivals

Efficient allocation of tasks to workers is a central problem in crowdsourcing. In this article, we consider a setting inspired by spatial crowdsourcing platforms, where both workers and tasks arrive at different times, and each worker-task assignment yields a given reward. The key challenge is to address the uncertainty in the stochastic arrivals from both workers and the tasks. In this work, we consider a ubiquitous scenario where the arrival patterns of worker “types” and task “types” are not erratic but can be predicted from historical data. Specifically, we consider a finite time horizon T and assume that in each time-step the arrival of a worker and a task can be seen as an independent sample from two (different) distributions. Our model, called Online Task Assignment with Two-sided Arrival (OTA-TSA), is a significant generalization of the classical online task assignment problem when all the tasks are statically available. For the general case of OTA-TSA, we present an optimal non-adaptive algorithm (NADAP), which achieves a competitive ratio (CR) of at least 0.295. For a special case of OTA-TSA when the reward depends only on the worker type, we present two adaptive algorithms, which achieve CRs of at least 0.343 and 0.355, respectively. On the hardness side, we show that (1) no non-adaptive can achieve a CR larger than that of NADAP, establishing the optimality of NADAP among all non-adaptive algorithms; and (2) no (adaptive) algorithm can achieve a CR better than 0.581 (unconditionally) or 0.423 (conditionally on the benchmark linear program), respectively. All aforementioned negative results apply to even unweighted OTA-TSA when every assignment yields a uniform reward. At the heart of our analysis is a new technical tool, called two-stage birth-death process , which is a refined notion of the classical birth-death process. We believe it may be of independent interest. Finally, we perform extensive numerical experiments on a real-world rideshare dataset collected in Chicago and a synthetic dataset, and results demonstrate the effectiveness of our proposed algorithms in practice.

Metric Space Indices for Dynamic Optimization in a Peer to Peer-Based Image Classification Crowdsourcing Platform

Metric Space Indices for Dynamic Optimization in a Peer to Peer-Based Image Classification Crowdsourcing Platform