Online Crowdsourcing Platforms Research Articles

Methodological considerations for behavioral studies relying on response time outcomes through online crowdsourcing platforms

This perspective paper explores challenges associated with online crowdsourced data collection, particularly focusing on longitudinal tasks with time-sensitive outcomes like response latencies. Based on our research, we identify two significant sources of bias: technical shortcomings such as low, variable frame rates, and human factors, contributing to high attrition rates. We explored potential solutions to these problems, such as enforcing hardware acceleration and defining study-specific frame rate thresholds, as well as pre-screening participants and monitoring hardware performance and task engagement over each experimental session. With this discussion, we intend to provide recommendations on how to improve the quality and reliability of data collected via online crowdsourced platforms and emphasize the need for researchers to be cognizant of potential pitfalls in online research.

(Mis)alignment between facial and textual emotions and its effects on donors engagement behavior in online crowdsourcing platforms

(Mis)alignment between facial and textual emotions and its effects on donors engagement behavior in online crowdsourcing platforms

Using mixture modeling to examine differences in perceptual decision-making as a function of the time and method of participant recruitment

We examine whether perceptual decision-making differs as a function of the time in the academic term and whether the participant is an undergraduate participating for course credit, a paid in-person participant, or a paid online participant recruited via Amazon Mechanical Turk. We use a mixture modeling approach within an evidence accumulation framework that separates stimulus-driven responses from contaminant responses, allowing us to distinguish between performance when a participant is engaged in the task and the consistency in this task focus. We first report a survey showing cognitive psychologists expect performance and response caution to be lower among undergraduate participants recruited at the end of the academic term compared to those recruited near the start, and highest among paid in-person participants. The findings from two experiments using common paradigms revealed very little evidence of time-of-semester effects among course credit participants on accuracy, response time, efficiency of information processing (when engaged in the task), caution, and non-decision time, or consistency in task focus. However, paid in-person participants did tend to be more accurate than the other two groups. Groups showed similar effects of speed/accuracy emphasis on response caution and of discrimination difficulty on information processing efficiency, but the effect of speed/accuracy emphasis on information processing efficiency was less consistent among groups. We conclude that online crowdsourcing platforms can provide quality perceptual decision-making data, but recommend that mixture modeling be used to adequately account for data generated by processes other than the psychological phenomena under investigation.

Exploring intrusions without awareness: A preliminary study of the characteristics and influences of meta-awareness failures

Background and ObjectivesResearch shows that people can lack meta-awareness (i.e., being explicitly aware) of their trauma-related thoughts, which impacts our understanding of re-experiencing symptoms, a key symptom type in posttraumatic stress disorder (PTSD), assessed through self-report. This preliminarily study explored differences between (meta-)aware and unaware intrusion characteristics to understand why some intrusions are not immediately apparent to individuals. MethodsTrauma-exposed participants (N = 78) were recruited from online crowd-sourcing platforms to complete an online meta-awareness task. During a reading task, participants were intermittently probed to index the occurrence of unreported (i.e., unaware) trauma-related intrusions. Once participants indicated trauma-related intrusions were present, they then completed a questionnaire that indexed intrusion characteristics. ResultsAlthough unaware intrusions did occur in a subset of the sample, there were no fundamental differences between aware and unaware intrusions in terms of modality of experience (imagery vs. non-imagery), meaningfulness, accessibility, or other characteristics (e.g., vividness). LimitationsThere was potential for lower participant engagement and attention due to the online delivery of the meta-awareness task, which may have minimized meta-awareness failure. Future research could consider using a continuous measure to index levels of meta-awareness. In addition, recruiting clinical samples (e.g., individuals with PTSD) who typically experience multiple daily intrusions would allow generalizability of the current findings to be tested. ConclusionsOur findings from this preliminary study suggest that unaware and aware intrusions show more commonality than not in their characteristics, with further research required to improve our understanding of the mechanisms leading to meta-awareness or lack of in PTSD.

Emoji, Text, and Sentiment Polarity Detection Using Natural Language Processing

Virtual users generate a gigantic volume of unbalanced sentiments over various online crowd-sourcing platforms which consist of text, emojis, or a combination of both. Its accurate analysis brings profits to various industries and their services. The state-of-art detects sentiment polarity using common sense with text only. The research work proposes an emoji-based framework for cognitive–conceptual–affective computing of sentiment polarity based on the linguistic patterns of text and emojis. The proposed emoji and text-based parser articulates sentiments with proposed linguistic features along with a combination of different emojis to generate the part of speech into n-gram patterns. In this paper, the sentiments of 650 world-famous personages consisting of 1,68,548 tweets have been downloaded from across the world. The results illustrate that the proposed natural language processing framework shows that the existence of emojis in sentiments many times seems to change the overall polarity of the sentiment. By extension, the CLDR name of the emoji is utilized to evaluate the accurate polarity of emoji patterns, and a dictionary of sentiments is adopted for evaluating the polarity of text. Eventually, the performances of three ML classifiers (SVM, DT, and Naïve Bayes) are evaluated for proposed distinctive linguistic features. The robust experiments indicate that the proposed approach outperforms the SVM classifier as compared to other ML classifiers. The proposed polarity detection generator has achieved an exceptional perspective of sentiments presented in the sentence by employing the flow of concept established, based on linguistic features, polarity inversion, coordination, and discourse patterns, surpassing the performance of extant state-of-the-art approaches.

Blurred border or safe harbor? Emotional well-being among sexual and gender minority adults working from home during COVID-19

During the COVID-19 pandemic, lesbian, gay, bisexual, transgender, and queer (LGBTQ) adults have experienced pronounced declines in well-being. However, less is known about how changes to daily routines and settings, such as the shift to remote work within many occupations, may be playing a role in well-being outcomes. Drawing on a unique time diary data source (N = 3515 respondents and 7650 episodes) collected between April 2020–July 2021 through online crowdsourcing platforms, we conducted random effects analyses to examine how working from home has been associated with experienced well-being among LGBTQ and cisgender heterosexual workers in the United States during the pandemic. Findings indicate LGBTQ adults felt significantly less stressed and tired while doing paid work at home than while working at a workplace. In addition, working at a workplace, rather than working from home, appeared to be more detrimental to LGBTQ adults’ well-being compared to their non-LGBTQ counterparts. Adjusting for work characteristics explained some of the difference, whereas adjusting for family characteristics had little impact on the results. It is possible that for LGBTQ employees, working from home mitigates some of the minority stressors experienced during paid work.

Crowdsourcing and language learning habits and practices in Turkey, Bosnia and Herzegovina, the Republic of North Macedonia and Poland in the pre-pandemic and pandemic periods

The popularity of online crowdsourcing platforms was slowly increasing among language learners before the pandemic, but COVID-19 changed the educational systems worldwide. This study aims to uncover whether or not, and if ‘YES’, how the attitudes and habits of language learners concerning the use of crowdsourcing materials in Turkey, Bosnia and Herzegovina, the Republic of North Macedonia and Poland changed during the pandemic. To compare the pre-and during the covid crowdsourcing tool usage, the cross-culturally appropriate questionnaire utilised in the pre-COVID-19 period was used again. The collected data were analysed qualitatively and quantitatively to identify the differences between the periods. The study’s findings showed that the shift from face-to-face to online learning significantly affected the development of crowdsourcing platforms worldwide and their employment in the studied countries. The results also demonstrated that a combination of factors, such as reduced interactions with teachers and peers, an increase in workload, and a lack of support on the part of institutions, led to students taking responsibility for their learning. The number and characteristics of the popular platforms changed from country to country since expectations from students varied.

Effects of Payment Rate and Country’s Income Level on Attitude Toward aCrowdsourcing Task

Using online crowdsourcing platforms has become an option for researchers to rapidly recruit participants for completing tasks that require human ingenuity. However, a growing concern is that workers on online crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), receive unfair compensation for tasks completed. In this article, we explored the effects of the income level of participant’s country and the rate of payment on perceived payment fairness, task quality, and subjective experience. We tested our hypothesis using 3-way ANOVA and chi-square test of independence. The results showed that lower compensation increased the number of participants whose data might be unusable for research. Participants in the lower compensation rate group reported better perceived performance compared to those who received higher compensation. We found that high income country participants report less perceived effort and frustration than lower-middle income country participants.

Recombinant Innovations: What Differentiates Experienced Designers in Open Crowdsourcing Contests?

Prior studies of online crowdsourcing platforms have found that experienced designers are more likely to win in crowdsourcing contests. However, what gives experienced designers an edge in these contests is not well understood. Our study seeks to understand what differentiates experienced designers from other participants, with a particular focus on how they leverage information in open design contests. We use a large-scale empirical analysis employing deep-learning algorithms and find that, experienced designers are more adept at integrating information from several prior highly-rated submissions from other designers within a contest, while less-experienced designers are more likely to excessively imitate individual prior highly-rated submissions. We also find that experienced designers whose submissions are closer in similarity to a synthesized image of several highly-rated prior submissions, are more likely to win. Our results are consistent with prior work on recombinant innovations which finds that a majority of innovations happen by a synthesis or recombination of prior innovations, and that designers, as they gain experience, learn optimal recombination strategies. Our findings provide new insights into the winning strategies of experienced designers in crowdsourcing platforms and have implications for the design of such markets.

The effects of exposure to others' ideas and their ratings on online crowdsourcing platforms on the quantity and novelty of subsequently generated ideas

AbstractResearch has shown that exposing people to others' ideas and the ratings of those ideas can influence their subsequent idea generation performance. In this paper, we utilize a socio‐cognitive perspective to study this in an online crowdsourcing context. The cognitive factor concerns knowledge activated in memory because of idea exposure, which can be used for subsequent idea generation. The social factor concerns idea ratings that may steer the search for ideas in memory in the direction of ideas that are more likely to receive approval. Our contribution to the literature concerns an investigation of the joint effect of (1) the type of ideas posted by others (i.e., common versus novel ideas) and (2) the ratings of those ideas (i.e., low versus high ratings) on subsequent idea generation performance. We find compelling evidence for an interaction effect between these two factors in four empirical studies. That is, exposure to posted ideas that are “highly‐rated and novel” or “poorly‐rated and common” can result in more novel subsequent ideas. Exposure to posted ideas that are “highly‐rated and common” can result in a larger number of subsequent ideas. These findings have implications for product innovation managers who wish to exert more control over the number and type of generated ideas, for example, on online crowdsourcing platforms.

Eating disorder related research using Amazon Mechanical Turk (MTurk): Friend or foe?: Commentary on Burnette et al. (2021).

Burnette et al. reported a study that they sought to undertake to validate common eating disorder questionnaires in sexual and gender minorities. The researchers took advantage of the online recruitment platform Amazon Mechanical Turk (MTurk). Contrary to their expectations, the study proved not feasible due to invalid answering. Thus, Burnette et al. raise concerns against the trustworthiness of crowd-sourced data that may be undermined by financial interests and other kinds of motivations. Our commentary highlights the potential of the COVID-19 pandemic to inflate especially those intentions, which are monetary. Against the background of the COVID-19 pandemic, a further problem seems to be that the anonymity of online crowd sourcing platforms might tempt participants to provide inconsistent answers, possibly reflecting tendencies of reactance. The reported pattern of paradoxical responses in Burnette et al.'s work does not reflect malingering; rather we believe that the study might have served some participants as an outlet for negative emotions. We discuss mechanisms of quality control and highlight the lack of interpersonal interaction associated with online data collections.

Do Social Dominance-Based Faultlines Help or Hurt Team Performance in Crowdsourcing Tournaments?

ABSTRACT Firms organize tournaments on online crowdsourcing platforms to outsource complex business problems to external solvers. Participants on these platforms often self-organize into ad-hoc virtual teams to compete in such tournaments. Social dominance-based faultlines, which originate from the alignment of members based on IT-enabled social dominance attributes (e.g., rank and tier), have emerged as a novel type of faultline in crowdsourcing teams. Building on the Categorization-Elaboration Model (CEM), we investigate the contingent effects of team ability and team effort on the relationship between social dominance-based faultlines and team performance in crowdsourcing tournaments. We collected data of 265 virtual teams from Kaggle.com. We discovered that IT-enabled social dominance-based faultlines positively influence the performance of teams with low ability and high effort, whereas the effect becomes negative for teams with high ability and low effort. Our study yields theoretical implications by advancing a novel type of social dominance-based faultline and extending the CEM with two contingent factors (i.e., team effort and team ability) pertinent to team performance on crowdsourcing tournaments. We also offer practical guidelines for team formation in crowdsourcing tournaments and for the design of crowdsourcing platforms.

Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application

Industrial robot programming tools increasingly rely on graphical interfaces, which aim at rendering the programming task more accessible to a wide variety of users. The usability of such tools is currently being evaluated in controlled environments, such as laboratories or companies, in which a group of participants is asked to carry out several tasks using the tool and then fill out a standardized questionnaire. In this context, this paper proposes and evaluates an alternative evaluation methodology, which leverages online crowdsourcing platforms to produce the same results as face-to-face evaluations. We applied the proposed framework in the evaluation of a web-based industrial robot programming tool called Assembly. Our results suggest that crowdsourcing facilitates a cost-effective, result-oriented, and reusable methodology for performing user studies anonymously and online.

A Financial Incentive Mechanism for Truthful Reporting Assurance in Online Crowdsourcing Platforms

In today’s world, crowdsourcing is regarded as an effective strategy to deal with a high volume of small issues whose solutions can have their own complexities in systems. Moreover, requesters are currently providing hundreds of thousands of tasks in online job markets and workers need to perform these tasks to earn money. Thus far, various aspects of crowdsourcing including budget management, mechanism design for price management, forcing workers to behave truthfully in bidding prices, or maximized gains of crowdsourcing have been considered in different studies. One of the main existing challenges in crowdsourcing is how to ensure truthful reporting is provided by contributing workers. Since the amount of pay to workers is directly correlated with the number of tasks performed by them over a period of time, it can be predicted that strong incentives encourage them to carry out more tasks by giving untruthful answers (providing the first possible answer without examining it) in order to increase the amount of pay. However, crowdsourcing requesters need to obtain truthful reporting as an output of tasks assigned to workers. In this study, a mechanism was developed whose implementation in crowdsourcing could ensure truthful reporting by workers. The mechanism provided in this study was evaluated as more budget feasible and it was also fairer for requesters and workers due to its well-defined procedure.

Evaluating Reddit as a Crowdsourcing Platform for Psychology Research Projects

Background: Online crowdsourcing platforms, such as Amazon Mechanical Turk (MTurk), have become popular alternatives to the ubiquitous student samples used in psychology research. r/SampleSize, an alternative pool on the website Reddit, allows for online participant recruitment without compulsory or immediate payment, making it potentially useful for students, research trainees, and course instructors. Objective: The current study sought to assess the viability of using r/SampleSize as a participant pool by comparing its data characteristics to MTurk and existing lab samples. Method: Two hundred and fifty-six MTurk workers and 277 r/SampleSize participants completed identical questionnaires on demographics, participation motivations, and standard psychology scales. Results: Participants recruited through r/SampleSize reported diverse ages, education levels, income, and employment, although White ethnic background and US residence were predominant. r/SampleSize participants were more internally motivated than MTurk to participate in research and had greater need for cognition but did not differ significantly in altruism or motivation to gain self-knowledge. r/SampleSize data reliability and quality were comparable to MTurk and lab samples across most analyses. Teaching Implications: r/SampleSize can be used to recruit relatively large and diverse samples for undergraduate research projects with minimal setup, labor, and cost. Conclusion: The findings suggest that r/SampleSize is a diverse and viable participant pool.

What Crowdsourcing Platforms Do for Solvers in Problem-Solving Contests: A Content Analysis of Their Websites

Solvers’ wide and continuous participation is imperative for the sustainable development of online crowdsourcing platforms (OCPs). Prior studies have deeply investigated what and how solvers’ motives and demographics, task attributes, requester attributes, and platform designs influence solvers’ participation. However, to our knowledge, few studies concentrated on what these OCPs do for solvers in practices that are concerned with solvers and thus influence their decision to participate. To fill this gap, this study conducted a content analysis of 25 typical OCPs focusing on problem-solving contests to identify service measures that they presented for solvers. Consequently, 14 major service measures that are related to contest management, solver management, and requester management were identified. Thereafter, we discussed the roles of these service measures in solvers’ participation. They are activating solvers to participate, providing solvers opportunities to participate, and supporting solvers to participate. Our analysis, on the one hand, presents a comprehensive list of service measures for solvers distributed on these OCPs separately and on the other hand aids the OCPs to improve their solver service and for solvers to compare and analyze their preferred OCPs as a reference.

Evaluating Visual Properties via Robust HodgeRank

Nowadays, how to effectively evaluate visual properties has become a popular topic for fine-grained visual comprehension. In this paper we study the problem of how to estimate such visual properties from a ranking perspective with the help of the annotators from online crowdsourcing platforms. The main challenges of our task are two-fold. On one hand, the annotations often contain contaminated information, where a small fraction of label flips might ruin the global ranking of the whole dataset. On the other hand, considering the large data capacity, the annotations are often far from being complete. What is worse, there might even exist imbalanced annotations where a small subset of samples are frequently annotated. Facing such challenges, we propose a robust ranking framework based on the principle of Hodge decomposition of imbalanced and incomplete ranking data. According to the HodgeRank theory, we find that the major source of the contamination comes from the cyclic ranking component of the Hodge decomposition. This leads us to an outlier detection formulation as sparse approximations of the cyclic ranking projection. Taking a step further, it facilitates a novel outlier detection model as Huber’s LASSO in robust statistics. Moreover, simple yet scalable algorithms are developed based on Linearized Bregman Iteration to achieve an even less biased estimator. Statistical consistency of outlier detection is established in both cases under nearly the same conditions. Our studies are supported by experiments with both simulated examples and real-world data. The proposed framework provides us a promising tool for robust ranking with large scale crowdsourcing data arising from computer vision.

Eating disorder prevalence among Amazon MTurk workers assessed using a rigorous online, self-report anthropometric assessment

Online, anonymous data collection is common and increasingly available to researchers studying eating disorders (ED), particularly since the development of online crowdsourcing platforms. Crowdsourcing for participant recruitment may also be one effective strategy to address ED research disruptions caused by the COVID-19 pandemic. We aimed to: (a) develop a rigorous method for assessing self-reported athropometrics; (b) determine if individuals with EDs self-select into MTurk studies assessing eating behaviors; and (c) characterize ED-related psychopathology in an MTurk sample. We recruited 400 US adults to complete an MTurk study assessing ED features. Results did not indicate the presence of a self-selection bias among individuals with EDs; however, 40% of the sample met criteria for a current ED diagnosis, with all diagnoses represented except ARFID, and 18.1% reported currently being in ED treatment. The sample was characterized by higher scores on measures of ED psychopathology compared to extant non-clinical norms. Approximately 66% of the overall sample and 73% of participants with EDs indicated that they have participated in more MTurk studies since the COVID-19 pandemic began. Finally, we identified an alternative approach to assessing self-reported height and weight that appears to reduce error, which we strongly recommend researchers conducting online surveys use. Our findings suggest that individuals with EDs appear to be overrepresented on MTurk and highlight the utility of crowdsourcing using MTurk as an ED data collection alternative during and after the COVID-19 pandemic.

Technological Factors Facilitating B40's Motivation in Malaysia to Continue Using Online Crowdsourcing Platform

The increasing number of retrenchments because of the current global pandemic, Covid-19, has led many to shift to the digital economy, especially among the low-income group (B40) in Malaysia. Crowdsourcing is the collection of information, opinions, or work from a group of people, usually sourced via the Internet. Fueled by the development of Internet-based platforms that provided its technological foundation, and the need for an agile and uniquely skilled workforce, crowdsourcing has grown from the grassroots, with a burgeoning body of research investigating its many aspects. However, very few studies examined crowd workers' motivation for continuous participation on online crowdsourcing platforms. Thus, this paper aims to explore the technological factors that facilitate B40's group motivation in Malaysia to continue to participate in online crowdsourcing platforms. This paper employed a qualitative approach, using a semi-structured interview. The thematic analysis method was used to decode the data extracted from the interview transcript. The finding of this study identified four main themes and seven sub-themes: (1) Technology efficacy, (2) Platform Management: client-worker management, safety net, payment mechanism, (3) Platform Design: Ul design, rating feature and (4) Infrastructure: Internet connection, technology infrastructure. This study can provide a guideline for managing crowdsourcing practices in Malaysia, especially for the crowdsourcing platform developer.

Identifying Emotions in Images and Their Effects on Donation Behavior in Online Crowdsourcing Platforms

Although textual emotional appeals in a donation context have been studied in past research, there has been little work looking at facial emotions expressed in posted images. Drawing on a panel data of 25,321 crowdfunding projects from Gofundme, we investigate how facial emotions expressed in posted images might be strategically used to increase individual project effectiveness as measured by metrics relevant to the platform, i.e., donation amount per project. In this research, we focus on four emotions that can be inferred accurately from facial expressions in images using artificial intelligence methods; happy, sad, anger and surprise. Our empirical analysis shows that increasing the degree to which facial expressions in an image convey any of these four emotions has a positive impact on the contributed amount to a project, with sadness having the biggest effect, followed by anger, surprise, and happiness. We also use MTurk studies to explore the process behind these effects, finding that perceived justice is the most dominant mediator in all the conditions of surprise, angry, happy and sad images, while perceived empathy and emotion contagion are mediators only in the case of surprise and happy images. And finally in a follow-up controlled study, we provide strategic recommendations for platform revenue management.