Amazon Mechanical Turk Platform Research Articles

Development and initial validation of the appropriate antibiotic use self-efficacy scale

ObjectiveWhile there are various medication self-efficacy scales that exist, none assess self-efficacy for appropriate antibiotic use. The Appropriate Antibiotic Use Self-Efficacy Scale (AAUSES) was developed, pilot tested, and its psychometric properties were examined. MethodsFollowing pilot testing of the scale, a 28-item questionnaire was examined using a sample (n = 289) recruited through the Amazon Mechanical Turk platform. Participants also completed other scales and items, which were used in assessing discriminant, convergent, and criterion-related validity. Test-retest reliability was also examined. ResultsAfter examining the scale and removing items that did not assess appropriate antibiotic use, an exploratory factor analysis was conducted on 13 items from the original scale. Three factors were retained that explained 65.51% of the variance. The scale and its subscales had adequate internal consistency. The scale had excellent test-retest reliability, as well as demonstrated convergent, discriminant, and criterion-related validity. ConclusionsThe AAUSES is a valid and reliable scale that assesses three domains of appropriate antibiotic use self-efficacy. Practice ImplicationsThe AAUSES may have utility in clinical and research settings in understanding individuals’ beliefs about appropriate antibiotic use and related behavioral correlates. Future research is needed to examine the scale’s utility in these settings.

Investigating the Amazon Mechanical Turk Market Through Tool Design

We developed TurkBench to better understand the work of crowdworkers on the Amazon Mechanical Turk (AMT) marketplace. While we aimed to reduce the amount of invisible, unpaid work that these crowdworkers performed, we also probed the day-to-day practices of crowdworkers. Through this probe we encountered a number of previously unreported difficulties that are representative of the difficulties that crowdworkers face in both building their own tools and working on AMT. In this article, our contributions are insights into 1) a number of breakdowns that are occurring on AMT and 2) how the AMT platform is being appropriated in ways that, at the same time, mitigate some breakdowns while exacerbating others. The breakdowns that we specifically discuss in this paper, are the increasing velocity of the market (good HITs are grabbed within seconds), the high amount of flexibility that requesters can and do exercise in specifying their HITs, and the difficulty crowdworkers had in navigating the market due to the large amount of variation in how HITs were constructed by requesters. When the velocity of the market is combined with a poor search interface, a large amount in variation in how HITs are constructed, and little infrastructural support for workers, the resulting work environment can be frustrating and difficult to thrive in.

Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning

With widespread use of machine learning methods in numerous domains involving humans, several studies have raised questions about the potential for unfairness towards certain individuals or groups. A number of recent works have proposed methods to measure and eliminate unfairness from machine learning models. However, most of this work has focused on only one dimension of fair decision making: distributive fairness, i.e., the fairness of the decision outcomes. In this work, we leverage the rich literature on organizational justice and focus on another dimension of fair decision making: procedural fairness, i.e., the fairness of the decision making process. We propose measures for procedural fairness that consider the input features used in the decision process, and evaluate the moral judgments of humans regarding the use of these features. We operationalize these measures on two real world datasets using human surveys on the Amazon Mechanical Turk (AMT) platform, demonstrating that our measures capture important properties of procedurally fair decision making. We provide fast submodular mechanisms to optimize the tradeoff between procedural fairness and prediction accuracy. On our datasets, we observe empirically that procedural fairness may be achieved with little cost to outcome fairness, but that some loss of accuracy is unavoidable.

Self-reported Health Status Differs for Amazon's Mechanical Turk Respondents Compared With Nationally Representative Surveys.

Amazon's Mechanical Turk (MTurk) platform has become a data source for peer-reviewed academic research publications, with over 24,000 Google Scholar search results. Although well-developed and supportive in other disciplines, the literature in health and medicine comparing results from samples generated on MTurk to gold standard, nationally representative health and medical surveys is beginning to emerge. To compare the demographic, socioeconomic, and self-reported health status variables in an MTurk sample to those from 2 prominent national probability surveys, including the Medical Expenditure Panel Survey (MEPS) and the Behavioral Risk Factor Surveillance System (BRFSS). We analyze weighted and unweighted tabulations of the MTurk, MEPS, and BRFSS. Wald tests identify statistical significance. Demographic, socioeconomic, and health status variables in an adult MTurk sample collected in 2016 (n=1916), the 2015 MEPS household survey component (n=21,210), and the 2015 BRFSS (n=283,502). Our findings indicate statistically significant differences in the demographic, socioeconomic, and self-perceived health status tabulations in the MTurk sample relative to the unweighted and weighted MEPS and BRFSS. The MTurk sample is more likely to be female (65.8% in MTurk, 50.9% in MEPS, 50.2% in BRFSS), white (80.1% in MTurk, 76.9% in MEPS, and 73.9% in BRFSS), non-Hispanic (91.1%, 82.4%, and 81.4%, respectively), younger, and less likely to report excellent health status (6.8% in MTurk, 28.3% in MEPS, and 20.2% in BRFSS). We find significant differences across variables that warrant hesitation in using MTurk data as a replacement for the gold standard datasets in health services research.

The role of brand reputation in organic food consumption: A behavioral reasoning perspective

This study examines the role of brand reputation in influencing organic food consumption. Specifically, we adopted the behavioral reasoning theory framework and examined the mechanisms by which consumers’ values affect their attitude and intention to consume organic food under varying levels of brand reputation. To test the theoretical framework, we recruited 617 respondents from Amazon's Mechanical Turk (Mturk) platform. The analysis found general support for the application of behavioral reasoning theory in the organic food consumption context. The results revealed that the relationship between consumer values and attitude is partially (fully) mediated by consumer reasoning in low (high) brand reputation conditions.

Comparing Amazon's Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature.

The goal of this article is to conduct an assessment of the peer-reviewed primary literature with study objectives to analyze Amazon.com 's Mechanical Turk (MTurk) as a research tool in a health services research and medical context. Searches of Google Scholar and PubMed databases were conducted in February 2017. We screened article titles and abstracts to identify relevant articles that compare data from MTurk samples in a health and medical context to another sample, expert opinion, or other gold standard. Full-text manuscript reviews were conducted for the 35 articles that met the study criteria. The vast majority of the studies supported the use of MTurk for a variety of academic purposes. The literature overwhelmingly concludes that MTurk is an efficient, reliable, cost-effective tool for generating sample responses that are largely comparable to those collected via more conventional means. Caveats include survey responses may not be generalizable to the US population.

Finding meaning in a hopeless place? The construction of meaningfulness in digital microwork

New forms of employment centered on the completion of simple and atomized tasks, such as online microwork, raise the question of the possible gratifications that could be derived from such work when compared to more traditional labor arrangements. Our research presented here focuses on how microworkers construct meaningfulness, based on the accounts of workers on the crowdsourcing platform Amazon Mechanical Turk. We draw upon a relational job design perspective to explore why microworkers experience meaningfulness in their work. We found four sources of meaningfulness: rewards, self-improvement, moral, and social. These four sources vary in the degree to which they were internal or external in focus, and in their level of rationalization (concrete or abstract). This may explain why such types of employment are appealing despite a lack of organizational-support structures and points to the need to better understand cue provision in virtual, platform-enabled work settings.

Automatic Task Classification via Support Vector Machine and Crowdsourcing

Automatic task classification is a core part of personal assistant systems that are widely used in mobile devices such as smartphones and tablets. Even though many industry leaders are providing their own personal assistant services, their proprietary internals and implementations are not well known to the public. In this work, we show through real implementation and evaluation that automatic task classification can be implemented for mobile devices by using the support vector machine algorithm and crowdsourcing. To train our task classifier, we collected our training data set via crowdsourcing using the Amazon Mechanical Turk platform. Our classifier can classify a short English sentence into one of the thirty-two predefined tasks that are frequently requested while using personal mobile devices. Evaluation results show high prediction accuracy of our classifier ranging from 82% to 99%. By using large amount of crowdsourced data, we also illustrate the relationship between training data size and the prediction accuracy of our task classifier.

The relationship between treatment accessibility and preference amongst out-of-treatment individuals who engage in non-medical prescription opioid use

BackgroundRelatively little is known regarding the perception of medication-assisted treatments (MATs) and other treatment options amongst individuals that engage in non-medical prescription opioid use. This study surveyed out-of-treatment individuals that misuse opioids to better understand how perceived access to treatment shapes treatment preference. MethodsParticipants (n=357) were out-of-treatment adults registered as workers on the Amazon Mechanical Turk platform who reported current non-medical prescription opioid use. Participants were surveyed regarding demographics, insurance status, attitudes toward opioid use disorder (OUD) treatments, and self-reported symptoms of OUD. ResultsParticipants who were male, did not have health insurance, and knew that counseling-type services were locally available were most likely to first attempt counseling/detox treatments (χ2(6)=30.19, p<0.001). Participants who met criteria for severe OUD, used heroin in the last 30days, knew their insurance covered MAT, and knew of locally available MAT providers were most likely to first attempt MAT (χ2(4)=26.85, p<0.001). Participants with insurance and who knew of locally available physicians were most likely to attempt physician visits without the expressed purpose of MAT (χ2(3)=24.75, p<0.001). ConclusionOut-of-treatment opioid users were particularly interested in counseling-based services and medical care that could be attained from a primary-care physician. Results suggest that insurance coverage and perceived access to OUD treatment modalities influences where out-of-treatment opioid users might first seek treatment; understanding the factors that shape treatment preference is critical in designing early interventions to effectively reach this population.

An Analysis of the Use of Qualifications on the Amazon Mechanical Turk Online Labor Market

Several human computation systems use crowdsourcing labor markets to recruit workers. However, it is still a challenge to guarantee that the results produced by workers have a high enough quality. This is particularly difficult in markets based on micro-tasks, where the assessment of the quality of the results needs to be done automatically. Pre-selection of suitable workers is a mechanism that can improve the quality of the results achieved. This can be done by considering worker’s personal information, worker’s historical behavior in the system, or through the use of customized qualification tasks. However, little is known about how requesters use these mechanisms in practice. This study advances present knowledge in worker pre-selection by analyzing data collected from the Amazon Mechanical Turk platform, regarding the way requesters use qualifications to this end. Furthermore, the influence of using customized qualification tasks in the quality of the results produced by workers is investigated. Results show that most jobs (93.6%) use some mechanism for the pre-selection of workers. While most workers use standard qualifications provided by the system, the few requesters that submit most of the jobs prefer to use customized ones. Regarding worker behavior, we identified a positive and significant correlation between the propensity of the worker to possess a particular qualification, and both the number of tasks that require this qualification, and the reward offered for the tasks that require the qualification, although this correlation is weak. To assess the impact that the use of customized qualifications has in the quality of the results produced, we have executed experiments with three different types of tasks using both unqualified and qualified workers. The results showed that, generally, qualified workers provide more accurate answers, when compared to unqualified ones.

Unfairness by Design? The Perceived Fairness of Digital Labor on Crowdworking Platforms

Based on a qualitative survey among 203 US workers active on the microwork platform Amazon Mechanical Turk, we analyze potential biases embedded in the institutional setting provided by on-demand crowdworking platforms and their effect on perceived workplace fairness. We explore the triadic relationship between employers, workers, and platform providers, focusing on the power of platform providers to design settings and processes that affect workers’ fairness perceptions. Our focus is on workers’ awareness of the new institutional setting, frames applied to the mediating platform, and a differentiated analysis of distinct fairness dimensions.

PTSD's relation with problematic smartphone use: Mediating role of impulsivity

Posttraumatic stress disorder (PTSD) frequently co-occurs with addictive behaviors. Recently, the addictive behavior of excessive smartphone use is being widely researched. Impulsivity commonly relates to PTSD severity and problematic smartphone use. However, unexamined is the mediating role of impulsivity facets (lack of premeditation, negative urgency, sensation seeking, and lack of perseverance) in the PTSD-problematic smartphone use relationship; this was the purpose of the current study. We used data collected from 346 participants recruited through Amazon's Mechanical Turk (MTurk) platform. PTSD severity, impulsivity facets, and problematic smartphone use were assessed using the PTSD Checklist for DSM-5 (PCL-5), the UPPS Impulsive Behavior Scale, and the Smartphone Addiction Scale-Short Version (SAS-SV), respectively. Negative urgency and lack of perseverance had significant positive correlations with both PTSD severity and problematic smartphone use. Results of mediation analyses indicated that negative urgency significantly mediated the relationship between PTSD severity and problematic smartphone use. Findings support the underlying role of negative urgency in the relation between PTSD severity and problematic smartphone use. Thus, treatments targeting negative urgency may potentially serve to reduce problematic smartphone use among individuals experiencing PTSD symptoms.

Conducting interactive experiments online

Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a potentially valuable complement to laboratory studies.

Comparing Crowdsourcing and Friendsourcing: A Social Media-Based Feasibility Study to Support Alzheimer Disease Caregivers.

BackgroundIn the United States, over 15 million informal caregivers provide unpaid care to people with Alzheimer disease (AD). Compared with others in their age group, AD caregivers have higher rates of stress, and medical and psychiatric illnesses. Psychosocial interventions improve the health of caregivers. However, constraints of time, distance, and availability inhibit the use of these services. Newer online technologies, such as social media, online groups, friendsourcing, and crowdsourcing, present alternative methods of delivering support. However, limited work has been done in this area with caregivers.ObjectiveThe primary aims of this study were to determine (1) the feasibility of innovating peer support group work delivered through social media with friendsourcing, (2) whether the intervention provides an acceptable method for AD caregivers to obtain support, and (3) whether caregiver outcomes were affected by the intervention. A Facebook app provided support to AD caregivers through collecting friendsourced answers to caregiver questions from participants’ social networks. The study’s secondary aim was to descriptively compare friendsourced answers versus crowdsourced answers.MethodsWe recruited AD caregivers online to participate in a 6-week-long asynchronous, online, closed group on Facebook, where caregivers received support through moderator prompts, group member interactions, and friendsourced answers to caregiver questions. We surveyed and interviewed participants before and after the online group to assess their needs, views on technology, and experience with the intervention. Caregiver questions were pushed automatically to the participants’ Facebook News Feed, allowing participants’ Facebook friends to see and post answers to the caregiver questions (Friendsourced answers). Of these caregiver questions, 2 were pushed to crowdsource workers through the Amazon Mechanical Turk platform. We descriptively compared characteristics of these crowdsourced answers with the friendsourced answers.ResultsIn total, 6 AD caregivers completed the initial online survey and semistructured telephone interview. Of these, 4 AD caregivers agreed to participate in the online Facebook closed group activity portion of the study. Friendsourcing and crowdsourcing answers to caregiver questions had similar rates of acceptability as rated by content experts: 90% (27/30) and 100% (45/45), respectively. Rates of emotional support and informational support for both groups of answers appeared to trend with the type of support emphasized in the caregiver question (emotional vs informational support question). Friendsourced answers included more shared experiences (20/30, 67%) than did crowdsourced answers (4/45, 9%).ConclusionsWe found an asynchronous, online, closed group on Facebook to be generally acceptable as a means to deliver support to caregivers of people with AD. This pilot is too small to make judgments on effectiveness; however, results trended toward an improvement in caregivers’ self-efficacy, sense of support, and perceived stress, but these results were not statistically significant. Both friendsourced and crowdsourced answers may be an acceptable way to provide informational and emotional support to caregivers of people with AD.

High-throughput phenotyping.

High-throughput phenotyping.

Human values and trust in scientific journals, the mainstream media and fake news

ABSTRACTWhat factors influence trust in online information? Americans increasingly get information from social media, public distrust in the mainstream media is growing and “fake news” is an important new phenomenon. This paper examines the factors that influence trust in scientific claims posted via social media, including the use of hyperlinks and readers' values. The paper describes a crowdsourcing‐based experimental design using Amazon's Mechanical Turk platform. The core of the experiment was a set of 10 scientific findings reported in open‐access, peer‐reviewed scientific journals, which were in turn linked to in articles in both the mainstream media and “fake news” sites. Data analysis involved exploration of relationships between trust and the presence or absence of hyperlinks, and between trust and human values using nonparametric statistical methods. In terms of the influence of hyperlinks on trust, inclusion of hyperlinks to scientific journals, mainstream media articles, and even hidden URLs led to higher trust than hyperlinks to “fake news” sites or posts without hyperlinks (p &lt; 0.001). Participants who clicked on hyperlinks to scientific articles placed higher trust in the claims than those who did not (p &lt; 0.001). In terms of the influence of values on trust, values had the most impact in cases where individuals saw, but decided not to click on, hyperlinks; this finding seems to indicate that in the absence of firsthand examination of the hyperlinked sites, participants tend to rely more heavily on their values to determine their trust in a scientific claim. These findings indicate that both the presence and absence of hyperlinks and the values of the reader both significantly impact trust judgments.

Conducting Interactive Experiments Online

Online labor markets provide new opportunities for behavioral research, but conducting economic experiments online raises important methodological challenges. This particularly holds for interactive designs. In this paper, we provide a methodological discussion of the similarities and differences between interactive experiments conducted in the laboratory and online. To this end, we conduct a repeated public goods experiment with and without punishment using samples from the laboratory and the online platform Amazon Mechanical Turk. We chose to replicate this experiment because it is long and logistically complex. It therefore provides a good case study for discussing the methodological and practical challenges of online interactive experimentation. We find that basic behavioral patterns of cooperation and punishment in the laboratory are replicable online. The most important challenge of online interactive experiments is participant dropout. We discuss measures for reducing dropout and show that, for our case study, dropouts are exogenous to the experiment. We conclude that data quality for interactive experiments via the Internet is adequate and reliable, making online interactive experimentation a valuable complement to laboratory studies.

Evaluating Task-Dependent Taxonomies for Navigation

Taxonomies of concepts are important across many application domains, for instance, online shopping portals use catalogs to help users navigate and search for products. Task-dependent taxonomies, e.g., adapting the taxonomy to a specific cohort of users, can greatly improve the effectiveness of navigation and search. However, taxonomies are usually created by domain experts and hence designing task-dependent taxonomies can be an expensive process: this often limits the applications to deploy generic taxonomies. Crowdsourcing-based techniques have the potential to provide a cost-efficient solution to building task-dependent taxonomies. In this paper, we present the first quantitative study to evaluate the effectiveness of these crowdsourcing based techniques. Our experimental study compares different task-dependent taxonomies built via crowdsourcing and generic taxonomies built by experts. We design randomized behavioral experiments on the Amazon Mechanical Turk platform for navigation tasks using these taxonomies resembling real-world applications such as product search. We record various metrics such as the time of navigation, the number of clicks performed, and the search path taken by a participant to navigate the taxonomy to locate a desired object. Our findings show that task-dependent taxonomies built by crowdsourcing techniques can reduce the navigation time up to $20\%$. Our results, in turn,demonstrate the power of crowdsourcing for learning complex structures such as semantic taxonomies.

Comparing human behavior models in repeated Stackelberg security games: An extended study

Several competing human behavior models have been proposed to model boundedly rational adversaries in repeated Stackelberg Security Games (SSG). However, these existing models fail to address three main issues which are detrimental to defender performance. First, while they attempt to learn adversary behavior models from adversaries' past actions (“attacks on targets”), they fail to take into account adversaries' future adaptation based on successes or failures of these past actions. Second, existing algorithms fail to learn a reliable model of the adversary unless there exists sufficient data collected by exposing enough of the attack surface – a situation that often arises in initial rounds of the repeated SSG. Third, current leading models have failed to include probability weighting functions, even though it is well known that human beings' weighting of probability is typically nonlinear.To address these limitations of existing models, this article provides three main contributions. Our first contribution is a new human behavior model, SHARP, which mitigates these three limitations as follows: (i) SHARP reasons based on success or failure of the adversary's past actions on exposed portions of the attack surface to model adversary adaptivity; (ii) SHARP reasons about similarity between exposed and unexposed areas of the attack surface, and also incorporates a discounting parameter to mitigate adversary's lack of exposure to enough of the attack surface; and (iii) SHARP integrates a non-linear probability weighting function to capture the adversary's true weighting of probability. Our second contribution is a first “repeated measures study” – at least in the context of SSGs – of competing human behavior models. This study, where each experiment lasted a period of multiple weeks with individual sets of human subjects on the Amazon Mechanical Turk platform, illustrates the strengths and weaknesses of different models and shows the advantages of SHARP. Our third major contribution is to demonstrate SHARP's superiority by conducting real-world human subjects experiments at the Bukit Barisan Seletan National Park in Indonesia against wildlife security experts.

United States Technology Use and Public Participation Survey

This dataset represents a large, nationally representative sample of United States residents (N = 1180) about their technology use behaviors and how they participate in social, civic, and political activities. The survey was conducted in 2012 on the Amazon Mechanical Turk platform. Question categories include the degree to which respondents use various technologies, how they use technologies for various purposes, and what kinds of activities they participate in regularly outside of the home. This dataset represents a lens into the daily technological lives of U.S. residents and how technology may influence various public activities, such as social, civic, and political participation.