Mechanical Turk Workers Research Articles

Rapid, precise, and reliable measurement of delay discounting using a Bayesian learning algorithm

Machine learning has the potential to facilitate the development of computational methods that improve the measurement of cognitive and mental functioning. In three populations (college students, patients with a substance use disorder, and Amazon Mechanical Turk workers), we evaluated one such method, Bayesian adaptive design optimization (ADO), in the area of delay discounting by comparing its test–retest reliability, precision, and efficiency with that of a conventional staircase method. In all three populations tested, the results showed that ADO led to 0.95 or higher test–retest reliability of the discounting rate within 10–20 trials (under 1–2 min of testing), captured approximately 10% more variance in test–retest reliability, was 3–5 times more precise, and was 3–8 times more efficient than the staircase method. The ADO methodology provides efficient and precise protocols for measuring individual differences in delay discounting.

Religious Crisis as an Independent Causal Predictor of Psychological Distress: Understanding the Unique Role of the Numinous for Intrapsychic Functioning

Religious and spiritual (R/S) struggles are tensions or conflicts one experiences in relationship to what is considered sacred or transcendent. In this study, we tested competing causal models of psychological distress as it relates to personality and R/S struggle using structural equation modeling. The study sample consisted of 226 (72.0%) females and 88 (28.0%) males (n = 314) drawn from the Amazon’s Mechanical Turk (MTurk) worker population. We found that though the five-factor model (FFM) of personality was a robust predictor of psychological distress, the R/S struggle added significant, incremental predictiveness. SEM analyses supported our contention that R/S struggle may represent a new, causal pathway of psychological distress that is independent from the FFM. Our findings are taken as evidence that R/S struggles require unique ways of conceptualizing their causal impact on clinical impairment and that psychological interventions need to systematically address numinous constructs in order to ensure that all aspects of emotional dysphoria are considered and their influences treated.

Amazon Mechanical Turk workers can provide consistent and economically meaningful data

AbstractAmazon Mechanical Turk (AMT) is an online labor market that is being used increasingly often in the social sciences. This occurs despite significant questions regarding efficacy of the platform. In this article, we attempt to address some of these questions by exploring the consistency of the characteristics of individuals who participate in studies posted on AMT. The primary individuals analyzed in this study are subjects who participated in at least two of eleven experiments that were run on AMT between September of 2012 and January of 2018. We demonstrate subjects consistently report their age, gender, subjective willingness to take risk, and impulsiveness. Further, subjective willingness to take risk is found to be significantly correlated with decisions made in a simple lottery experiment with real stakes—even when the subjective risk measure is reported months, sometimes years, in the past. This suggests the quality of data obtained via AMT is not terribly harmed by the lack of control and low stakes.

Face Validity of Remembering and Knowing: Empirical Consensus and Disagreement Between Participants and Researchers

Ever since Endel Tulving first distinguished between episodic and semantic memory, the remember/know paradigm has become a standard means of probing the phenomenology of participants’ memorial experiences by memory researchers, neuropsychologists, neuroscientists, and others. However, this paradigm has not been without its problems and has been used to capture many different phenomenological experiences, including retrieval from episodic versus semantic memory, recollection versus familiarity, strength of memory traces, and so on. We first conducted a systematic review of its uses across the literature and then examined how memory experts, other cognitive psychology experts, experts in other areas of psychology, and lay participants (Amazon Mechanical Turk workers) define what it means when one says “I remember” and “I know.” From coding their open-ended responses using a number of theory-bound dimensions, it seems that lay participants do not see eye to eye with memory experts in terms of associating “I remember” responses with recollection and “I know” responses with familiarity. However, there is general consensus with Tulving’s original distinction, linking remembering with memory for events and knowing with semantic memory. Recommendations and implications across fields are discussed.

The Mental Landscape of Imagining Life Beyond the Current Life Span: Implications for Construal and Self-Continuity.

Background and ObjectivesWith rapid advancements in medicine, technology, and nutrition, the future holds the possibility of longer and healthier lives. Despite garnering attention from myriad disciplines, psychological perspectives on life extension are scarce. In three studies, we addressed this gap by exploring key mental characteristics and psychological variables associated with simulating an expanded life span and thus an extremely distant future self.Research Design and MethodsThree studies investigated the construal (i.e., valence, vividness, and visual perspective) of extremely distant future simulations and the extent to which participants felt connected to their future selves (i.e., self-continuity). Studies 1 and 2 investigated the characteristics of imagery associated with different ages ranging from near the current species maximum (e.g., 120, 150) to more highly hypothetical ages (e.g., 201, 501). Study 3 probed the mental construal of extreme aging among different populations (i.e., life-extension supporters, students, and Mechanical Turk workers). Studies also assessed participants’ general feelings about the ethicality and likelihood of techniques that halt or reverse biological aging to help individuals live beyond the current life expectancy.ResultsParticipants in all studies reported being able to vividly imagine expanded aging scenarios (increased chronological, without biological, and aging), but these simulations were characterized by a decreased sense of connection to one’s future self (i.e., self-continuity) compared to a control condition. Temporal distance did not, however, impact ratings of self-continuity when comparing experimental conditions (i.e., imagining one’s self 120 vs 150 or 201 vs 501). Curiously, a sense of self-continuity (when simulating oneself well beyond the current life expectancy) remained intact for individuals who belonged to a community of life-extension supporters. The perceived likelihood and ethicality of extended life-span scenarios also varied significantly across different populations.Discussion and ImplicationsThe current work is the first to quantify the disconnect between one’s current and extremely distant (i.e., beyond the current life expectancy) future self. Given the behavioral implications of feeling disconnected from one’s future self (e.g., failing to save for retirement or care for one’s own physical health), these findings inform a critical barrier of extended life spans and provide insight into potential remedies (e.g., enhancing the perceived likelihood of living longer). Theoretical implications of hypotheticality and temporal distance, two key dimensions of Construal Level Theory, and their impact on the construal and self-continuity associated with future simulations are also discussed.

The behavior of gradual types: a user study

There are several different gradual typing semantics, reflecting different trade-offs between performance and type soundness guarantees. Notably absent, however, are any data on which of these semantics developers actually prefer. We begin to rectify this shortcoming by surveying professional developers, computer science students, and Mechanical Turk workers on their preferences between three gradual typing semantics. These semantics reflect important points in the design space, corresponding to the behaviors of Typed Racket, TypeScript, and Reticulated Python. Our most important finding is that our respondents prefer a runtime semantics that fully enforces statically declared types.

Subjective interpretation of "objective" video evidence: Perceptions of male versus female police officers' use-of-force.

The police face great scrutiny after highly publicized instances of lethal force. Dash-camera footage ostensibly provides "objective" evidence of whether the force was excessive. We tested whether participants interpreted the same "objective" video of an officer exerting force differently based on the officer's gender and race. We predicted that when (a) a male (vs. female) officer used force and (b) a Black (vs. White) officer used force, participants would endorse more internal and less external explanations for their use-of-force, which would be associated with less trust in and perceived effectiveness of the officer. We randomly assigned Amazon's Mechanical Turk workers (N = 452; 53% female, 80% White) to (a) see a segment of a police-civilian interaction video that either included or did not include exertion of force, and to believe that the officer was (b) male versus female, and (c) Black versus White. They reported their trust in the officer and perceptions of the officer's effectiveness, and their degree of agreement with external and internal attributions for the officer's behavior. When officers used force, people trusted officers less (d = 1.13) and perceived them to be less effective (d = .78) relative to when they did not. Despite all participants viewing the same interaction, people who thought they saw a male (vs. female) officer perceived his use-of-force to be driven more by internal traits, such as being aggressive and emotionally reactive, and less by the external situation, which was associated with decreased trust and perceived effectiveness. In contrast, people perceived female (vs. male) officers' force to be driven more by external aspects of the dangerous situation, which was associated with increased trust and perceived effectiveness. This pattern did not depend on the officers' race or participants' gender. This constitutes a rare instance of women benefiting from violating gender stereotypes in the workplace because people assumed her counterstereotypical behavior was more justified by the situation and less about her being an aggressive and emotionally reactive person. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Ideological Conflict and Prejudice: An Adversarial Collaboration Examining Correlates and Ideological (A)Symmetries

In an adversarial collaboration, we examined associations among factors that could link ideological conflict—perceiving that members of a group do not share one’s ideology—to prejudice and affiliation interest. We also examined whether these factors would possess similar (“symmetrical”) or different (“asymmetrical”) associative strength among liberals and conservatives. Across three samples (666 undergraduate students, 347 Mechanical Turk workers), ideological conflict was associated with perceived dissimilarity on political and nonpolitical topics, as well as negative emotions. Perceived political and nonpolitical dissimilarity were also associated with negative emotions, prejudice, and lower affiliative intentions among both liberals and conservatives. Importantly, however, perceived political dissimilarity was associated with negative emotions, prejudice, and lower affiliative intentions more strongly among liberals. Some inconsistent evidence also suggested that perceived nonpolitical dissimilarity was associated with prejudice and lower affiliative intentions more strongly among conservatives. These findings document nuance in relationships that could link ideological conflict to prejudice.

Determining the Pathways to Alcohol Use Consequences: a Chained Mediation Approach

Research has attempted to explain how emotion dysregulation and impulsivity relate to alcohol use problems. This study extended the literature by testing different pathways to problem alcohol use. We hypothesized that approach impulsivity/disinhibition and alcohol use frequency would mediate the relationship between emotion dysregulation and negative alcohol use consequences. We also hypothesized that alcohol use frequency would mediate the relationship between sensation seeking/reward sensitivity and negative alcohol use consequences. A cross-sectional chained mediation effects model was tested using data from 508 Amazon Mechanical Turk workers who resided in the USA (mean age = 33.66, standard deviation = 11.70, 59.6% female). Significant simple mediation effects from emotion dysregulation to alcohol use consequences through approach impulsivity/disinhibition, and from sensation seeking/reward sensitivity to alcohol use consequences through alcohol use frequency were found. There were no chained mediation effects. Two distinct pathways to alcohol use consequences are identified: one from emotion dysregulation through approach impulsivity/disinhibition, and another from sensation seeking/reward sensitivity through alcohol use frequency. This study highlights the importance of understanding the different pathways to problem drinking, as it can be crucial for developing refined treatment techniques.

Facilitating Episodic Simulation in Anxiety: Role of Sensory Scaffolding and Scenario Modality

Cognitive bias modification of interpretation style (CBM-I) is a family of cognitive training programs that seek to reduce anxious thinking by training people to assign relatively more positive meanings to ambiguous situations. CBM-I’s effects may be enhanced by encouraging more vivid imagery-based episodic simulation of events and by increasing engagement with the training materials. This study investigated the role of sensory scaffolding (whether pictures, or pictures + sound were added) and verbal scenario modality (whether scenarios were delivered visually or aurally) on episodic simulation (Vivid; Plausible; Changing Perspective ratings) and user engagement (Relatable, Comprehensible, Enjoyable ratings). Amazon Mechanical Turk workers (N = 187) with varied anxiety symptom severity read or listened to brief scenarios that varied by sensory scaffolding and verbal scenario modality. Results were somewhat mixed. Generally, picture scaffolding tended to facilitate both episodic simulation and user engagement (relative to no scaffolding), irrespective of scenario modality and anxiety level.

Shepherd’s Office. The Politics of Digital Labor and Its Impact on the Amazon Mechanical Turk Workers

Shepherd’s Office (SO) is a collaborative performance and research project between artists Daniela Vainio and Zaneta Zukalova, first introduced at Tate Modern (Tate Exchange) London in 2017, which questions the systematic order of Amazon Mechanical Turk. In May 2019, SO held a live investigative performance at the Art Fair Suomi in Helsinki, with an aim to collect the data that informs this article; SO’s focus was on the socio-political background of Amazon Mechanical Turk Workers and their personal relations to and opinions on the platform. The present article introduces both the artistic and investigative aspects of the project and the problematics of Amazon Mechanical Turk, and offers an insight into the research inputs, processes, and outcomes.

Faces extracted from Time Magazine 1923-2014

We present metadata of labeled faces extracted from a Time magazine archive that contains 3,389 issues ranging from 1923 to 2012. The data we are publishing consists of three subsets: Dataset 1) the gender labels and image characteristics for each of the 327,322 faces that were automatically-extracted from the entire Time archive, Dataset 2) a subset of 8,789 faces from a sample of 100 issues that were labeled by Amazon Mechanical Turk (AMT) workers according to ten dimensions (including gender) and used as training data to produce Dataset 1, and Dataset 3) the raw data collected from the AMT workers before being processed to produce Dataset 2.

Altruistic Self-Concept Mediates the Effects of Personality Traits on Volunteering: Evidence from an Online Experiment

The way we describe and evaluate ourselves or self-concept has a significant effect on our behaviors. An individual who describes herself as altruistic is more likely to engage in helpful behaviors than someone who does not possess such self-concept. Self-concept, in turn, is influenced by personality. In this paper, we propose that self-concept mediates the effects of personality on volunteering. We tested this idea experimentally with Amazon Mechanical Turk workers. In our experiment, participants answered a series of questionnaires, including the Big Five personality test and attitudinal questions that we used to construct an index representing altruistic self-concept. Participants were also given the option to make and keep a promise to volunteer by donating time and effort to a charity. Our results show that agreeableness has a strong influence on volunteering through its effect on altruistic self-concept.

The Association between Psychopathy and Influencing Others to Use Substances

Background: Although a large body of research has examined the impact of peer influence on substance use, little is known about the psychology of who influences others to use substances and why. The Dark Triad is a set of personality traits that may inform the characteristics of people who influence others to use substances.Objective: The aim of the present two studies was to examine potential associations between Dark Triad personality traits, especially psychopathy, and influencing others to use substances.Methods: Two samples of 162 and 161 Mechanical Turk workers were recruited for Study 1 to complete an online survey that assessed Dark Triad traits, history of influencing others to use substances, and motivations for doing so. Similarly, 303 Mechanical Turk workers were recruited for Study 2 to complete an online survey that assessed personal substance use as well as psychopathy levels of participants’ closest friends.Results: Study 1 found that individuals high in psychopathy were more likely to have provided illegal substances to others and indicated a willingness to do so in the future. Study 2 found that participants reported a higher likelihood of using substances if they indicated that their best friend was high in psychopathy and had themselves used substances in the past.Conclusions: Taken together, these findings suggest that psychopathy has a unique association with influencing others to use substances. These findings have implications for intervention targets.

The persistence of pay inequality: The gender pay gap in an anonymous online labor market.

Studies of the gender pay gap are seldom able to simultaneously account for the range of alternative putative mechanisms underlying it. Using CloudResearch, an online microtask platform connecting employers to workers who perform research-related tasks, we examine whether gender pay discrepancies are still evident in a labor market characterized by anonymity, relatively homogeneous work, and flexibility. For 22,271 Mechanical Turk workers who participated in nearly 5 million tasks, we analyze hourly earnings by gender, controlling for key covariates which have been shown previously to lead to differential pay for men and women. On average, women’s hourly earnings were 10.5% lower than men’s. Several factors contributed to the gender pay gap, including the tendency for women to select tasks that have a lower advertised hourly pay. This study provides evidence that gender pay gaps can arise despite the absence of overt discrimination, labor segregation, and inflexible work arrangements, even after experience, education, and other human capital factors are controlled for. Findings highlight the need to examine other possible causes of the gender pay gap. Potential strategies for reducing the pay gap on online labor markets are also discussed.

Abstract No. 454 Crowd-sourced assessment of interventional radiology technical skill: a method to guide future training of interventional radiologists

Crowd sourced assessment of surgical skill using standardized evaluation has been demonstrated in the literature to be as accurate as expert opinion. A small pilot study demonstrated translation of this task to interventional radiology (IR). Here, next steps in the development of a grading scale for IR technical skill is pursued with further evaluation of the accuracy of a crowd-sourced assessment. Three simulator performances of prostate artery catheterization and embolization performed by IR trainees were recorded. A survey was created consisting of a 13-question grading scale specific to prostate artery embolization (PAE) and a 10-question grading scale on general IR technical skills. 50 Amazon Mechanical Turk (MTurk) workers and 5 IR faculty viewed the three videos and completed the entire assessment. The MTurk grading was compared with that of the experienced IR faculty. Means and standard deviations were calculated for individual items and in aggregate across videos. To determine interrater reliability among faculty, a two-way intraclass correlation coefficient was calculated per each item. Dissimilar from multiple prior studies, the MTurk workers did not replicate expert scoring. In contradistinction, when questions were considered in aggregate, the MTurk workers rated lowest (mean, 2.71; standard deviation [SD], 1.06) the video rated highest (mean, 2.83; SD, 0.88) by the faculty, and rated highest (mean, 2.83; SD, 0.99) the video rated lowest (mean, 1.61; SD, 0.83) by the faculty. However, IR faculty were noted to demonstrate decent inter-rater reliability with interreader agreement for 15 of 23 items (65%) at a threshold of P = 0.05. Although initial data has suggested that IR technique may be successfully graded by standardized evaluation, this data suggests that the complexity and intricacy of more advanced technical evaluation may not be as well suited to crowd sourced assessment. However, the good level of inter rater reliability among faculty, most notably of the poorly skilled video, suggest standardized evaluation by experienced faculty may have merit. Further study is needed to refine and validate the evaluation form.

Immanent Justice Reasoning by Spatial Proximity

Immanent justice reasoning involves causally attributing someone’s bad outcome to their prior immoral actions. Building on the idea that causality is mentally linked with spatial proximity, we investigated whether such reasoning might lead participants to spatially bind together immoral actions and bad outcomes. Across four experiments (N = 553, Mechanical Turk workers), participants positioned sentences describing other people’s bad (vs. good) outcomes closer in space to previous immoral behaviors. This effect was observed both when the position of the initial action remained in a fixed location and when it “chased” the outcome across the screen. Importantly, we also found that this spatial positioning of immoral actions and bad outcomes is mediated by perceived deservingness of the outcome and is not merely due to perceived similarity of events. These findings suggest that perceived deservingness biases the spatial proximity of representations of others’ random bad outcomes and their prior immoral actions.

Prosociality in the economic Dictator Game is associated with less parochialism and greater willingness to vote for intergroup compromise

AbstractIs prosociality parochial or universalist? To shed light on this issue, we examine the relationship between the amount of money given to a stranger (giving in an incentivized Dictator Game) and intergroup attitudes and behavior in the context of randomly assigned teams (a minimal group paradigm) amongN= 4,846 Amazon Mechanical Turk workers. Using a set of Dynamic Identity Diffusion Index measures, we find that participants who give more in the Dictator Game show less preferential identification with their team relative to the other team, and more identification with all participants regardless of team. Furthermore, in an incentivized Voter Game, participants who give more in the Dictator Game are more likely to support compromise by voting for the opposing team in order to avoid deadlock. Together, these results suggest that – at least in this subject pool and using these measures – prosociality is better characterized by universalism than parochialism.

Impact of Loss and Gain Forecasting on the Behavior of Pricing Decision-making

Recent forecasting research has shown a paradigm shift from algorithm aversion to appreciation. Despite growing trust in technological decision support, business decisions are often made based on gut feeling and intuition, ignoring part or all of the available data and information. Creating effective decision support solutions necessitates the understanding of the impact of emerging artificial intelligence and machine learning technologies on business decision-making processes. This study examines whether forecasting information delivery at a time when a business decision is made influences or changes the decision maker’s mind, thereby leading to a different decision. The study employs a 2 × 2 between-subject experimental setting where forecasted results (gain/loss) and automated advice (risk/certainty) were crossed-examined. A sample of 137 participants was asked to make four different product price change decisions assisted by automated decision aid. The experiment involved two independent samples, one taken from Amazon Mechanical Turk workers and the other from the members of LinkedIn managerial groups. Results show that decision-makers are more likely to rely on automated recommendation and change their initial decision when forecasted decision outcomes lead to gain, whereas they would discount algorithmic aid if a loss is forecasted. This research adds to the extant literature in the field of human-technology interactions and contributes to the descriptive and prescriptive decision theories by illustrating that gain forecasting has a higher impact on the algorithm appreciation than loss forecasting.

Eliciting Human Judgment for Prediction Algorithms

Even when human point forecasts are less accurate than data-based algorithm predictions, they can still help boost performance by being used as algorithm inputs. Assuming one uses human judgment indirectly in this manner, we propose changing the elicitation question from the traditional direct forecast (DF) to what we call the private information adjustment (PIA): how much the human thinks the algorithm should adjust its forecast to account for information the human has that is unused by the algorithm. Using stylized models with and without random error, we theoretically prove that human random error makes eliciting the PIA lead to more accurate predictions than eliciting the DF. However, this DF-PIA gap does not exist for perfectly consistent forecasters. The DF-PIA gap is increasing in the random error that people make while incorporating public information (data that the algorithm uses) but is decreasing in the random error that people make while incorporating private information (data that only the human can use). In controlled experiments with students and Amazon Mechanical Turk workers, we find support for these hypotheses.