Mechanical Turk Sample Research Articles

Utilizing Five Factor Model facets to conceptualize counterproductive, unethical, and organizational citizenship workplace behaviors

General personality traits have potential utility for assisting with employee selection processes and predicting employee behaviors. Previous research has examined and provided evidence for the Five Factor Model (FFM) personality domains with positive and negative workplace behaviors, but few studies have examined these relationships at the facet level. To address this gap in the literature, the current study examined the association between workplace behaviors and FFM facets in both university students and Amazon Mechanical Turk (MTurk) samples. Resampling techniques were utilized to examine the relationships of FFM personality traits, with both positively oriented (i.e., organizational citizenship) and negatively oriented (i.e., workplace deviance, and unethical) workplace behaviors. The results demonstrated that FFM facets are meaningfully associated with positive and negative workplace behaviors and highlighted the evidence for the utility of considering facet level relationships, beyond domain level analyses. Counterproductive workplace behaviors were primarily associated negatively with facets of agreeableness and conscientiousness, and organizational citizenship behaviors were primarily associated positively with facets of extraversion and conscientiousness and negatively with neuroticism. The applied utility of personality trait assessment and facet-level analysis is discussed relative to workplace behaviors.

The Message Matters: The Influence of Fact-Checking on Evaluations of Political Messages

Fact-checks have become prolific in U.S. campaigns over the last ten years. As a result, fact-checks have become one of the easiest ways for individuals to analyze the truthfulness of politicians’ statements. The increase in both fact-checking and its accessibility to voters led us to ask whether fact-checks influence individuals’ attitudes and evaluations of political candidates and campaign messages. To examine the impact of fact-checking, we conduct two original experiments using the 2012 Ohio Senate race between Republican challenger Josh Mandel and Democratic incumbent Sherrod Brown. The first experiment utilized a sample of over 300 students during the fall of 2013. The second experiment is a crowdsourced Amazon Mechanical Turk Sample in the fall of 2014. We find the content of fact-check messages are influential in altering assessments of candidates’ advertisements. We also find the source of the fact-check only modestly impacts assessments. The findings illustrate the potential power of fact-checks to influence the effectiveness of candidates’ messages and reaffirm the important role the news media plays in validating candidate claims and arguments during political campaigns.

Temporal framing and the hidden-zero effect: rate-dependent outcomes on delay discounting.

Recent research suggests that presenting time intervals as units (e.g., days) or as specific dates, can modulate the degree to which humans discount delayed outcomes. Another framing effect involves explicitly stating that choosing a smaller-sooner reward is mutually exclusive to receiving a larger-later reward, thus presenting choices as an extended sequence. In Experiment 1, participants (N = 201) recruited from Amazon Mechanical Turk completed the Monetary Choice Questionnaire in a 2 (delay framing) by 2 (zero framing) design. Regression suggested a main effect of delay, but not zero, framing after accounting for other demographic variables and manipulations. We observed a rate-dependent effect for the date-framing group, such that those with initially steep discounting exhibited greater sensitivity to the manipulation than those with initially shallow discounting. Subsequent analyses suggest these effects cannot be explained by regression to the mean. Experiment 2 addressed the possibility that the null effect of zero framing was due to within-subject exposure to the hidden- and explicit-zero conditions. A new Amazon Mechanical Turk sample completed the Monetary Choice Questionnaire in either hidden- or explicit-zero formats. Analyses revealed a main effect of reward magnitude, but not zero framing, suggesting potential limitations to the generality of the hidden-zero effect.

Law and Psychology Grows Up, Goes Online, and Replicates

Over the last 30 years, legal scholars have increasingly deployed experimental studies, particularly hypothetical scenarios, to test intuitions about legal reasoning and behavior. That movement has accelerated in the last decade, facilitated in large part by cheap and convenient Internet participant recruiting platforms like Amazon Mechanical Turk. The widespread use of online subjects, a practice that dramatically lowers the barriers to entry for experimental research, has been controversial. At the same time, the field of experimental psychology is experiencing a public crisis of confidence widely discussed in terms of the “replication crisis.” At present, law and psychology research is arguably in a new era, in which it is both an accepted feature of the legal landscape and also a target of fresh skepticism. The moment is ripe for taking stock. In this article, we bring an empirical approach to these problems. Using three canonical law and psychology findings, we document the challenges and the feasibility of reproducing results across platforms. We evaluate the extent to which we are able to reproduce the original findings with contemporary subject pools (Amazon Mechanical Turk, other national online platforms, and in‐person labs). We partially replicate all three results, and show marked similarities in subject responses across platforms. In the context of the experiments here, we conclude that meaningful replication requires active intervention in order to keep the materials relevant and sensible. The second aim is to compare Amazon Mechanical Turk subjects to the original samples and to the replication samples. We find, consistent with the weight of recent evidence, that the Amazon Mechanical Turk samples are reasonably appropriate for these kinds of scenario studies. Subjects are highly similar to subjects on other online platforms and in‐person samples, though they differ in their high level of attentiveness. Finally, we review the growing replication literature across disciplines, as well as our firsthand experience, to propose a set of standard practices for the publication of results in law and psychology.

Cues to paternity: Do partner fidelity and offspring resemblance predict daughter-directed sexual aversions?

Abstract Despite the profound influence of relatedness on mating and cooperative behavior in humans, the cues men use to assess paternity and guide offspring-directed behavior have yet to be fully resolved. According to leading theories of kin detection, kinship cues should influence both sexual and altruistic motivations because of fitness consequences associated with inbreeding and welfare tradeoff decisions, respectively. Prior work with paternity assessment, however, has generally evaluated candidate cues solely by demonstrating associations with altruism. Here we (i) replicate past work that found effects of phenotypic resemblance and perceived partner fidelity on offspring investment; and (ii) evaluate whether both phenotypic resemblance and perceived partner fidelity meet the more stringent criteria suggested by theory—that is, whether they also predict inbreeding aversions. We report on two studies, one from a population-based sample of Finnish fathers ( N = 390), the other from a Mechanical Turk sample ( N = 700), and furnish evidence in strong support of perceived partner fidelity as a cue to paternity. Support for resemblance as a cue to paternity was decidedly weaker. We discuss a non-kin-based role that resemblance might play in altruistic decision-making, consider whether men might use additional kinship cues to meet the computational challenges associated with paternity assessment, and provide suggestions for future research.

Economic Myths Regarding Gay Men and Lesbians

ABSTRACTThe present research was aimed at developing a measure of economic myths regarding the gay community. Using undergraduate (N = 332) and Amazon Mechanical Turk (N = 212) samples, Study 1 resulted in a 6-item unidimensional and reliable measure, the Economic Myths Regarding Gays Scale (EMGS), that was minimally related to other measures of sexual orientation attitudes. Study 2 (N = 210 undergraduates) found that endorsement of economic myths was associated with system justification beliefs that economic inequalities were natural and due to individual merit; however, general attitudes toward gays showed no such relationships. Study 3 (N = 85 undergraduates) found that unlike general stereotyping, economic myth endorsement increased in response to induced economic threat from the gay community. Collectively, the present research suggests that the EMGS is largely distinct from other measures of attitudes and stereotyping and can potentially widen research regarding sexual orientation.

A cross-level investigation of informal field-based learning and performance improvements.

Organizations often operate in complex and dynamic environments which place a premium on employees' ongoing learning and acquisition of new competencies. Additionally, the majority of learning in organizations does not take place in formal training settings, but we know relatively little about how informal field-based learning (IFBL) behaviors relate to changes in job performance. In this study, we first clarified the construct of IFBL as a subset of informal learning. Second, on the basis of this clarified construct definition, we developed a measure of IFBL behaviors and demonstrated its psychometric properties using (a) a sample of subject matter experts who made item content validity judgments and (b) both an Amazon Mechanical Turk sample (N = 400) and a sample of 1,707 healthcare employees. Third, we advanced a grounded theory of IFBL in healthcare, and related it to individuals' regulatory foci and contextual moderators of IFBL behaviors-job performance relationships using a cross-level design and lagged nonmethod bound measures. Specifically, using a sample of 407 healthcare workers from 49 hospital units, our results suggested that promotion-focused individuals, especially in well-staffed units, readily engage in IFBL behaviors. Additionally, we found that the IFBL-changes in job performance relationship was strengthened to the extent that individuals worked in units with relatively nonpunitive climates. Interestingly, staffing levels had a weakening moderating effect on the positive IFBL-performance improvements relationship. Detailed follow-up analyses revealed that the peculiar effect was attributable to differential relationships from IFBL subdimensions. Implications for future theory building, research, and practice are discussed. (PsycINFO Database Record

A network of dark personality traits: What lies at the heart of darkness?

The question of whether there is a common element at the core of the various dark personality traits (e.g., psychopathy, narcissism, Machiavellianism, spitefulness, aggressiveness) has been the subject of debate. Callousness, manipulativeness, and disagreeableness have all been nominated as possibly serving as the core of these dark traits. Network analysis, which graphically and quantitatively describes the centrality of various related traits, provides a novel technique for examining this issue. We estimated an association network and an Adaptive Least Absolute Shrinkage and Selection Operator network for two large samples, one college student sample (N = 2831) and one mixed college student and Mechanical Turk sample (N = 844). Interpersonal manipulation and callousness were the traits that were central to the networks.

Law and Psychology Grows Up, Goes Online, and Replicates

Over the last 30 years, legal scholars have increasingly deployed experimental studies, particularly hypothetical scenarios, to test intuitions about legal reasoning and behavior. That movement has accelerated in the last decade, facilitated in large part by cheap and convenient Internet participant recruiting platforms like Amazon Mechanical Turk. The widespread use of online subjects, a practice that dramatically lowers the barriers to entry for experimental research, has been controversial. At the same time, the field of experimental psychology is experiencing a public crisis of confidence widely discussed in terms of the “replication crisis.” At present, law and psychology research is arguably in a new era, in which it is both an accepted feature of the legal landscape and also a target of fresh skepticism. The moment is ripe for taking stock. In this article, we bring an empirical approach to these problems. Using three canonical law and psychology findings, we document the challenges and the feasibility of reproducing results across platforms. We evaluate the extent to which we are able to reproduce the original findings with contemporary subject pools (Amazon Mechanical Turk, other national online platforms, and in‐person labs). We partially replicate all three results, and show marked similarities in subject responses across platforms. In the context of the experiments here, we conclude that meaningful replication requires active intervention in order to keep the materials relevant and sensible. The second aim is to compare Amazon Mechanical Turk subjects to the original samples and to the replication samples. We find, consistent with the weight of recent evidence, that the Amazon Mechanical Turk samples are reasonably appropriate for these kinds of scenario studies. Subjects are highly similar to subjects on other online platforms and in‐person samples, though they differ in their high level of attentiveness. Finally, we review the growing replication literature across disciplines, as well as our firsthand experience, to propose a set of standard practices for the publication of results in law and psychology.

Gambling and the Reasoned Action Model: Predicting Past Behavior, Intentions, and Future Behavior.

Gambling is a serious concern for society because it is highly addictive and is associated with a myriad of negative outcomes. The current study applied the Reasoned Action Model (RAM) to understand and predict gambling intentions and behavior. Although prior studies have taken a reasoned action approach to understand gambling, no prior study has fully applied the RAM or used the RAM to predict future gambling. Across two studies the RAM was used to predict intentions to gamble, past gambling behavior, and future gambling behavior. In study 1 the model significantly predicted intentions and past behavior in both a college student and Amazon Mechanical Turk sample. In study 2 the model predicted future gambling behavior, measured 2weeks after initial measurement of the RAM constructs. This study stands as the first to show the utility of the RAM in predicting future gambling behavior. Across both studies, attitudes and perceived normative pressure were the strongest predictors of intentions to gamble. These findings provide increased understanding of gambling and inform the development of gambling interventions based on the RAM.

The relationship between cognitive style and political orientation depends on the measures used

AbstractThe present research investigated the reason for mixed evidence concerning the relationship between analytic cognitive style (ACS) and political orientation in previous research. Most past research operationalized ACS with the Cognitive Reflection Test (CRT), which has been criticized as relying heavily on numeracy skills, and operationalized political orientation with the single-item self-placement measure, which has been criticized as masking the distinction between social and economic conservatism. The present research recruited an Amazon Mechanical Turk sample and, for the first time, simultaneously employed three separate ACS measures (CRT, CRT2, Baserate conflict problems), a measure of attitudes toward self-critical and reflective thinking (the Actively Open-Minded Thinking Scale; AOT), and separate measures of social and economic conservatism, as well the standard measure of political orientation. As expected, the total ACS score (combination of the separate measures) was negatively related to social, but not economic, conservatism. However, the CRT by itself was not related to conservatism, in parallel with some past findings, while the two other measures of ACS showed the same pattern as the combined score. Trait reflectiveness (AOT) was related negatively to all measures of political conservatism (social, economic, and general). Results clearly suggest that the conclusion reached regarding the ACS-political orientation relationship depends on the measure(s) used, with the measure most commonly employed in past research (CRT) behaving differently than other measures. Future research must further pursue the implications of the known differences (e.g., reliance on numeracy vs. verbal skills) of ACS measures and distinguish different senses of reflectiveness.

What If the Rival Drives a Porsche?

Previous research found that men conspicuously consume luxury products to attract a mate and to signal their mate value. However, these studies have yet neglected to investigate the function of male conspicuous consumption in same-sex competition. Given that intersexual selection and intrasexual selection are closely related processes, it stands to reason that a further function of male conspicuous consumption could be to impress and deter same-sex rivals. An 2 (intrasexual competition context vs. control) × 2 (conspicuous luxury vs. inconspicuous nonluxury) between-subjects experimental study conducted with an Amazon Mechanical Turk sample ( N = 160) revealed that men reported both higher liking of and an intent to purchase a conspicuous luxury car compared to an inconspicuous nonluxury car due to increased feelings of social status. This effect was stronger in the intrasexual competition than in the control context. An additional perception study using a single-factor between-subjects design (conspicuous luxury vs. inconspicuous nonluxury car) among German men ( N = 405) indicated that male participants rated a man who displayed a conspicuous luxury car more as a rival and mate poacher and less as a friend. They further perceived him to be superior on various mate value characteristics (i.e., attractiveness, intelligence, ambition, and status) and rated him as more oriented toward short-term mating. In sum, our findings add to previous research in the field of evolutionary consumer psychology by suggesting that male conspicuous consumption of luxuries may also serve a function in male–male competition.

Validating the B‐Scan Self: A self‐report measure of psychopathy in the workplace

The surging interest in corporate psychopathy has underscored the need for a reliable and valid measure of psychopathic features that is suitable for research in organizational settings. The B‐Scan Self is a new self‐report measure of corporate psychopathy that was developed with Hare's Psychopathy Checklist‐Revised (PCL‐R) as a framework. Validity studies, using two independent Mechanical Turk samples, were designed to examine its factor structure and validity. Results indicated that B‐Scan Self facets were internally consistent and unidimentional and strongly related to another self‐report measure of psychopathy (SRP‐III). Confirmatory factor analyses supported a reliable fifteen facets and four‐factor model consistent with the PCL‐R four‐factor model of psychopathy. Furthermore, B‐Scan Self facets were positively correlated with the Dark Triad of personality traits and negatively correlated with FFM traits of Agreeableness and Conscientiousness. More importantly, B‐Scan Self facets presented the same pattern of correlations with FFM traits as the SRP‐III and different patterns than the two other Dark Triad measures. Although this constitutes the first validation study of the B‐Scan Self and more research is needed, we believe that these results are encouraging and that the B‐Scan Self provides an opportunity to study psychopathic features through the measure of work‐related behavior.

Perceptions of e-cigarettes: a comparison of adult smokers and non-smokers in a Mechanical Turk sample

ABSTRACTBackground: Given plans to extend its regulatory authority to e-cigarettes, the Food and Drug Administration (FDA) urgently needs to understand how e-cigarettes are perceived by the public. Objectives: To examine how smoking status impacts adult perceptions and expectations of e-cigarettes. Methods: We used Mechanical Turk (MTurk), a “crowdsourcing” platform, to rapidly survey a large (n = 796; female = 381; male = 415), diverse sample of adult ever (44%) and never smokers (56%), including ever (28%) and never (72%) users of e-cigarettes. Results: Smokers and non-smokers learned about e-cigarettes primarily through the internet and conversations with others. Ever smokers were more likely than never smokers, and female current smokers were more likely than female former smokers, to have learned about e-cigarettes from point of sale advertising (p’s < 0.05) and to believe that e-cigarettes help smokers quit (ps < 0.05). Among never users of e-cigarettes, current smokers were more likely than never smokers and former smokers to report that they would try e-cigarettes in the future (ps < 0.01). Current smokers’ top reason for wanting to try e-cigarettes was to quit or reduce smoking (56%), while never and former smokers listed curiosity. In contrast, female current smokers’ top reason for not trying e-cigarettes was health and safety concerns (44%) while males were deterred by expense (44%). Conclusions: Adult smokers and non-smokers have different perceptions and expectations of e-cigarettes. Public health messages regarding e-cigarettes may need to be tailored separately for persons with and without a history of using conventional cigarettes. Tailoring messages by gender within smoker groups may also improve their impact.

Finding meaning in a celebrity’s death: The relationship between parasocial attachment, grief, and sharing educational health information related to Robin Williams on social network sites

Celebrity health events can have a positive effect on public health education and awareness, in part because people spread health information related to the celebrity on social network sites. This may be particularly true in the event of celebrity deaths or other tragedies, when social network sites provide a convenient, public space to connect with others to mourn. However, little is understood about the circumstances that encourage health information sharing following these events. Using a Mechanical Turk sample (N = 281), this study employed a questionnaire to examine the role of parasocial attachment to Robin Williams and experienced grief following his suicide in explaining health information sharing on social network sites (i.e., sharing information about clinical depression, suicide prevention, substance abuse treatment, and Parkinson’s disease). Grief mediated the effect of parasocial attachment on health information sharing, but a moderated mediation analysis revealed that grief motivated health education on social network sites only for individuals who were coping by trying to find positive meaning in his death.

Exploring Citizen Assessments of Unilateral Executive Authority

This study investigates the interaction between constitutional considerations and democratic context in evaluations of executive authority. An identical experiment is conducted using undergraduate and Mechanical Turk samples. A hypothetical article raising the question of executive power varies the (1) issue context, (2) expert assessment of constitutional authority, and (3) level of public support for proposed action. Measures of participants’ issue preferences and level of satisfaction with President Obama are also included in the analysis. Results indicate that participants think differently about the desirability and legitimacy of proposed executive action. Constitutional considerations and satisfaction with the President weigh most heavily in assessments of the appropriateness of executive conduct. Differences observed across samples demonstrate that institutional rules have the potential to constrain the influence of political factors in assessments legitimacy, but this is not inevitably the case. Feelings about the President may be especially important where experts agree that he lacks authority to take action. This could indicate that citizens will rally behind presidents they like, and think more critically of those they do not, in times of constitutional crisis.

Psychological research in the internet age: The quality of web-based data

The internet is increasingly used in psychological research to solicit participants and collect data. This paper includes two studies examining the quality of data obtained via web-based methods administered either inside or outside the lab. Both studies used item recognition accuracy as a proxy for attention to questions. Study 1 examined the extent to which undergraduate participants (N = 504) read and attended to questions either inside or outside the lab. Study 2 (N = 744) replicated Study 1, added a Mechanical Turk sample, and examined attention to non-intuitive survey instructions. Results indicated that participants demonstrated good item recognition, regardless of locale or sample; however, small sex effects on accuracy were found in both studies. Specifically, women were more accurate at identifying previously seen items than men in both Study 1 and Study 2. In Study 2, Mechanical Turk participants were more likely to read instructions than undergraduate participants, regardless of whether they participated inside or outside of the lab. The findings support the use of the internet for sampling purposes as well as survey administration, and suggest that researchers use care when studies include non-intuitive instructions.

Comparing In-Person, Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs

Electronic data collection and participant pool management tools give researchers new ways to conduct research. The current study investigated the equivalency of in-person and online administrations of the Right-Wing Authoritarianism, Social Dominance Orientation, and Modern Racism scales across three modalities (administration in person, online through Sona Systems, and online through Mechanical Turk). Results indicate that in-person administration was largely equivalent to the randomly assigned online sample (Sona Systems) in terms of their intercorrelations, mean scores, variability, and reliability. However, the Sona sample consistently responded with strong attitudes for all measures, suggesting that social desirability may be decreased when completing these measures online. In addition, the Mechanical Turk sample differed in many ways from both in-person and Sona administration; although this nonequivalence is at least partially explained by sample demographic differences, other considerations may have exacerbated nonequivalence, including prior exposure to the measures and a desire to respond correctly.

The Demographic and Political Composition of Mechanical Turk Samples

One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.

Quantifying the Qualitative: Measuring the Insight Experience

No scales currently exist that measure variability in the insight experience. Two scales were created to measure two factors hypothesized to be key drivers of the insight experience: insight radicality (i.e., perceived deviation between previous and new problem representations) and restructuring experience (i.e., the subjective experience of the problem representation being restructured). Data were collected online in two studies using a Mechanical Turk sample (N = 212) and a Zoomerang sample (N = 168) that reported on past experiences working to solve a challenging problem at work. Insight radicality and restructuring experience each positively correlated with and accounted for unique variance in (a) the nature of insight experiences (e.g., intensity), (b) insight terminology used to describe them (e.g., “a breakthrough”), and (c) perceived insight impact (e.g., on field/discipline). The 2 scales accounted for unique variance in these variables both individually and above and beyond improved processing fluency from the insight. The new scales offer a means of establishing the generalizability of experimental findings to insights in natural settings and of more precisely characterizing insight experiences across both experimental and natural settings.