Mechanical Turk Sample Research Articles

Development and Validation of the Five-Factor Borderline Inventory-Super Short Form and Screener.

The Five-Factor Borderline Inventory (FFBI) and FFBI-Short Form (FFBI-SF) are 120-item and 48-item measures that assess the underlying maladaptive personality traits of borderline personality disorder (BPD). The purpose of this study was to develop a super short form (FFBI-SSF) and an FFBI-Screener to facilitate the use of dimensional trait measures for BPD. Using item response theory analyses, the 48-item measure was reduced to 22 items using a large undergraduate sample (N = 1300) and then retested using a Mechanical Turk sample (N = 602), demonstrating strong replicability. IRT was again used to further reduce the measure from 22 items to four items to provide a brief screening tool. Correlations of the FFBI-SSF and Screener with measures of BPD-related variables were compared across five samples (N = 919, 204, 580, 281, and 488). Overall, the FFBI-SSF showed similar relations to the FFBI-SF at the full scale and domain-level scales, while the FFBI-screener demonstrated similar relations at the full scale level. This super short form and screener may best be used in large-scale research studies or as part of a screening tool in clinical settings.

Prescription for Love: An Experimental Investigation of Laypeople’s Relative Moral Disapproval of Love Drugs

New technologies regularly bring about profound changes in our daily lives. Romantic relationships are no exception to these transformations. Some philosophers expect the emergence in the near future of love drugs: a theoretically achievable biotechnological intervention that could be designed to strengthen and maintain love in romantic relationships. We investigated laypeople’s resistance to the use of such technologies and its sources. Across two studies (Study 1, French and Peruvian university students, N after exclusion = 186; Study 2, Amazon Mechanical Turk sample, N after exclusion = 693, pre-registered), we found that the use of love drugs designed to strengthen and maintain love in romantic relationships are considered as more morally problematic than psychological therapy with the same aim. In Study 2, we show that this last effect is partially due to the fact that the love resulting from the use of love drugs is perceived as less authentic, intense, and durable. We discuss the specific role of authenticity in the relative moral disapproval of love drugs.

Measurement invariance of a Posttraumatic Stress Disorder symptoms measure (PCL-5) in college student and Amazon's mechanical TURK samples.

College student and Amazon's Mechanical TURK (MTURK) samples are regularly utilized in trauma research. Recent literature, however, has criticized these samples for not being generalizable to the general U.S. The purpose of this study was to determine whether college student (n = 255) and MTURK (n = 316) samples are invariant on the Posttraumatic Stress Disorder Checklist for DSM-5. Measurement invariance using confirmatory factor analyses was used to determine whether groups are invariant across factor structure, factor loadings, item intercepts, and residual error variances on a given measure of Post-traumatic Stress Disorder (PTSD) symptom severity. Model fit indices indicated the seven-factor Hybrid model was the best-fitting model, but the six-factor Anhedonia model was the most parsimonious model. Both models demonstrated equivalence in factor at the strictest level, indicating MTURK and college student samples are similar in regard to PTSD symptom severity. These findings provide evidence that these groups can be combined in future studies to increase sample size for trauma research. Only the Anhedonia factor exhibited mean differences between groups, which may be related to true differences between college students and MTURK survey-takers. This study provides further evidence that the findings from trauma studies using these populations are generalizable to each other. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Outcomes Affect Evaluations of Decision Quality: Replication and Extensions of Baron and Hershey’s (1988) Outcome Bias Experiment 1

Outcome bias is the phenomenon whereby decisions which resulted in successful outcomes were rated more favorably than when the same decisions resulted in failures. We conducted a pre-registered replication and extension of Experiment 1 (original’s: N = 20) from the classic Baron and Hershey (1988) with an online Amazon Mechanical Turk sample using CloudResearch (N = 692), switching from a within-participants design in the original experiment to a between-participants design. We tested outcome bias by measuring participants’ ratings of the quality of decisions in medical scenarios. For the replication (pre-registered) part of the study, we successfully replicated signal and direction of the outcome bias (original: dpaired = 0.21 – 0.53; replication: dindependent = 0.77 [0.62, 0.93] to 1.1 [0.94, 1.26]), and even for participants who stated that outcomes should not be taken into consideration when evaluating decisions (d = 0.64 [0.21, 1.08]). For the extension part of the study, we found differences, dependent on outcome types, in evaluations of the perceived importance of considering the outcome, the perceived responsibility of decision-makers, and the perception that others would act similarly given the choice by outcome type. Materials, data, and code are available on Open Science Framework (OSF): https://osf.io/knjhu/.

Revisiting the Links Between Numeracy and Decision Making: Replication Registered Report of Peters et al. (2006) With an Extension Examining Confidence

Numeracy is individuals’ capacity to understand and process basic probability and numerical information required to make decisions. We conducted a Replication Registered Report of Peters et al. (2006) examining numeracy as a predictor of positive-negative framing effect (Study 1), frequency-percentage effect (Study 2), ratio effect (Study 3), and bets effect (Study 4). With an online US American Amazon Mechanical Turk sample (N = 860), our replication using the target’s dichotomizing of the numeracy measure found support for the original findings regarding interactions between numeracy and three decision-making effects. Numeracy was associated with weaker framing effect (η2p = 0.01, 90% CI [0.00, 0.02]), weaker ratio bias (Cramer’s V = 0.17, 95% CI [0.10, 0.24]), and stronger bets effect (η2p = 0.02, 90% CI [0.01, 0.04]), yet we found no support for the frequency-percentage effect (η2p = 0.00, 90% CI [0.00, 0.01]). However, we found support for associations with all four studies when treating numeracy as a continuous variable. We extended the replication to examine confidence, yet the results were mixed with support found for only three conditions (Study 1 positive framing condition: r = -0.11, 95% CI [-0.20, -0.02]; Study 3: r = 0.15, 95% CI [0.08, 0.21]; Study 4 no-loss bet condition: r = 0.10, 95% CI [0.01, 0.20]), suggesting a much weaker and more complex relationship than anticipated. Materials, data, and code are available on: https://osf.io/4hjck/.

Supplemental Material for Measurement Invariance of a Posttraumatic Stress Disorder Symptoms Measure (PCL-5) in College Student and Amazon’s Mechanical TURK Samples

Supplemental Material for Measurement Invariance of a Posttraumatic Stress Disorder Symptoms Measure (PCL-5) in College Student and Amazon’s Mechanical TURK Samples

Revisiting the Role of Pornography Use in the Confluence Model Theory of Sexual Aggression

ABSTRACT This research tested Confluence Model reasoning that pornography use should be related to sexual aggression among men who are high but not low in the predisposing risk factors of hostile masculinity (HM) and impersonal sexuality (IS). This hypothesis was examined with three online surveys of young adult males, including an American Mechanical Turk sample (N 1 = 1,528, Mage = 22.46 years); a national sample of Canadian students (N 2 = 1,049, Mage = 20.89 years); and a national sample of Canadian non-students (N 3 = 905, Mage = 21.66 years). As expected, synergistic interactions between HM and IS reliably predicted self-reported sexual aggression across samples. Results with respect to interactions with pornography use were more complex. The Confluence Model hypothesis was supported when pornography use was operationalized as the use of nine specific magazines but it was not supported when pornography use was operationalized with a contemporary inclusive approach that included use of internet materials. These discrepant findings are difficult to account for with Confluence Model theorizing and highlight the non-equivalence of pornography use measures in survey research.

A trifecta: patterns in orgasm consistency, relational satisfaction, and sexual satisfaction.

We based this study on the sexual wholeness model, which considers emotional, physical, and meaning elements of sexuality, and previous research that indicates variability in the associations among orgasm consistency, relational satisfaction, and sexual satisfaction. We performed a person-based latent class approach to examine variables linked with the trifecta of orgasm consistency, relational satisfaction, and sexual satisfaction. We used a Mechanical Turk sample that consisted of 1645 sexually active participants (1032 women) to evaluate classes and related variables. Women divided into 5 classes and men into 3 classes. High orgasm consistency alone did not ensure sexual or relational satisfaction. We found that 5 groups emerged for women: high orgasm/low satisfaction (6%), low trifecta (8%), low orgasm/high satisfaction (16%), moderate orgasm/high satisfaction (15%), and high trifecta (55%). For men, 3 groups emerged: lower orgasm/high satisfaction (12%), moderate trifecta (14%), and high trifecta (74%). We also found that high orgasm consistency for women did not guarantee relational and sexual satisfaction, nor did lower orgasm consistency for men or women guarantee problematic relational and sexual satisfaction. Communication, attachment, and sexual harmony were key elements associated with positive trifecta outcomes. Clinicians and educators may emphasize the importance of these variables when individuals encounter sexual or relational distress. Data analytic techniques revealed a variety of groups for men and women and associations between a host of physical, psychological, and meaning variables and the trifecta of orgasm consistency, relational satisfaction, and sexual satisfaction. However, this study is cross-sectional and correlational, which limits the conclusions that we can draw from it. While the variables of orgasm consistency, relational satisfaction, and sexual satisfaction are each informative, together they help clinicians, educators, and researchers better understand the complexity of sex.

Revisiting the Differential Centrality of Experiential and Material Purchases to the Self: Replication and Extension of Carter and Gilovich (2012)

Carter and Gilovich (2012) investigated the centrality of experiential and material purchases to the self and concluded people have stronger associations with their experiential than their material purchases. In a pre-registered experiment with a US American Amazon Mechanical Turk sample, (N=743), we successfully replicated their Studies 3A, 3B, 3C, and 5. Experiential purchases were perceived as more reflective of true-self than material purchases for both self (d=0.65[0.57,0.73]) and for strangers (d=0.88[0.80,0.96]), and that when meeting a new person, information about experiential purchases was considered to be more insightful (d=1.13[1.04,1.22]), useful (d=1.14[1.05,1.23]), and fun to talk about (d=1.96[1.83,2.08]) than material purchases. Self-concept was more strongly associated with experiential purchases than with material purchases (d=0.39[0.25,0.54]), and that there was a negative association between experiential purchase satisfaction and the willingness to exchange memory (r=-.34[-.43,-.24]) (all effects above were p<.001). We added an extension examining change in evaluations of material and experiential purchases over time and found that current evaluations were more negative than past evaluations, yet to a lesser extent for experiential compared to material purchases. Materials, data, and code are available on: https://osf.io/v2w5h/

The destination marketing and media profile – travel motives nexus amid tourism crisis: the mediating effect of the nation brand

PurposeThe ongoing COVID-19 pandemic highlights the importance of destination marketing and media profiling to re-engage international tourists. However, potential crisis-induced nation brand (NB) deficits must be addressed to re-ignite tourism demand. The study examines the possible intervening effect of the contemporary NB in the international destination marketing and media-travel motives nexus.Design/methodology/approachA deductive quantitative study was undertaken with an online Amazon Mechanical Turk sample of n = 454 respondents. Hypotheses were tested using PROCESS Macro, Model 4.FindingsThe results show that the NB [people and negative events] had a practically significant partial mediating effect in the destination marketing – nature-cultural oriented travel motivation nexus.Practical implicationsNew insights are provided via a practical model which facilitates the measurement of potential nuances in the influence of destination marketing and media profiling on leisure tourists' travel motives amid crises. The intervening effect implies that a better understanding of the NB as an indirect antecedent to travel motivation may result in more effective crisis communications and tourism recovery-oriented marketing.Originality/valueThe study is amongst the first to extend marketing and behavioural theory to explore the interplay between the marketing and media profile, a nation's brand and tourists' travel behaviour amid a crisis. The study addresses a discernible dearth of knowledge related to the influence of the NB on tourist behaviour from an emerging market perspective.

Comparing Data Collected on Amazon's Mechanical Turk to National Surveys.

In this study, we examined the impact of a range of methods to improve data quality on the demographic and health status representativeness of Amazon Mechanical Turk (MTurk) samples. We developed and field-tested a general survey of health on MTurk in 2017 among 5755 participants and 2021 among 6752 participants. We collected information on participant demographic characteristics and health status and implemented different quality checks in 2017 and 2021. Adding data quality checks generally improves the representativeness of the final MTurk sample, but there are persistent differences in mental health and pain conditions, age, education, and income between the MTurk population and the broader US population. We conclude that data quality checks improve the data quality and representativeness.

The Roles of Gender and Employment Status in Abdication Decisions.

Abdication-the act of giving up a choice to another person-has been shown to prompt reciprocal generosity (Kardas et al., 2018), but it is not clear whether men or women are more likely to engage in abdication. The present research utilized a gift card scenario adapted from Kardas et al. (2018) in two Amazon Mechanical Turk samples to explore whether women abdicated more frequently than men, whether men abdicated to women less frequently than to other men, and whether employment status predicts abdication. In experiment one, participants (N = 520) were assigned to one of three conditions in which they had to either choose who received a hypothetical gift card themselves or abdicate the decision to a friend (sex not specified, male friend, or female friend). Chi-square analyses revealed no significant differences between participants who abdicated and those who allocated; men and women had similar abdication patterns; there was no significant difference in abdication as a function of friend gender for male or female participants. In experiment two, participants (N = 707) were again assigned to one of the three conditions but were also asked their employment status. Chi-square analyses indicated that participants generally preferred to be allocators rather than abdicators; women showed a similar abdication pattern to men; there was no significant difference in abdication as a function of friend gender for male or female participants. Lastly, unemployed participants abdicated more frequently (70%) than their employed counterparts (42%), and this was especially likely for women. These results have implications for potential factors that influence abdication decisions.

Characteristics and clinical implications of the relationship between veganism and pathological eating behaviours.

Veganism may serve as a socially acceptable means to restrict food intake and disguise pathological eating behaviours. Studies that include vegan participants typically group them with other meat avoiders (e.g., vegetarians), potentially masking risk factors unique to veganism. We addressed this issue by recruiting two Amazon Mechanical Turk samples of 110 vegan and 118 omnivore participants, with comparable gender composition. We aimed to examine whether vegans showed higher disordered eating than omnivores, and if motives for pursuing a vegan diet impacted disordered eating. We assessed disordered eating using the Eating Attitudes Test, the Eating Disorder Examination-Questionnaire, and the Eating Pathology Symptom Inventory. Vegans displayed more pathological eating behaviours than omnivores, which was significantly predicted by cognitive restraint. However, body dissatisfaction levels were higher in omnivores than vegans. Diet motives did not influence vegans' disordered eating. We propose vegans have high levels of cognitive restraint, possibly due to their intention to avoid animal products. In turn, cognitive restraint subscales in eating disorder measures might be over-pathologising rates of eating disorders in vegans. Future research should monitor the progression of people's eating-related attitudes and behaviours before and after they transition to veganism to establish whether veganism increases the risk of disordered eating, or vice versa. IV, cross-sectional study.

Perceived morality of direct versus indirect harm: Replications of the preference for indirect harm effect

Royzman and Baron (2002) demonstrated that people prefer indirect harm to direct harm: they judge actions that produce harm as a by-product to be more moral than actions that produce harm directly. In two preregistered studies, we successfully replicated Study 2 of Royzman and Baron (2002) with a Hong Kong student sample (N = 46) and an online American Mechanical Turk sample (N = 314). We found con- sistent evidential support for the preference for indirect harm phenomenon (d = 0.46 [0.26, 0.65] to 0.47 [0.18, 0.75]), weaker than effects reported in the original findings of the target article (d = 0.70 [0.40, 0.99]). We also successfully replicated findings regarding reasons underlying a preference for indirect harm (di- rectness, intent, omission, probability of harm, and appearance of harm). All materials, data, and code are available at osf.io/ewq8g.

Competitive gossip: the impact of domain, resource value, resource scarcity and coalitions.

Those with better reputations often obtain more resources than those with poorer reputations. Consequently, gossip might be an evolved strategy to compete for valuable and scarce material and social resources. Influenced by models of non-human primate competition, we test the hypotheses that gossip: (i) targets aspects of reputation relevant to the domain in which the competition is occurring, (ii) increases when contested resources are more valuable, and (iii) increases when resources are scarcer. We then test hypotheses derived from informational warfare theory, which proposes that coalitions strategically collect, analyse and disseminate gossip. Specifically, we test whether: (iv) coalitions deter negative gossip, and (v) whether they increase expectations of reputational harm to competitors. Using experimental methods in a Mechanical Turk sample (n = 600), and survey and ego network analysis methods in a sample of California sorority women (n = 74), we found that gossip content is specific to the context of the competition; that more valuable and scarcer resources cause gossip, particularly negative gossip, to intensify; and that allies deter negative gossip and increase expectations of reputational harm to an adversary. These results support social competition theories of gossip. This article is part of the theme issue 'The language of cooperation: reputation and honest signalling'.

Worker Affectivity and Work Attitudes: A Meta-Analysis of Mechanical Turk Samples

The purpose of this paper is to meta-analyze the relationship between (positive and negative) affect and work attitudes (job satisfaction, organizational commitment, and turnover intentions) for Mechanical Turk data samples. MTurk has become a popular convenience sample choice for OB/HR scholars. MTurk workers are also part of the evolving digital gig economy, for whom the attitudinal and behavioral outcomes may differ from traditional employees due to the former’s varying employment relationships. We analyzed a total of 50 papers and 55 unique samples consisting of 23,427 MTurkers for PA studies and 14,798 MTurkers for NA studies. Consistent with conventionally sourced samples, PA was positively related to job satisfaction and organizational commitment and NA was negatively related to turnover intentions. The effect size magnitude for MTurker job satisfaction was statistically different and higher than conventional samples. Implications and the need to conduct such meta-analytic reviews for other attitudinal and behavioral outcome variables for MTurkers are discussed.

Psychometric Properties of the AUDIT-C within an Amazon Mechanical Turk Sample

Objectives: Amazon's Mechanical Turk (MTurk) has become a popular data collection tool in the addiction sciences. We sought to examine the psychometric properties of the AUDIT-C in an MTurk sample. Methods: Data collection was facilitated via MTurk (N=309; 52.8% female), where an online survey assessed demographic data, alcohol use behaviors (AUDIT-C), and alcohol-related consequences (CAPS-r). Responses to the AUDIT-C were subjected to a principal component analysis to evaluate the structure of the 3-item measure. Alcohol-related consequences were used as a measure of convergent validity. Results: Results provided evidence for a single-factor structure. Pearson's product-moment correlation coefficients between AUDIT-C scores and CAPS-r scores produced statistically significant results (r = 0.51, p < .001). Using biological sex-based suggested cut-off scores for the AUDIT-C, hazardous drinkers (M = 19.15, SD = 8.27) demonstrated statistically significantly higher levels of alcohol-related consequences than non-hazardous drinkers (M = 12.56, SD = 5.35; t(295) = -8.34, p < .001). Reliability and stability statistics demonstrated strong internal consistency. Conclusions: Results demonstrate the sound psychometric properties of the AUDIT-C for an MTurk sample and provide evidence supporting the use of AUDIT-C as a screening tool to be employed with digitally accessed populations to identify and reach hazardous drinkers.

Orthorexia nervosa vs. healthy orthorexia: relationships with disordered eating, eating behavior, and healthy lifestyle choices

BackgroundOrthorexia nervosa (OrNe) is a pattern of disordered eating behavior characterized by excessive preoccupation with overvalued ideas about healthy eating. Healthy orthorexia (HeOr) refers to a non-pathological interest in healthy eating and nutrition. Despite converging evidence that OrNe is correlated with measures of psychopathology and personality traits, there is relatively little empirical data relating OrNe and HeOr to eating and health-promoting behavior.MethodsAim 1: exploratory structural equation modeling was used to evaluate the factor structure of an English-language version of the Teruel Orthorexia Scale, the only measure of OrNe that also assesses HeOr in two samples (Yoga practitioners, N = 469, and Mechanical Turk workers, N = 453). Aim 2: conducted in the Mechanical Turk sample, partial correlation and linear regression analyses were used to evaluate the relationships of OrNe and HeOr with other symptoms of, and clinical impairment from, disordered eating, food choices, diet quality, body mass index (BMI), nutrition knowledge, alcohol/tobacco/vaping use, and sedentary behavior.ResultsThe two-factor structure of the TOS was replicated. HeOr and OrNe had opposing relationships with measures of disordered eating behavior and distress, food choices, diet quality, and nutrition knowledge. HeOr was negatively related to BMI, whereas OrNe was positively related to substance use and sedentary behavior.ConclusionsOrNe and HeOr are distinct latent constructs, with the latter reflecting non-pathological behavior. Only OrNe is related to elevated disordered eating and clinical impairment. Despite involving obsessions and compulsions related to healthy eating, OrNe was associated with relatively unhealthy eating and other lifestyle behaviors.Level of evidenceLevel V, descriptive cross-sectional study.

Correlates of obsessive-compulsive and related disorders symptom severity during the COVID-19 pandemic

We investigated changes in the severity of obsessive-compulsive and related disorders (OCRDs) symptoms as a result of the COVID-19 pandemic. An Amazon Mechanical Turk sample of 829 individuals was evaluated with a series of instruments assessing the severity of the OCRDs before and during the pandemic. Additional questionnaires about sociodemographic factors, personal and family histories of OCRD, COVID-19 related events, compulsivity and impulsivity traits, schizotypal symptoms, and the severity of depression, anxiety and stress levels, were also used. Participants reported that OCD, hoarding disorder (HD) and skin picking disorder (SPD) symptoms significantly worsened during the pandemic along with increased disability, more affective symptoms and reduced quality of life. Female gender, a higher number of COVID-19 related stressful events, and higher pre-COVID-19 fear of harm and symmetry symptoms predicted more severe OCD symptoms during the pandemic, whereas lack of a HD diagnosis by a mental health professional and more severe schizotypal symptoms predicted worsened hoarding symptoms. Greater compulsivity traits were associated with more severe COVID-19 pandemic obsessive-compulsive and hoarding symptoms. These data indicate that the immense distress resulting from the COVID-19 included significant deterioration of OCRDs' symptoms, particularly of OCD, HD and SPD. It was also possible to identify a pre-pandemic profile of people most at risk of pandemic-related deterioration in OCRDs' symptoms, which may prove valuable for preventative initiatives in relation to the likely future waves of COVID-19 or of other communicable diseases. Future studies should follow up these findings longitudinally.

Sensitivity to Sunk Costs Depends on Attention to the Delay.

In the WebSurf task, humans forage for videos paying costs in terms of wait times on a time-limited task. A variant of the task in which demands during the wait time were manipulated revealed the role of attention in susceptibility to sunk costs. Consistent with parallel tasks in rodents, previous studies have found that humans (undergraduates measured in lab) preferred shorter delays, but waited longer for more preferred videos, suggesting that they were treating the delays economically. In an Amazon Mechanical Turk (mTurk) sample, we replicated these predicted economic behaviors for a majority of participants. In the lab, participants showed susceptibility to sunk costs in this task, basing their decisions in part on time they have already waited, which we also observed in the subset of the mTurk sample that behaved economically. In another version of the task, we added an attention check to the wait phase of the delay. While that attention check further increased the proportion of subjects with predicted economic behaviors, it also removed the susceptibility to sunk costs. These findings have important implications for understanding how cognitive processes, such as the deployment of attention, are key to driving re-evaluation and susceptibility to sunk costs.