MTurk Workers Research Articles

The Value of Activity Traces in Peer Evaluations: An Experimental Study

Peer evaluations are a well-established tool for evaluating individual and team performance in collaborative contexts, but are susceptible to social and cognitive biases. Current peer evaluation tools have also yet to address the unique opportunities that online collaborative technologies provide for addressing these biases. In this work, we explore the potential of one such opportunity for peer evaluations: data traces automatically generated by collaborative tools, which we refer to as "activity traces". We conduct a between-subjects experiment with 101 students and MTurk workers, investigating the effects of reviewing activity traces on peer evaluations of team members in an online collaborative task. Our findings show that the usage of activity traces led participants to make more and greater revisions to their evaluations compared to a control condition. These revisions also increased the consistency and participants' perceived accuracy of the evaluations that they received. Our findings demonstrate the value of activity traces as an approach for performing more reliable and objective peer evaluations of teamwork. Based on our findings as well as qualitative analysis of free-form responses in our study, we also identify and discuss key considerations and design recommendations for incorporating activity traces into real-world peer evaluation systems.

High levels of loneliness in people with hoarding disorder

Attachment theory suggests that hoarding is an attempt at compensating for unmet relatedness needs. We therefore expect high loneliness levels in hoarding disorder (HD). While previous studies have shown a positive association between hoarding and loneliness in non-clinical samples, few studies have examined loneliness levels in clinical HD samples. In Study 1, we examined loneliness in a treatment-seeking HD sample (n = 39). Results showed that 87.2% of HD sufferers reported high loneliness levels, which was significantly higher than loneliness rates in community samples. Loneliness was positively associated with hoarding severity even after controlling for depression. In Study 2, we examined loneliness in MTurk workers with clinical levels of hoarding (high hoarding [HH] group; n = 305) compared to MTurk workers with low hoarding symptoms (LH group; n = 775). Results showed high loneliness levels in 77.7% of participants in the HH group compared to 36.8% in the LH group. Differences remained significant for HH vs LH groups matched on depression. The positive association between loneliness and hoarding was also significant even after controlling for age, gender, marital status, and depression. Our results underscore the importance of assessing and addressing loneliness in the HD treatment.

A novel experimental investigation of online imagery rescripting for obsessive-compulsive prospective imagery

Emerging evidence suggests that rescripting intrusive mental imagery may reduce OCD symptoms. We tested whether a similar process could be adapted to an online experimental format. Amazon mTurk workers high in OCD traits (n = 198) were asked to identify OCD-related imagery of the future. They were then randomised to one of three audio-guided imagery conditions: rescripting, exposure, or a distracting control. We found that self-appraisals improved across all three conditions. Participants in the rescripting condition also reported greater reductions in aversive emotions, the vividness of their imagery, and their urge to neutralise the imagery, compared to those in the exposure condition. Results from the control condition were consistent with distraction. These experimental findings suggest that imagery rescripting in a standardised audio-guided format may facilitate emotional processing. Suggestions to improve on methodological limitations are discussed.

Psychological flexibility prospectively predicts COVID-19 PPE use across a two-month interval

ABSTRACT Perceived susceptibility, psychological flexibility and health behaviors (PPE use, social distancing) were measured at two time points spaced 2 months apart during the height of the COVID-19 pandemic and lockdown in the U.S.A (Time 1 April 2020; Time 2 June 2020). Demographic variables were also collected. Data from 151 Mturk workers indicated that Time 1 psychological flexibility significantly predicted PPE use and social distancing. The effect sizes were substantial. Perceived susceptibility was not a significant predictor. Psychological flexibility messaging and interventions could be an important way to increase the likelihood of people performing health protective behaviors to better constrain the continuing COVID-19 outbreak.

Adjusting to the COVID-19 outbreak in the United States: The impact of disruptions on habits and changes in health behaviors.

The COVID-19 pandemic provides a naturalistic test of whether pandemic-related disruptions weaken habits and undermine behavior stability. We hypothesized that better capacity to effortfully guide behavior (self-regulation) would buffer this effect and be associated with behavior stability and development of new habits to accomplish daily behaviors. A cross-sectional study of 416 MTurk workers recruited in April 2020 (Mage = 34.60, SD = 11.51) indicated that pandemic-related disruptions generally exceeded people's capacity to effortfully modify their behavior. Self-regulation related to the development of new habits and to lower likelihood that work productivity decreased. Self-regulation also protected against the effect of disruption on the likelihood that substance use increased. Besides these associations, self-regulation was largely unrelated to health-related behaviors and, in some instances, associated with poorer outcomes. These findings underscore the need to appreciate the impact of contextual disruptions in interpreting and promoting change in health-related behaviors.

Police body-worn camera footage: Attitudes toward the public right to view and police communication

ABSTRACT The aim of this study is to learn more about the factors that influence citizen perceptions of the public right to view BWC footage of a critical incident as well as police communication afterwards. To accomplish this, we used a 2 × 2 vignette where citizen race and object were manipulated to different conditions. Using a survey instrument sent to Amazon Mturk workers, several notable findings related to demographic and media perceptions were found regarding the public’s right to view the footage. White respondents were more likely to believe that the public should be able to view the footage involving the Black suspect holding a cellphone, whereas White respondents were less likely to believe that the public should be able to view the footage involving a Black suspect holding a handgun. Age was positively related to supporting increased communication from police for the scenario with a Black suspect with a cellphone,among several other significant relationships. As media hostility increased, respondents were less likely to support increased communication from the police. Perceptions of media hostility consistently influenced respondent perceptions of the public’s right to view BWC footage and police communication across all manipulations.

You never get a chance to undo a negative first impression: Social anxiety is associated with impaired positive updating of social information

In an ever-changing social world, learning and updating beliefs about others are essential for smooth interpersonal functioning. Social anxiety is a common and burdensome condition involving difficulties in interpersonal functioning. However, the processes governing the learning and updating of beliefs regarding others, processes crucial for these interactions, are poorly understood. In order to address this gap, we used a novel modification of a reversal-learning task. The task consisted of two phases. In the first (learning) phase, participants learned that interactions with certain individuals were associated with negative outcomes and other individuals with positive outcomes. In the second (updating) phase, these associations were reversed. Hence, negative individuals became positive and vice-versa, and participants had to update their initial beliefs. Study 1 (n = 87; undergraduate students) revealed that social anxiety was not associated with biases in learning positive or negative information about others. However, social anxiety was associated with a deficit in positively updating existing negative beliefs. Study 2 (n = 248; Mturk workers) replicated these findings in a representative and demographically diverse sample, controlling for depression severity and age. The current research suggests that social anxiety-related difficulty in the positive updating of negative social information may contribute to the impairment in interpersonal functioning.

Who's in the Crowd Matters: Cognitive Factors and Beliefs Predict Misinformation Assessment Accuracy

Misinformation runs rampant on social media and has been tied to adverse health behaviors such as vaccine hesitancy. Crowdsourcing can be a means to detect and impede the spread of misinformation online. However, past studies have not deeply examined the individual characteristics - such as cognitive factors and biases - that predict crowdworker accuracy at identifying misinformation. In our study (n = 265), Amazon Mechanical Turk (MTurk) workers and university students assessed the truthfulness and sentiment of COVID-19 related tweets as well as answered several surveys on personal characteristics. Results support the viability of crowdsourcing for assessing misinformation and content stance (i.e., sentiment) related to ongoing and politically-charged topics like the COVID-19 pandemic, however, alignment with experts depends on who is in the crowd. Specifically, we find that respondents with high Cognitive Reflection Test (CRT) scores, conscientiousness, and trust in medical scientists are more aligned with experts while respondents with high Need for Cognitive Closure (NFCC) and those who lean politically conservative are less aligned with experts. We see differences between recruitment platforms as well, as our data shows university students are on average more aligned with experts than MTurk workers, most likely due to overall differences in participant characteristics on each platform. Results offer transparency into how crowd composition affects misinformation and stance assessment and have implications on future crowd recruitment and filtering practices.

The iconic motivation for the morphophonological distinction between noun-verb pairs in American Sign Language does not reflect common human construals of objects and actions.

Across sign languages, nouns can be derived from verbs through morphophonological changes in movement by (1) movement reduplication and size reduction or (2) size reduction alone. We asked whether these cross-linguistic similarities arise from cognitive biases in how humans construe objects and actions. We tested nonsigners' sensitivity to differences in noun-verb pairs in American Sign Language (ASL) by asking MTurk workers to match images of actions and objects to videos of ASL noun-verb pairs. Experiment 1a's match-to-sample paradigm revealed that nonsigners interpreted all signs, regardless of lexical class, as actions. The remaining experiments used a forced-matching procedure to avoid this bias. Counter our predictions, nonsigners associated reduplicated movement with actions not objects (inversing the sign language pattern) and exhibited a minimal bias to associate large movements with actions (as found in sign languages). Whether signs had pantomimic iconicity did not alter nonsigners' judgments. We speculate that the morphophonological distinctions in noun-verb pairs observed in sign languages did not emerge as a result of cognitive biases, but rather as a result of the linguistic pressures of a growing lexicon and the use of space for verbal morphology. Such pressures may override an initial bias to map reduplicated movement to actions, but nevertheless reflect new iconic mappings shaped by linguistic and cognitive experiences.

The influence of cognitive bias on crisis decision-making: Experimental evidence on the comparison of bias effects between crisis decision-maker groups

A crisis requires the affected population, governments or non-profit organizations, as well as crisis experts, to make urgent and sometimes life-critical decisions. With the urgency and uncertainty they create, crises are particularly amenable to inducing cognitive biases that influence decision-making. However, there is limited empirical evidence regarding the impact of cognitive biases on estimation, judgment, and decision-making tasks in crises. Possible biases occurring in crises are: (1) to be influenced by how information is framed (i.e., framing effect), (2) to overly rely on information that confirms rather than opposes preliminary assumptions (i.e., confirmation bias), (3) to rely heavily on a skewed informational cue when making estimations (i.e., anchoring bias), and (4) to see the own decision-making as less biased than decision-making of others (i.e., bias blind spot). We investigate these four cognitive biases using three online survey experiments targeting crisis-affected people of the general public (n = 460, mTurk workers), governmental and non-profit workers (n = 50, mTurk workers), and crisis experts (n = 21, purposefully sampled). Our findings show that crisis experts are the least biased group but are still significantly affected by anchoring, framing, and bias blind spot. Crisis-affected people from the general public showed the strongest susceptibility to all four biases studied. The findings have implications for future research on crisis information systems (IS) design. As crisis response is increasingly facilitated through IS, we propose debiasing functions that account for biased user behavior in crises.

Sadder ≠ Wiser: Depressive Realism Is Not Robust to Replication

The theory of depressive realism holds that depressed individuals are less prone to optimistic bias, and are thus more realistic, in assessing their control or performance. Since the theory was proposed 40 years ago, many innovations have been validated for testing cognitive accuracy, including improved measures of bias in perceived control and performance. We incorporate several of those innovations in a well-powered, pre-registered study designed to identify depressive realism. Amazon MTurk workers (N = 246) and undergraduate students (N = 134) completed a classic contingency task, an overconfidence task, and measures of mental health constructs, including depression and anxiety. We measured perceived control throughout the contingency task, allowing us to compare control estimates at the trial-level to estimates assessed at task conclusion. We found no evidence that depressive symptoms relate to illusory control or to overconfidence. Our results suggest that despite its popular acceptance, depressive realism is not replicable.

Symptoms of Attention Deficit/Hyperactivity Disorder Are Associated with Sub-Optimal and Inconsistent Temporal Decision Making.

The link between Attention-Deficit/Hyperactivity Disorder (ADHD) and steeper delay discounting has been established and incorporated into theories of ADHD. This study examines a novel interpretation according to which ADHD is linked to sub-optimal temporal decision-making and suggests inconsistency as a potential underlying mechanism. In two experiments, MTurk workers completed a self-report questionnaire on symptoms of ADHD and a temporal decision making task consisting of choices between smaller–immediate and larger–delayed options. The delayed option was better in some items, whereas the immediate option was better in others. The rate of choices of the delayed option and the consistency of choices were measured. The results of both studies show that high symptoms of ADHD were linked to fewer choices of the delayed option when it was better, but also to more choices of the delayed option when it was not better. In addition, ADHD was linked to higher inconsistency in both conditions. The findings suggest that ADHD is linked to sub-optimal temporal decision-making rather than steeper delay discounting, and provide further support to the phenomenon of inconsistency in ADHD.

Big Five Personality Traits and Moral-Dilemma Judgments: Two Preregistered Studies using the CNI Model

A growing line of research suggests that disagreement in moral-dilemma judgments may be rooted in basic personality traits. Using the CNI model, two preregistered studies (N = 490) aimed to replicate findings of prior exploratory research on relations between the Big Five and sensitivity to consequences (C), sensitivity to moral norms (N), and general preference for inaction versus action (I) in responses to moral dilemmas. While only one of six previously obtained relations replicated in an undergraduate student sample (Study 1), all six relations replicated in an MTurk worker sample (Study 2). The results highlight the importance of examining the generalizability of relations between basic personality traits and moral judgments in diverse samples for the reproducibility of psychological findings.

Can You Help Me Stay Fit?

The main purpose of this study is to reveal the impact of consumer satisfaction on continuance intention to use wearable fitness devices. Building upon the IS Continuance Intention Model, the authors explored the effects of confirmation of ease of use, confirmation of perceived usefulness, positive and negative feelings, and perceived control on consumer satisfaction. The effects of health motivation and social influence on continuance intention were examined alongside satisfaction. Our model consists of twelve constructs and eleven hypotheses. An online survey was conducted among 216 Amazon M-Turk workers to collect data. The measurement model was first tested and validated. Next, structural equation modeling was used to test the hypotheses in the research model. Nine out of eleven hypotheses were supported. The model explains 50.1% of variances in continuance intention, and 63.9% of variance in consumer satisfaction is explained by the aforementioned factors. Both theoretical contributions and practical implications are discussed in the context of wearable technology.

Appraisals Associated with Interpersonal Negative Emotions: What Distinguishes Anger, Contempt, Dislike, and Hatred?

To aid in understanding the determinants of negative interpersonal and intergroup behaviours, this research tested theories specifying which appraisals of events would be associated with distinct negative emotions felt towards other individuals. To test hypotheses, we analysed survey responses from 128 MTurk workers and undergraduates in the USA who wrote about current and prior experiences of either anger, contempt, dislike, or hatred, and rated scales measuring hypothesised appraisals and emotional responses. As predicted, anger was associated with perceiving another person as blocking one’s goals, whereas contempt was associated with perceiving another person as beneath one’s standards; and anger, contempt, dislike, and hatred were each associated with perceiving events as motive-inconsistent and caused by another person. However, only one item measuring prospective control fit the predicted pattern of anger and contempt involving higher control potential than dislike and hatred. These results replicate and extend previous findings on appraisal-emotion relationships in India and the United States. Similarities and differences across cultures in appraisal-emotion relationships are discussed and applied to intergroup relations in developing societies.

Eating Disorder Examination-Questionnaire and Clinical Impairment Assessment norms for intersectional identities using an MTurk sample.

There are limited data to guide the interpretation of scores on measures of eating-disorder psychopathology among underrepresented individuals. We aimed to provide norms for the Eating Disorder Examination-Questionnaire (EDE-Q) and Clinical Impairment Assessment (CIA) across racial/ethnic, gender, and sexual identities, and sexual orientationsand their intersections by recruiting a diverse sample of Amazon MTurk workers (MTurkers; N =1782). We created a comprehensive, quantitative assessment of racial/ethnic identification, gender identification, sex assigned at birth, current sexual identification, and sexual orientation called the Demographic Assessment of Racial, Sexual, and Gender Identities (DARSGI). We calculated normative data for each demographic category response option. Our sample was comprised of 68% underrepresented racial/ethnic identities, 42% underrepresented gender identities, 13% underrepresented sexes, and 49% underrepresented sexual orientations. We reported means and standard deviations for each demographic category response option and, where possible, mean estimates by percentile across intersectional groups. EDE-Q Global Score for a subset of identities and intersections in the current study were higher than previously reported norms for those identities/intersections. This is the most thorough reporting of norms for the EDE-Q and CIA among racial/ethnic, sexual, and gender identities, and sexual orientations and the first reporting on multiple intersections, filling some of the gaps for commonly used measures of eating-disorder psychopathology. These norms may be used to contextualize eating-disorder psychopathology reported by underrepresented individuals. The data from the current study may help inform research on the prevention and treatment of eating-disorder psychopathology in underrepresented groups. We provide the most thorough reporting on racial/ethnic, sexual, and gender identities, and sexual orientations for the Eating Disorder Examination - Questionnaire and Clinical Impairment Assessment, and the first reporting on intersections, which fills some of the gaps for commonly used measures of eating-disorder psychopathology. These norms help inform research on the prevention and treatment of eating-disorder psychopathology in underrepresented groups.

Frustration and ennui among Amazon MTurk workers.

Academics are increasingly turning to crowdsourcing platforms to recruit research participants. Their endeavors have benefited from a proliferation of studies attesting to the quality of crowdsourced data or offering guidance on managing specific challenges associated with doing crowdsourced research. Thus far, however, relatively little is known about what it is like to be a participant in crowdsourced research. Our analysis of almost 1400 free-text responses provides insight into the frustrations encountered by workers on one widely used crowdsourcing site: Amazon’s MTurk. Some of these frustrations stem from inherent limitations of the MTurk platform and cannot easily be addressed by researchers. Many others, however, concern factors that are directly controllable by researchers and that may also be relevant for researchers using other crowdsourcing platforms such as Prolific or CrowdFlower. Based on participants’ accounts of their experiences as crowdsource workers, we offer recommendations researchers might consider as they seek to design online studies that demonstrate consideration for respondents and respect for their time, effort, and dignity.

A Psychometric Pilot Study Examining the Functions of Suicidal Communications Using IRT and Factor Analysis.

Background: Suicide prevention, an important public health issue, relies on suicidal communications to identify and intervene with those at risk. Scant research tests explicit theories of suicidal communication impeding applications to prevention science. The current study pilots a new measure assessing the functions of suicidal communications using factor analysis and item response theory. Methods: MTurk workers (n = 898) completed an anonymous survey. The original scale included 35 items refined using exploratory and confirmatory factor analysis, bifactor modeling, and item response theory. Results: The initial EFA identified a two-correlated-factor solution. The two-correlated-factor and unidimensional models yielded a poor fit. A bifactor model yielded a borderline to acceptable fit. The final four items were identified using a bifactor model and item response theory graded response models capturing ambivalence resolution defined as behaviors aimed to aid in suicide decision making. The final model yielded an excellent fit: 𝝌2(2) = 1.81, CFI (1.00), TLI (1.00), RMSEA (0.00), and SRMR (0.01). Conclusions: There may be one function of suicidal communications. Disclosure may elicit connection and reasons for living that serve as barriers to suicide and resolve ambivalence. Key limitations include convenience sampling and limited validity measures. Future research should partner with participants to improve scale and theory development efforts.

Are Biasing Factors Idiosyncratic to Measures? A Comparison of Interpersonal Conflict, Organizational Constraints, and Workload.

Widespread concern has been raised about the possibility of potential biasing factors influencing the measurement of organizational variables and distorting inferences and conclusions reached about them. Recent research calls for a measure-centric approach in which every measure is independently evaluated to assess what factor(s) may uniquely bias it. This paper examines three popular stressor measures from this perspective. Across three studies, we examine factors that may bias three popular measures of job stressors: The Interpersonal Conflict at Work Scale (ICAWS), the Organizational Constraints Scale (OCS), and the Quantitative Workload Inventory (QWI). The first study used a two-wave design to survey 276 MTurk workers to assess the three stressor scales, four strains, and five measures of potential bias sources: hostile attribution bias, negative affectivity, mood, neutral objects satisfaction, and social desirability. The second study used an experimental design with 439 MTurk workers who were randomly assigned to a positive, negative, or no mood induction condition to assess effects on means of the three stressor measures and their correlations with strains. The third study surveyed 161 employee-supervisor dyads to explore the convergence of results involving the three stressor measures across sources. Based on several forms of evidence we conclude that potential biasing factors affect the three stressor measures differently, supporting the merits of a measure centric approach, even among measures in the same domain.

Differentiations in Interpersonal Functioning Across Narcissism Dimensions.

Narcissism can be conceived hierarchically at three levels: as a global construct (Level 1), as two dimensions (Level 2; grandiosity and vulnerability), and as a trifurcated model with three underlying dimensions: interpersonal antagonism, narcissistic neuroticism, and agentic extraversion (Level 3). The aim of the study was to examine how narcissism dimensions across the three levels differ in their associations with various forms of interpersonal functioning. The authors assessed multiple domains of interpersonal functioning using data collected from 447 MTurk workers, 606 students, and 365 informants. Each narcissism dimension showed unique interpersonal profiles. The profile of interpersonal antagonism largely resembles grandiose and total narcissism in its interpersonal characteristics, narcissistic neuroticism largely resembles vulnerable narcissism, and agentic extraversion does not differ much from the traditional conceptualization of extraversion in its interpersonal qualities (e.g., high communion). Future studies may benefit from studying narcissism and how it relates to other psychological constructs using the trifurcated model.