Large-scale Online Experiment Research Articles

Memory for rewards guides retrieval

Rewards paid out for successful retrieval motivate the formation of long-term memory. However, it has been argued that the Motivated Learning Task does not measure reward effects on memory strength but decision-making during retrieval. We report three large-scale online experiments in healthy participants (N = 200, N = 205, N = 187) that inform this debate. In experiment 1, we found that explicit stimulus-reward associations formed during encoding influence response strategies at retrieval. In experiment 2, reward affected memory strength and decision-making strategies. In experiment 3, reward affected decision-making strategies only. These data support a theoretical framework that assumes that promised rewards not only increase memory strength, but additionally lead to the formation of stimulus-reward associations that influence decisions at retrieval.

Chilling results: how explicit warm glow appeals fail to boost pro-environmental behaviour

AbstractWe conducted a large-scale online experiment to examine whether climate change messaging can induce emotions and motivate pro-environmental action. We study how exposure to explicit positive (‘warm glow’) and negative (‘cold prickle’) emotional appeals as well as a traditional social norm communication affects pro-environmental action. We find that a simple call to take action to mitigate climate change is at least as affective as social norm message framing and emotional appeals. Our results highlight the difficulty of designing messaging interventions that effectively harness emotional incentives to promote pro-environmental action. Messages that explicitly emphasise the personal emotional benefits of contributing to environmental causes or the adverse emotional effects of not doing so seem to fall short of motivating pro-environmental effort. Our findings underscore the need for caution when incorporating emotive appeals into policy interventions.

Does improved upward social mobility foster frustration and conflict? A large-scale online experiment testing Boudon’s model

The rise of populism has reignited scholarly interest in the paradox of societal advancement leading to frustration and social tension. Globalization and digitalization have increased social opportunities for parts of the population, but a substantial portion of society feels disadvantaged, resulting in discontent. This study, rooted in Boudon’s model of relative deprivation, examines the mechanisms that fuel this frustration. We conducted an online experiment involving 2114 US-based MTurk participants, in which we manipulated the availability of status positions to create varying degrees of upward social mobility. We also varied group sizes to ensure robustness. We assessed relative deprivation with structural, subjective, and behavioral measures. For example, frustration was measured using the “joy-of-destruction game,” in which subjects had to make the costly decision to destroy part of another player’s winnings. Contrary to the model’s prediction, we found that the proportion of individuals who were worse off, the losers, decreased consistently as mobility increased. This outcome can be attributed to overentry in conditions of low mobility and underentry in conditions of intermediate or high mobility. The losers displayed increased frustration and hostility towards noncompetitors and winners. Intriguingly, winners also exhibited heightened hostility. However, at the aggregate level, hostile behavior did not surge as conditions improved. In our exploratory analyses at the individual level, we identified several distinct patterns. Risk-tolerant individuals and women were more likely to enter competition. Conversely, those with advanced education levels showed a decreased inclination to competitiveness. Risk-tolerant individuals reported greater feelings of frustration and displayed increased hostility. This effect was also observed particularly among politically right-leaning individuals.

Stress, intertemporal choice, and mitigation behavior during the COVID-19 pandemic.

Delayed gratification is an important focus of research, given its potential relationship to forms of behavior, such as savings, susceptibility to addiction, and pro-social behaviors. The COVID-19 pandemic may be one of the most consequential recent examples of this phenomenon, with people's willingness to delay gratification affecting their willingness to socially distance themselves. COVID-19 also provides a naturalistic context by which to evaluate the ecological validity of delayed gratification. This article outlines four large-scale online experiments (total N = 12, 906) where we ask participants to perform Money Earlier or Later (MEL) decisions (e.g., $5 today vs. $10 tomorrow) and to also report stress measures and pandemic mitigation behaviors. We found that stress increases impulsivity and that less stressed and more patient individuals socially distanced more throughout the pandemic. These results help resolve longstanding theoretical debates in the MEL literature as well as provide policymakers with scientific evidence that can help inform response strategies in the future. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Profiles in the Teaching of Experimental Design and Analysis

The design and analysis of experiments (DOE) has historically been an important part of an education in statistics, and with the increasing complexity of modern production processes and the advent of large-scale online experiments, it continues to be highly relevant. In this article, we provide an extensive review of the literature on DOE pedagogy, and provide five perspectives on the subject: one from each of the authors as well as a composite profile derived from a survey of DOE instructors. Our work provides a snapshot of current DOE pedagogy that showcases both the similarities and variety in how the subject is taught, as well as a look ahead at how its instruction may evolve. Supplementary materials for this article are available online.

Scheduling on a budget: Avoiding stale recommendations with timely updates

Recommendation systems usually create static models from historical data. Due to concept drift and changes in the environment, such models are doomed to become stale, which causes their performance to degrade. In live production environments, models are therefore typically retrained at fixed time-intervals. Of course, every retraining comes at a significant computational cost, making very frequent model updates unrealistic in practice. In some cases, the cost is worth it, but in other cases an update could be redundant and the cost an unnecessary loss. The research question then consists of finding an acceptable update schedule for your recommendation system, given a limited budget. This work provides a pragmatic analysis of model staleness for a variety of collaborative filtering algorithms in news and retail domains, where concept drift is a known impediment. We highlight that the rate at which models become stale is highly dependent on the environment they perform in and that this property can be derived from data. These findings are corroborated by empirical observations from four large-scale online experiments. Instead of retraining at regular intervals, we propose an adaptive scheduling method that aims to maximise the accuracy of the recommendations within a fixed resource budget. Offline experiments show that our proposed approach improves recommendation performance while keeping the cost constant. Our findings can guide practitioners to spend their available resources more efficiently.

Effects of Soundscape Complexity on Urban Noise Annoyance Ratings: A Large-Scale Online Listening Experiment.

Noise annoyance has been often reported as one of the main adverse effects of noise exposure on human health, and there is consensus that it relates to several factors going beyond the mere energy content of the signal. Research has historically focused on a limited set of sound sources (e.g., transport and industrial noise); only more recently is attention being given to more holistic aspects of urban acoustic environments and the role they play in the noise annoyance perceptual construct. This is the main approach promoted in soundscape studies, looking at both wanted and unwanted sounds. In this study, three specific aspects were investigated, namely: (1) the effect of different sound sources combinations, (2) the number of sound sources present in the soundscape, and (3) the presence of individual sound source, on noise annoyance perception. For this purpose, a large-scale online experiment was carried out with 1.2k+ participants, using 2.8k+ audio recordings of complex urban acoustic environments to investigate how they would influence the perceived noise annoyance. Results showed that: (1) the combinations of different sound sources were not important, compared, instead, to the number of sound sources identified in the soundscape recording (regardless of sound sources type); (2) the annoyance ratings expressed a minimum when any two clearly distinguishable sound sources were present in a given urban soundscape; and (3) the presence (either in isolation or combination) of traffic-related sound sources increases noise annoyance, while the presence (either in isolation or combination) of nature-related sound sources decreases noise annoyance.

Efficient search termination without task experience.

As a general rule, if it is easy to detect a target in a visual scene, it is also easy to detect its absence. To account for this, models of visual search explain search termination as resulting either from counterfactual reasoning over second-order representations of search efficiency, automatic extraction of ensemble statistics of a display, or heuristic adjustment of a search termination strategy based on previous trials. Traditional few-subjects/many-trials lab-based experiments render it impossible to disentangle the unique contribution of these different processes to absence pop-out - the immediate recognition that a feature is missing from a display. In 2 preregistered large-scale online experiments (N1 = 1187; N2 = 887) we show that search termination times are already aligned with target identification times in the very first trials of the experiment before any experience with target presence. Exploratory analysis reveals that explicit metacognitive knowledge about search efficiency is not necessary for efficient search termination. We conclude that for basic stimulus properties, efficient inference about absence is independent of task experience and of explicit metacognitive knowledge about visual search. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Accelerating climate protection by behavioural insights: the Planetary Health Action Survey (PACE)

BackgroundThe Planetary Health Action Survey investigates the gap between scientific consensus and collective action to protect our planet's and our own health. Following a theory driven and theory testing approach, we aim to better understand social cognitive processes leading to citizens’ readiness to act against climate change. Our findings could contribute to designing and communicating climate protection policies so that they are supported by the public. To achieve this, planetary health behaviour needs to be better understood. MethodsIn a serial cross-sectional design and accompanying large-scale online experiments, we examine psychological factors that affect health and climate protection-related attitudes and behaviours. Our first survey took place in July, 2021 (n=1521; a German quota sample representing the distribution of age, gender, and federal state of the adult general population). Currently, the survey runs every 4 weeks (around 1000 participants). Along with demographic variables, we assess different paths to understand people's readiness to act against climate change. We assess several determinants of readiness to act, including risk perception, trust in institutions, social norms, perceived response efficacy, self-efficacy, knowledge, and prosociality. Readiness to act is assessed via several indicators that go beyond individual behaviours and include acceptance of climate protection measures, political participation, psychological reactance towards measures, and compliance with measures. FindingsRespondents judged weather events as likely to seriously impact health. They judged health consequences, such as low food quality or poor mental health, as less relevant. Individuals who think that they will be exposed to health risks consider them more serious and are more likely to agree with climate protection measures. From 80 potential measures surveyed (as proposed by the German Climate Citizen Assembly [Bürgerrat Klima Deutschland]), 17 (21%) received more than 75% approval. High approval rates were found for cheaper local transport and reducing food waste, whereas low approval rates were found for regulation of renewable energy or reducing livestock. 167 (11%) of 1521 individuals showed low or no support for all measures. InterpretationWe found that most participants were pro climate protection. Risk perception differed between individuals. The links between climate change and health risks that go beyond heat and extreme weather should be communicated more effectively. Frameworks should be created that facilitate climate-friendly collective and individual action. FundingKlaus Tschira Foundation, German Federal Ministry of Health, Robert Koch Institute, and Federal Center of Health Education.

Energy efficiency of consideration sets and choices: the impact of label format

PurposeThis research examines how the design of the online energy label can be improved to stimulate consumer choice of energy-efficient household products in Web stores. Based on general evaluability theory, the authors propose new label formats that aim to improve the evaluability of the label information for consumers and test their influence during two distinct stages in the online decision-making process: consideration set formation and final choice.Design/methodology/approachTwo large-scale controlled online experiments are conducted with over 10,000 consumers in 10 European countries. The experiments test label alternatives in simulated online store environments, mimicking the two distinct decision stages, for four product categories to enhance generalizability. The data are analyzed using random-intercept linear and logistic regression models to account for their multi-level structure.FindingsThe results show that the impact of the online energy label on consumers’ online decision-making depends on both the label format and the decision stage (consideration vs choice), but in a different way than expected. The findings reveal that the current online energy label is significantly outperformed by a label that provides reference information by incorporating the scale range. This alternative label is particularly effective in the consideration set formation stage, and among consumers who consider energy efficiency a relatively unimportant choice criterion.Research limitations/implicationsOnline energy labels encourage consumers to consider and choose more energy-efficient products, especially if scale range information is included. The present results stress the importance of presenting this information early on in the online decision process. They also show that, particularly at this early stage and particularly for consumers who find energy efficiency a relatively unimportant choice criterion, label format matters.Practical implicationsThe present findings provide important input for policymakers in the context of the ongoing revision of the EU energy label. They also help online retailers make decisions about when and how to present product information on their websites.Originality/valueThis study contributes to the literature on product labelling by examining the effects of relatively unexplored types of reference information in two distinct stages of the consumer decision-making process. To the best of the authors’ knowledge, this study is the first to test the effectiveness of the online energy label.

Unequal opportunities, social groups, and redistribution: Evidence from Germany

In this paper we investigate the generalizability of the role of unequal opportunities and social group membership in redistributive preferences and examine the interaction between these two dimensions. We present results from a large-scale online experiment with more than 4,000 participants from Germany. The experiment consists of a real-effort task and a subsequent dictator game with native Germans and immigrants to Germany. We find that dictator transfers to the own group by native Germans and immigrants are higher under unequal opportunities than under equal opportunities. While we confirm the main findings reported in previous literature regarding the role of inequality of opportunity in redistribution for native Germans and immigrants, we find distinctively different patterns between both groups concerning the influence of social group membership and its interaction with unequal opportunities on redistribution. In particular, contrary to natives, immigrant dictators transfer more to in-group than to out-group receivers under unequal opportunities and do not compensate for unequal opportunities of out-group members. We conclude that in order to increase the understanding of patterns reported in the literature, it is crucial to also investigate the generalizability of findings to individuals from the general population and to explicitly cover participants such as immigrants who represent important parts of our society.

Debiasing through experience sampling: The case of myopic loss aversion

We introduce a training intervention based on a novel tool to mitigate behavior consistent with myopic loss aversion (MLA). We present the results of a large-scale online experiment with 894 student participants. The study featured a two-step debiasing training intervention based on experience sampling and a subsequent elicitation of MLA. We found that participants in the baseline treatment exhibit behavior consistent with MLA, which was not the case for decision makers who underwent the debiasing training intervention. Nonetheless, we found no statistically significant difference-in-difference effect of the training intervention on the magnitude of MLA. However, when we focused on the more attentive participants, the magnitude of the difference-in-difference effect of the training intervention increased strongly and became statistically significant when controlling for age, gender, education, field of study, investment experience, and risk preferences.

Do people demand fact-checked news? Evidence from U.S. Democrats

In a large-scale online experiment with U.S. Democrats, we examine how the demand for a newsletter about an economic relief plan changes when the newsletter content is fact-checked. We first document an overall muted demand for fact-checking when the newsletter features stories from an ideologically aligned source, even though fact-checking increases the perceived accuracy of the newsletter. The average impact of fact-checking masks substantial heterogeneity by ideology: fact-checking reduces demand among Democrats with strong ideological views and increases demand among ideologically moderate Democrats. Furthermore, fact-checking increases demand among all Democrats when the newsletter features stories from an ideologically non-aligned source.

People perceive themselves to adhere more strictly to COVID-19 guidelines than others

ABSTRACT People have a fair idea of how they are supposed to behave to slow down the spread of COVID-19. But what about people’s perception of their own compared to others’ adherence to the guidelines? Building on prior research on self-enhancement biases, we predicted that people perceive themselves to adhere more strictly to the COVID-19 guidelines than others. To test this hypothesis, we conducted a large-scale online experiment (N = 1,102), using a sample from four countries (UK, US, Germany, Sweden). As predicted, people perceived themselves to adhere to the COVID-19 guidelines more strictly than both the average citizen of their country and their close friends. These findings were robust across countries. Furthermore, findings were not moderated by whether people first thought about themselves or about others. In conclusion, our study provides a robust demonstration of how a long-standing psychological effect perseveres, even during a once-in-a-lifetime health crisis.

Why do so many modern people hate insects? The urbanization–disgust hypothesis

While negative perceptions of insects and other terrestrial arthropods (hereafter referred to collectively as ‘insects’) are widespread around the world, the underlying causes of this phenomenon remain unknown. Negative attitudes towards insects manifest as the emotion disgust, which is regarded as a psychological adaptation to produce pathogen-avoidance behavior. Based on theories of evolutionary psychology (parasite avoidance theory of disgust and error management theory), we hypothesized that there are two pathways by which urbanization increases the intensity and breadth of feelings of disgust towards insects: (1) urbanization increases the extent to which people see insects indoors, and insects that are seen more often indoors induce stronger feelings of disgust than is induced by insects seen outdoors; and (2) urbanization reduces people's natural history knowledge about insects, and decreased knowledge results in a broader range of insects eliciting feelings of disgust. To test our hypothesis, we conducted a large-scale online questionnaire survey and questionnaire experiment ( n = 13,000) across Japan to quantify the association between the level of urbanization, places of insect sightings (outdoor/indoor), knowledge about insects, and respondents' feelings of disgust. Our results supported both hypothetical pathway and suggested that psychological mechanisms shaped by past evolutionary pressure to avoid pathogen infection may underlie the current prevalence of insect disgust, and this is being reinforced by urbanization. Because negative perceptions of nature can reduce motivation for its conservation, negative attitudes towards insects exacerbated by ongoing urbanization is a potential global risk for biodiversity. • While negative perceptions of insects are widespread around the world, the causes of this phenomenon remain unknown. • Based on evolutionary psychology, we hypothesized that urbanization is the key driver of the prevalence of insect disgust. • Large-scale online experiment and survey ( n = 13,000) supported this hypothesis. • Urbanization increases the seeing of insects indoors and insects seen indoors induce stronger disgust than outdoor insects. • Urbanization reduces insect knowledge, and this decreased knowledge results in a wider range of insects that elicit disgust.

The Usefulness of Financial Accounting Information: Evidence from the Field

ABSTRACTWe examine how investment professionals assess the usefulness of financial accounting information depending on their information acquisition objectives and preparers' earnings management incentives. We conduct a survey experiment based on face-to-face interviews with investment professionals and document two main results. First, we find that, compared with investment professionals assigned a firm valuation objective, those assigned a managerial performance evaluation objective assess accounting information as significantly less useful. Second, we find no systematic evidence that preparers' earnings management incentives negatively affect investment professionals' assessments of accounting information usefulness. To elucidate this second finding, we conduct a large-scale follow-up online experiment. Our results continue to offer no support for the effect of earnings management incentives on investment professionals' assessments of accounting information usefulness, irrespective of preparers' corporate governance quality. Instead, we find that poor corporate governance, by itself, reduces the usefulness of accounting information to investment professionals.JEL Classifications: G15; G18; G38; M41; M48.

Random Utility and Limited Consideration

The random utility model (RUM, McFadden and Richter, 1990) has been the standard tool to describe the behavior of a population of decision makers. RUM assumes that decision makers behave as if they maximize a rational preference over a choice set. This assumption may fail when consideration of all alternatives is costly. We provide a theoretical and statistical framework that unifies well-known models of random (limited) consideration and generalizes them to allow for preference heterogeneity. We apply this methodology in a novel stochastic choice dataset that we collected in a large-scale online experiment. Our dataset is unique since it exhibits both choice set and (attention) frame variation. We run a statistical survival race between competing models of random consideration and RUM. We find that RUM cannot explain the population behavior. In contrast, we cannot reject the hypothesis that decision makers behave according to the logit attention model (Brade and Rehbeck, 2016).

Do People Demand Fact-Checked News? Evidence from U.S. Democrats

In a large-scale online experiment with U.S. Democrats, we examine how the demand for a newsletter about an economic relief plan changes when the newsletter content is fact-checked. We first document an overall muted demand for fact-checking when the newsletter features stories from an ideologically aligned source, even though fact-checking increases the perceived accuracy of the newsletter. The average impact of fact-checking masks substantial heterogeneity by ideology: fact-checking reduces demand among Democrats with strong ideological views and increases demand among ideologically moderate Democrats. Furthermore, fact-checking increases demand among all Democrats when the newsletter features stories from an ideologically non-aligned source.

Object-selective cortex shows distinct representational formats along the posterior-to-anterior axis: evidence from brain-behavior correlations

Object information is transformed along the ventral visual stream from early formats into increasingly invariant representations. To gain insight into the nature of these representations, we measured different facets of similarity with three behavioral tasks and related them to neural responses. First, we measured similarity implicitly using a visual search task, measuring the time it took participants to find one target amongst distractors in a large-scale online experiment (72 images of inanimate objects, 2,556 pairs, n=1,272). Next, we measured similarity with an explicit shape-sorting task, asking participants to arrange items in a circular arena based on their shape similarity (n=25). Then, we measured similarity with the same task, but without any guidance on what object properties to focus on while placing more similar objects nearby (n=26). Finally, brain responses to each image were obtained using functional magnetic resonance imaging (n=10). Our results show that (i) there are clear differences in the measured similarity space of objects across tasks, (ii) the implicit visual search similarity was most strongly correlated with the posterior half of object-selective cortex, (iii) the explicit shape-sorting similarity was most strongly correlated to the anterior half of object-selective cortex, and (iv) the similarity measured with the unguided sorting task was not well-correlated with any responses along occipitotemporal cortex. These observations were confirmed quantitatively with linear mixed-effect modeling: the brain similarity structure along the hierarchy was better explained when including the interaction between behavioral task and location along the hierarchy, compared to a model without the interaction: 𝜒2(34)=178.73, p<0.001). Broadly, these results reveal a clear dissociation in the way posterior and anterior occipitotemporal cortex represents the similarity of objects. We hypothesize that this relatively sharp transition along the ventral stream hierarchy may reflect a passage from more pictorial to more structural representations of inanimate object information.

"At the End of the Day Facebook Does What ItWants"

Interest has grown in designing algorithmic decision making systems for contestability. In this work, we study how users experience contesting unfavorable social media content moderation decisions. A large-scale online experiment tests whether different forms of appeals can improve users' experiences of automated decision making. We study the impact on users' perceptions of the Fairness, Accountability, and Trustworthiness of algorithmic decisions, as well as their feelings of Control (FACT). Surprisingly, we find that none of the appeal designs improve FACT perceptions compared to a no appeal baseline. We qualitatively analyze how users write appeals, and find that they contest the decision itself, but also more fundamental issues like the goal of moderating content, the idea of automation, and the inconsistency of the system as a whole. We conclude with suggestions for -- as well as a discussion of the challenges of -- designing for contestability.