Experimenter Demand Effects Research Articles

Reactivity in the human sciences

The reactions that science triggers on the people it studies, describes, or theorises about, can affect the science itself and its claims to knowledge. This phenomenon, which we call reactivity, has been discussed in many different areas of the social sciences and the philosophy of science, falling under different rubrics such as the Hawthorne effect, self-fulfilling prophecies, the looping effects of human kinds, the performativity of models, observer effects, experimenter effects and experimenter demand effects. In this paper we review state-of-the-art research that falls under the remit of the philosophy of reactivity by considering ontological, epistemic and moral issues that reactivity raises. Along the way, we devote special attention to articles belonging to this journal's Topical Collection entitled “Reactivity in the Human Sciences”.

Predicting compliance: Leveraging chat data for supervised classification in experimental research

Behavioral and experimental economics have conventionally employed text data to facilitate the interpretation of decision-making processes. This paper introduces a novel methodology, leveraging text data for predictive analytics rather than mere explanation. We detail a supervised classification framework that interprets patterns in chat text to estimate the likelihood of associated numerical outcomes. Despite the unique advantages of experimental data in correlating textual and numerical information for predictive modeling, challenges such as limited sample sizes and potential data skewness persist. To address these, we propose a comprehensive methodological framework aimed at optimizing predictive modeling configurations, particularly in small experimental behavioral research datasets. We also present behavioral experimental data from a preregistered tax evasion game (n=324), demonstrating that chat behavior is not influenced by experimenter demand effects. This establishes chat text as an unbiased variable, enhancing its validity for prediction. Our findings further indicate that beliefs about others’ dishonesty, lying attitudes, and risk preferences significantly impact compliance decisions.

The relationship between monetary incentives, social status, and physical activity

We conduct a field experiment to better understand the role of social status with and without monetary incentives as motivation to increase physical activity. We find that social status alone does not induce a change in physical activity. When social status is combined with monetary incentives, however, we find a change in the number of daily steps. This change is heterogeneous. Individuals with low physical activity increase their number of steps by 12%, while those with high physical activity decrease the number of steps by 25%. An incentives treatment with exogenous social status – uncorrelated with physical activity – provides robustness to our findings and, together with the control condition, rules out potential experimenter demand effects and other factors driving the results. Our results call for a cautionary approach for analyzing the role of social status, in many cases unobserved, for physical activity intervention programs.

Designing Information Provision Experiments

Information provision experiments allow researchers to test economic theories and answer policy-relevant questions by varying the information set available to respondents. We survey the emerging literature using information provision experiments in economics and discuss applications in macroeconomics, finance, political economy, public economics, labor economics, and health economics. We also discuss design considerations and provide best-practice recommendations on how to (i) measure beliefs; (ii) design the information intervention; (iii) measure belief updating; (iv) deal with potential confounds, such as experimenter demand effects; and (v) recruit respondents using online panels. We finally discuss typical effect sizes and provide sample size recommendations.(JEL C90, D83, D91)

Social Reference Points and Risk Taking

Social reference points have been identified to be important determinants of individuals’ welfare. We investigate the consequences of social reference points for risk taking in a laboratory experiment. In the main treatments, risk-taking subjects observe the predetermined earnings of peer subjects when making a risky choice. We exogenously manipulate peers’ earnings and find a significant treatment effect: decision makers make less risk-averse choices in the case of larger peers’ earnings. The treatment effect is consistent with an application of prospect theory to social reference points and cannot be explained by reference points based on counterfactual information, anchoring, and experimenter demand effects. In additional analyses, we show that diminishing sensitivity seems to play an important role in subjects’ risky choices. We explore also whether inequity aversion and expectations-based reference points can account for our findings and conclude that they do not provide plausible alternative explanations for them. This paper was accepted by Yan Chen, behavioral economics and decision analysis. Funding: This research was financially supported by the Bonn Graduate School of Economics and the Center for Economics and Neuroscience in Bonn. Supplemental Material: The online appendix and data are available at https://doi.org/10.1287/mnsc.2023.4698 .

The net effect of advice on strategy-proof mechanisms: an experiment for the Vickrey auction

We conduct laboratory experiments for the multi-unit Vickrey auction with and without advice to subjects on strategy-proofness. The rate of truth-telling among the subjects without advice stays at 20%, whereas the rate increases to 47% among those who have received advice. By conducting similar experiments for the pay-your-bid auction, which is not strategy-proof, we confirm that the increase in truth-telling is due significantly to the net advice effect (i.e., the effect beyond the so-called experimenter demand effect). Moreover, we find that providing advice improves efficiency in the Vickrey auction, particularly in the early periods, when the subjects are less experienced. In general, subjects tend to overbid in Vickrey auction experiments. Our results indicate the possibility that providing simple advice decreases such overbidding by promoting a better understanding of the strategy-proofness of the Vickrey auction. Strategy-proof mechanisms are sometimes criticized because players often fail to recognize the benefit of telling the truth. However, our observations show that introducing advice on the property of strategy-proofness helps them behave “correctly.”

Inducing Perspective-Taking for Prosocial Behaviour in Natural Resource Management

Natural resource management often involves social dilemmas. Institutional and behavioural economics have shown that other-regarding preferences and pro-social behaviour can help overcome such dilemmas. Interventions that induce resource users to consider a perspective broader than their own may then be useful to promote and strengthen pro-social behaviour. Such interventions are often applied in participatory resource management approaches. To the best of our knowledge, nonetheless, no previous study has systematically assessed the effect of induced perspective-taking on resource users’ prosocial behaviour in a controlled manner. In this study, we do so in the context of watershed management. We conducted a lab-in-the-field experiment with downstream farmers in a Peruvian watershed. In the experiment, farmers were induced to imagine the perspective of upstream farmers before deciding on a donation that can help these upstream farmers improve their wellbeing without compromising the water supply downstream. We find that induced perspective-taking increases prosocial behaviour. This effect cannot be explained by the additional information on the social and ecological characteristics of the watershed received during the perspective-taking experience, nor by an ‘experimenter demand effect’. Rather the effect of the perspective-taking intervention is likely to work via an activation or strengthening of other-regarding preferences. Our results contribute to the study of pro-social behaviour and the ways it could be induced by interventions targeting other-regarding preferences.

Designing Information Provision Experiments

We review methodological questions relevant for the design of information provision experiments. We first provide a literature review of major areas in which information provision experiments are applied. We then outline key measurement challenges and design recommendations that may be of help for practitioners planning to conduct an information experiment. We discuss the measurement of subjective beliefs, including the role of incentives and ways to reduce measurement error. We also discuss the design of the information intervention, as well as the measurement of belief updating. Moreover, we describe ways to mitigate potential experimenter demand effects and numerical anchoring arising from the information treatment. Finally, we discuss typical effect sizes in information experiments.

The Net Effect of Advice on Strategy-Proof Mechanisms: An Experiment for the Vickrey Auction

We conduct laboratory experiments for the multi-unit Vickrey auction with and without advice to subjects on strategy-proofness. The rate of truth-telling among the subjects without advice stays at 20%, whereas the rate increases to 47% among those who have received advice. By conducting similar experiments for the pay-your-bid auction, which is not strategy-proof, we confirm that the increase in truth-telling is due significantly to the net advice effect (i.e., the effect beyond the so-called experimenter demand effect). Moreover, we find that providing advice improves efficiency in the Vickrey auction, particularly in the early periods, when the subjects are less experienced. In general, subjects tend to overbid in Vickrey auction experiments. Our results indicate the possibility that providing simple advice decreases such overbidding by promoting a better understanding of the strategy-proofness of the Vickrey auction. Strategy-proof mechanisms are sometimes criticized because players often fail to recognize the benefit of telling the truth. However, our observations show that introducing advice on the property of strategy-proofness helps them behave “correctly.”

Designing Information Provision Experiments

We review methodological questions relevant for the design of information provision experiments. We first provide a literature review of major areas in which information provision experiments are applied. We then outline key measurement challenges and design recommendations that may be of help for practitioners planning to conduct an information experiment. We discuss the measurement of subjective beliefs, including the role of incentives and ways to reduce measurement error. We also discuss the design of the information intervention, as well as the measurement of belief updating. Moreover, we describe ways to mitigate potential experimenter demand effects and numerical anchoring arising from the information treatment. Finally, we discuss typical effect sizes in information experiments.

The Heaven Dictator Game: Costless taking or giving

We present experimental data from the Heaven-Dictator Game, a generalization of the Dictator Game that investigates the overstatement of inequality reduction in the motivation of social preferences. Two players start with equal endowments and the Heaven-Dictator player, without incurring any pecuniary cost or profit, chooses among increasing, decreasing or maintaining the earnings of the recipient player. Any choice except for the status quo generates unequal payoffs. The design avoids the experimenter demand effect of the standard “give only” version while simultaneously allowing participants to manifest antisocial preferences, inequity reduction or retaliation cannot be called for as motives. We find that the majority (75.4%) of subjects choose to increase their partners’ earnings; there is a non-negligible 24.6% of subjects that either choose the status quo (11.9%) or to decrease (12.7%) others’ earnings. Based on the psychological literature on music as a mood-inducing stimulus and on the effects of mood on helping behavior, we study the effect of exposure to different types of music on the Heaven-Dictator choices. Although at first sight observed distributional preferences are independent of the music condition, further analysis reveals that classical music seems to foster social welfare rather than inequality aversion.

A Strategy-Proof Mechanism Should Be Announced to Be Strategy-Proof: An Experiment for the Vickrey Auction

We conduct laboratory experiments for the multi-unit Vickrey auction with and without providing advice to subjects on strategy-proofness. Although the rate of truth-telling among the subjects stays at 20% without advice, the rate increases to 47% with advice. By conducting similar experiments for the pay-your-bid auction, which is not strategy-proof, we confirm that our results are not due to so-called experimenter demand effects. Moreover, advice improves efficiency in the Vickrey auction, particularly in early periods in which subjects are less experienced. It is well known that subjects tend to overbid in several Vickrey auction experiments. Our results indicate the possibility that simple advice decreases such overbidding by promoting better understanding of the strategy-proofness property in the Vickrey auction. Strategy-proof mechanisms are sometimes criticized because players often fail to find the benefit of truth-telling, but our observations show that introducing advice on the property of strategy-proofness helps them to behave “correctly.”

How natural field experiments have enhanced our understanding of unemployment.

Natural field experiments investigating key labour market phenomena such as unemployment have only been used since the early 2000s. This paper reviews the literature and draws three primary conclusions that deepen our understanding of unemployment. First, the inability to monitor workers perfectly in many occupations complicates the hiring decision in a way that contributes to unemployment. Second, the inability to determine a worker's attributes precisely at the time of hiring leads to discrimination on the basis of factors such as race, gender, age and ethnicity. This can lead to systematically high and persistent levels of unemployment for groups that face discrimination. Third, the importance of social and personal dynamics in the workplace can lead to short-term unemployment. Much of the knowledge necessary for these conclusions could only be obtained using natural field experiments due to their ability to combine randomized control with an absence of experimenter demand effects.

Demand Effects in Survey Experiments: An Empirical Assessment

Survey experiments are ubiquitous in social science. A frequent critique is that positive results in these studies stem from experimenter demand effects (EDEs)—bias that occurs when participants infer the purpose of an experiment and respond so as to help confirm a researcher’s hypothesis. We argue that online survey experiments have several features that make them robust to EDEs, and test for their presence in studies that involve over 12,000 participants and replicate five experimental designs touching on all empirical political science subfields. We randomly assign participants information about experimenter intent and show that providing this information does not alter the treatment effects in these experiments. Even financial incentives to respond in line with researcher expectations fail to consistently induce demand effects. Research participants exhibit a limited ability to adjust their behavior to align with researcher expectations, a finding with important implications for the design and interpretation of survey experiments.

The dynamics of behavior in modified dictator games.

We investigate the dynamics of individual pro-social behavior over time. The dynamics are tested by running the same experiment with the same subjects at several points in time. To exclude learning and reputation building, we employ non-strategic decision tasks and a sequential prisoners-dilemma as a control treatment. In the first wave, pro-social concerns explain a high share of individual decisions. Pro-social decisions decrease over time, however. In the final wave, most decisions can be accounted for by assuming pure selfishness. Stable behavior in the sense that subjects stick to their decisions over time is observed predominantly for purely selfish subjects. We offer two explanation for our results: diminishing experimenter demand effects and moral self-licensing.

Demand Effects in Survey Experiments: An Empirical Assessment

Survey experiments are ubiquitous in social science. A frequent critique is that positive results in these studies stem from Experimenter Demand Effects (EDEs)---bias that occurs when participants infer the purpose of an experiment and respond so as to help confirm a researcher's hypothesis. We argue that survey experiments have several features that make them robust to EDEs, and test for their presence in studies that involve over 12,000 participants and replicate five experimental designs touching on all empirical political science subfields. We randomly assign participants information about experimenter intent and show that providing this information does not alter the treatment effects in these experiments. Even financial incentives to respond in line with researcher expectations fail to consistently induce demand effects. Research participants exhibit a limited ability to adjust their behavior to align with researcher expectations, a finding with important implications for the design and interpretation of survey experiments.

Measuring and Bounding Experimenter Demand

We propose a technique for assessing robustness of behavioral measures and treatment effects to experimenter demand effects. The premise is that by deliberately inducing demand in a structured way we can measure its influence and construct plausible bounds on demand-free behavior. We provide formal restrictions on choice that validate our method, and a Bayesian model that microfounds them. Seven pre-registered experiments with eleven canonical laboratory games and around 19,000 participants demonstrate the technique. We also illustrate how demand sensitivity varies by task, participant pool, gender, real versus hypothetical incentives, and participant attentiveness, and provide both reduced-form and structural analyses of demand effects.

Bidding for nothing? The pitfalls of overly neutral framing

ABSTRACTNeutral framing is a standard tool of experimental economics. However, overly neutral instructions, which lack any contextual clues, can lead to strange behaviour. In a contextless second-price auction for a meaningless good, a majority of subjects enter positive bids – likely a case of cognitive experimenter demand effect. Subjects may interpret the lack of context as being tasked with bidding in the experiment. Adding another auction that has a context drastically reduces the positive bids in the meaningless auction.

Old age and prosocial behavior: Social preferences or experimental confounds?

Experimental and field evidence indicate a positive link between social preferences and age, most strikingly between the elderly and young adults. However, it is possible that the seemingly positive link between age and preferences stems from confounds in experimental procedure. In a dictator game study we find that elderly participants do indeed transfer higher shares of their endowments to their peers than a standard sample of student participants. This result holds good even in treatments accounting for wealth differences and experimenter demand effects. However, we observe no difference in behavior when we compare elderly participants and students who have not previously participated in economic experiments. Accordingly, it is possible that the seemingly stronger social preferences of the elderly are due to confounds associated with lack of experience with economic experiments. In addition, when comparing incentivized and hypothetical transfer decisions, we observe a hypothetical bias in treatments with a “take” framing, but not in treatments with the standard “give” framing.

Rule-Inequality-Aversion Preference and Conditional Cooperation in Public Goods Experiments: Economic Experiment Evidence from China

We provide an experimental investigation of the fair rule preference and its effect on anonymous cooperation, intending to test the hypothesis that people who exhibit high degrees of rule inequality aversion—consider “equality and mutual obligation under the social contract” as justified, are disgusted with free-rider behavior in the public goods game and are more likely to be conditional cooperators and make conditional contributions. Based on experimental work related to rule-preferences Chlas and Moffatt (Giving in dictator games: experimenter demand effect or preference over the rules of the game? (No. 2012, 044). Jena Economic Research Papers, 2012) and the public goods game Fischbacher et al. (Am Econ Rev 100(1):541–556, 2010), we designed a dictator game with the option of buying a veto (when the veto was purchased, the level of rule inequality decreased, making it an ultimatum game) to measure rule preference, i.e., rule inequality aversion, and conducted a two-stage public goods experiment to identify the cooperation preferences and repeated cooperation levels of participants. We investigated the relationship between rule preferences and cooperative preferences (conditional reciprocal preference) and tried to explain the adjustments people made when making repeated contributions of public goods (conditional cooperation behavior) from a rule preference perspective. The results show that: (1) Heterogeneous rule-inequality-aversion preferences are found in lab; (2) The stronger the rule-inequality-aversion preference is, the higher the probability is for the individual to become a conditional cooperator in public goods experiment; (3) Individuals who have stronger rule-inequality-aversion are more sensitive to the contributions that are deviated from the norm. Compared to the theory of conditional cooperation types, rule-inequality-aversion preference has a better explanation power when it comes to the behavior adjustment in repeated public goods contribution.