Randomized Field Experiment Research Articles

Social Lobbying

We theorize that direct social lobbying—the meeting of a lobbyist and public official outside of a formal office—persuades officials to support publicly policies favored by interest groups. Social lobbying influences public officials because the social environment allows for greater receptivity to interest group messages. A randomized field experiment was conducted by a lobbying firm in a US state legislature. Legislators randomly assigned to be socially lobbied more frequently expressed public support for the interest group’s preferred policy than did legislators lobbied in their offices or not contacted by the lobbyist. In addition, an original survey of registered lobbyists was conducted in 10 US states demonstrating that social lobbying regularly occurs. Political elites are influenced by the social environment; interest group direct lobbying is influential when conducted in places not easily observed by the public.

Reducing bias in preference elicitation for environmental public goods*

The recent stated preference literature emphasises the importance of incentive compatible elicitation methods, which depend on respondent beliefs that payment can be collected if provision occurs. We investigate this condition in a randomised field experiment where stated choices are incentivised financially. The objective of the treatment was to make choices salient by making each decision financially relevant and to increase the respondents' beliefs that future payments will be enforced. Our results show that the treatment increases estimates of the marginal utility of income, with the effect being economically and statistically significant for low‐income respondents. We develop a stylised theoretical framework that allows us to quantify the bias that is implied by the observed differences between the treated and control groups. We find that failure to account for respondents' doubts about payment coercion in an otherwise well‐designed survey inflates the marginal willingness to pay among low‐income respondents by a factor of at least 1.72.

Medicaid prevalence and opioid use disorder treatment access disparities.

To examine how variation in the size of the local Medicaid population moderates Medicaid-to-private treatment access differentials for women with opioid use disorder (OUD). County-level information on total Medicaid enrollment combined with randomized field experiment data from 10 diverse states that used a simulated patient (audit) methodology to examine buprenorphine providers' appointment granting behavior. We used multiple regression modeling approaches to capture the moderating influence of Medicaid prevalence on differences in the likelihood of receiving an insurance-covered appointment between Medicaid and privately insured female patients. Completed calls to buprenorphine treatment providers. We find a 0.37 percentage point (p value <0.01) narrowing of the Medicaid-to-private access gap with each one percentage point increase in the local insured population on Medicaid. There is effectively no difference in the likelihood of being granted an insurance-covered appointment across the two payer groups in the top tercile of Medicaid penetration. When Medicaid is a common source of insurance within the local population, buprenorphine providers are much less likely to discriminate between Medicaid and privately insured prospective patients. Efforts to enhance equitable access across patient groups are perhaps best targeted where Medicaid prevalence is lower.

Competitive Poaching in Search Advertising: Two Randomized Field Experiments

Keyword searches with brand names enable firms to generate traffic from search advertising by bidding not only on their own keywords but also on competitors’ keywords. The strategy of bidding on competitors’ keywords, known as competitive poaching, presents unique opportunities for practitioners. This study examines factors that influence the effectiveness of competitive poaching. We collected data from two randomized field experiments, one with a business school in the Northeastern United States and the other one with a leading automobile dealership company, where these firms bid on keywords of competing brands and randomly display different types of ad copies in the sponsored search listings. We find that, when poaching on keywords of high-quality brands, ad copies that feature vertical differentiation through quality signals are more effective than the control ad copies that do not convey any differentiation or prescriptive messages. We also find that when poaching from low-quality brands, ad copies featuring horizontal differentiation through nonquality attributes perform better than the control ad copies. Finally, the presence of the poached brand’s own ad has a positive association with the ad effectiveness of the poaching brand when that poached brand is high quality and a negative association when the poached brand is low quality.

The effect of information nudges on energy saving: Observations from a randomized field experiment in Finland

Field experiments have shown that information nudging can help households to save energy, however, the effectiveness varies depending on aspects such as information content, delivery mode and study area. This article evaluates the impacts of information nudges on residential electricity consumption with a randomized field experiment. This opt-in experiment was conducted in Finland. Information was administered via monthly email newsletters and an online energy service platform. The aim is to find out whether i) energy saving tips combined with and without online energy service platform providing electricity consumption information, and ii) peer comparisons (i.e., social norm) influence households’ electricity consumption. The results show a high seasonal variation in the treatment effects within the groups who were registered users of the online energy service platform. Those with access to usage feedback and versatile energy savings tips (without the social norm comparisons) reduced their electricity consumption around 10% in wintertime. The results imply challenges in encouraging energy saving behavior among households less interested in following their electricity consumption.

Nudging student participation in online evaluations of teaching: Evidence from a field experiment

This paper reports the results of a large randomized field experiment that investigates the extent to which nudges can stimulate student participation in teaching evaluations. The three nudges that we used were designed to either: (1) heighten students’ perceived impact of teaching evaluations, (2) communicate a descriptive norm of high participation, and (3) use the commitment-consistency principle by asking students to commit to participation. We find that none of the nudges were effective: all treatment effects are insignificant and close to zero in magnitude. Exploring heterogeneous treatment effects, we find evidence that the effectiveness of both the impact and commitment treatments differed across students. The impact treatment had a negative effect on the participation of bachelor-level students, but not on that of master-level students. The commitment treatment increased participation among students with good average grades, whereas it decreased participation for students whose average grades were poor.

Home-Tutoring Services Assisted with Technology: Investigating the Role of Artificial Intelligence Using a Randomized Field Experiment

Despite a rising interest in artificial intelligence (AI) technology, research in services marketing has not evaluated its role in helping firms learn about customers’ needs and increasing the adaptability of service employees. Therefore, the authors develop a conceptual framework and investigate whether and to what extent providing AI assistance to service employees improves service outcomes. The randomized controlled trial in the context of tutoring services shows that helping service employees (tutors) adapt to students’ learning needs by providing AI-generated diagnoses significantly improves service outcomes measured by academic performance. However, the authors find that some tutors may not utilize AI assistance (i.e., AI aversion), and factors associated with unforeseen barriers to usage (i.e., technology overload) can moderate its impact on outcomes. Interestingly, tutors who significantly contribute to the firm's revenue relied heavily on AI assistance but unexpectedly benefited little from AI in improving service outcomes. Given the wide applicability of AI assistance in a variety of services marketing contexts, the authors suggest that firms should consider the potential difficulties employees face in using the technology rather than encourage them to use it as it is.

“Outside Lobbying” over the Airwaves: A Randomized Field Experiment on Televised Issue Ads

We present the first field experiment on how organized interest groups’ television ads affect issue opinions. We randomized 31,404 voters to three weeks of interest group ads about either immigration or transgender nondiscrimination. We then randomly assigned voters to receive ostensibly unrelated surveys either while the ads aired, one day after they stopped, or three days afterwards. Voters recalled the ads, but three ads had a minimal influence on public opinion, whereas a fourth’s effects decayed within one day. However, voters remembered a fact from one ad. Our results suggest issue ads can affect public opinion but that not every ad persuades and that persuasive effects decay. Despite the vast sums spent on television ads, our results are the first field experiment on their persuasive power on issues, shedding light on the mechanisms underpinning—and limits on—both televised persuasion and interest group influence.

Your Child Missed Learning the Alphabet Today: A Randomized Trial of Sending Teacher-Written Postcards Home to Reduce Absences

As a growing base of evidence links student success with attendance, many states have chosen to include measures of student absenteeism in their accountability systems under ESSA. Consequently, school districts face strong incentives to improve attendance. We report on a randomized field experiment in which parents of early elementary students received personalized information about the academic content their child missed while absent. Following an absence, school staff sent postcards to parents detailing how many days of school their child had missed alongside a handwritten note from their teacher summarizing the academic material covered during the absence. We randomized the intervention across schools and classrooms in two urban school districts. The analysis sample included 5,552 students in preschool through second grade. Overall, the treatment reduced absences by 8.3% (95% CI, 2.6–14.0%). The results provide evidence that a postcard intervention designed and implemented by schools and teachers rather than 3rd parties can be effective in reducing absences. The teachers’ annotations with personalized academic and absence information are a unique feature of the intervention that suggests areas for further research.

Do exhausted primary school students cheat more? A randomized field experiment.

Motivated by the two-decade-long scientific debate over the existence of the ego-depletion effect, our paper contributes to exploring the scope conditions of ego-depletion theory. Specifically, in a randomized experiment, we depleted students' self-control with a cognitively demanding task that required students' effort. We measured the effect of depleted self-control on a subsequent task that required self-control to not engage in fraudulent cheating behavior-measured with an incentivized dice-roll task-and tested ego-depletion in a large-scale preregistered field experiment that was similar to real-life situations. We hypothesized that treated students would cheat more. The data confirms the hypothesis and provides causal evidence of the ego-depletion effect. Our results provide new insights into the scope conditions of ego-depletion theory, contribute methodological information for future research, and offer practical guidance for educational policy.

Face mask use and physical distancing before and after mandatory masking: No evidence on risk compensation in public waiting lines

During the COVID-19 pandemic, the introduction of mandatory face mask usage triggered a heated debate. A major point of controversy is whether community use of masks creates a false sense of security that would diminish physical distancing, counteracting any potential direct benefit from masking. We conducted a randomized field experiment in Berlin, Germany, to investigate how masks affect distancing and whether the mask effect interacts with the introduction of an indoor mask mandate. Joining waiting lines in front of stores, we measured distances kept from the experimenter in two treatment conditions – the experimenter wore a mask in one and no face covering in the other – in two time spans – before and after mask use becoming mandatory in stores. We find no evidence that mandatory masking has a negative effect on distance kept toward a masked person. To the contrary, masks significantly increase distancing and the effect does not differ between the two periods. However, we show that after the mandate distances are shorter in locations where more non-essential stores, which were closed before the mandate, had reopened. We argue that the relaxations in general restrictions that coincided with the mask mandate led individuals to reduce other precautions, like keeping a safe distance.

Affirmative Action and Human Capital Investment: Evidence from a Randomized Field Experiment

Pre-College human capital investment occurs within a competitive environment and depends on market incentives created by Affirmative Action (AA) in college admissions. These policies affect mechanisms for rank-order allocation of college seats, and alter the relative competition between blacks and whites. We present a theory of AA in university admissions, showing how the effects of AA on human capital investment differ by student ability and demographic group. We then conduct a field experiment designed to mimic important aspects of competitive investment prior to the college market. We pay students based on relative performance on a mathematics exam in order to test the incentive effects of AA, and track study efforts on an online mathematics website. Consistent with theory, AA increases average human capital investment and exam performance for the majority of disadvantaged students targeted by the policy, by mitigating so-called discouragement effects. The experimental evidence suggests that AA can promote greater equality of market outcomes and narrow achievement gaps at the same time.

Taking a few deep breaths significantly reduces children's physiological arousal in everyday settings: Results of a preregistered video intervention.

This preregistered, randomized field experiment tested the effectiveness of a brief deep breathing intervention on children's concurrent physiological arousal in naturalistic settings (N=342; Mage =7.48 years; 46% female; 53% Asian, 26% White; 21% other race/ethnicity). The treatment consisted of an animated video that introduced deep breathing as a self-regulation strategy and scaffolded the child in taking a few slow-paced breaths, while the control group watched an informational video featuring similar animated images. Respiratory sinus arrhythmia (RSA) and heart rate (HR) were measured while children were sitting still (baseline) and subsequently while watching 1-min videos. Relative to baseline arousal, RSA increased and HR decreased only in response to the deep-breathing treatment video. Effects were larger in the second 30-s epoch of the video, which included most of the deep breathing practice. RSA fully mediated the intervention's effects on HR. By analyzing all children exposed to intervention video regardless of their engagement in the deep breathing practice (intention-to-treat design) and by using easily scalable treatment videos, the study identifies an effective and pragmatic approach to reducing children's physiological arousal in everyday, group settings. Implications for advancing applied developmental psychophysiological research are discussed.

Leveraging the Power of Peer Groups for Refugee Integration

Refugee integration, one long-term solution to the large number of people fleeing their home countries, constitutes a challenge for both refugees and host societies. ICT and especially online peer groups seem promising to support this process. Building on literature demonstrating the societal benefits of peer groups, this paper proposes a novel peer-group-based approach to address refugee integration and introduces both an online and offline realization. A randomized field experiment in cooperation with public (refugee) services and a non-governmental organization makes it possible to expand existing research by quantitatively demonstrating societal benefits of online peer groups and ICT for refugee integration. Further, this paper is the first to assess the effectiveness of online and offline peer groups in one experimental setup comparatively. Results show that peer groups provide substantial value with respect to the integration domains social bridges, social bonds, rights and citizenship as well as safety and stability. While the outcome of the various integration domains differs for online and offline peer groups, participants’ adoption rates were higher for online peer groups.

Face masks increase compliance with physical distancing recommendations during the COVID-19 pandemic

Governments across the world have implemented restrictive policies to slow the spread of COVID-19. Recommended face mask use has been a controversially discussed policy, among others, due to potential adverse effects on physical distancing. Using a randomized field experiment (N = 300), we show that individuals kept a significantly larger distance from someone wearing a face mask than from an unmasked person during the early days of the pandemic. According to an additional survey experiment (N = 456) conducted at the time, masked individuals were not perceived as being more infectious than unmasked ones, but they were believed to prefer more distancing. This result suggests that wearing a mask served as a social signal that led others to increase the distance they kept. Our findings provide evidence against the claim that mask use creates a false sense of security that would negatively affect physical distancing. Furthermore, our results suggest that behavior has informational content that may be affected by policies.

Increasing Enrollment in Income‐Driven Student Loan Repayment Plans: Evidence from the Navient Field Experiment

ABSTRACTWe report evidence from a randomized field experiment conducted by a major student loan servicer, Navient, in which student loan borrowers received prepopulated applications for income‐driven repayment (IDR) plans. Treatment increased IDR enrollment by 34 percentage points relative to the control group. Using the random treatment assignment as an instrument for IDR enrollment, we furthermore provide local average treatment effect (LATE) estimates of the effects of IDR enrollment on new delinquencies, monthly student loan payments, and consumer spending. Our study is the first field‐experimental evaluation of a U.S. government program designed to address the soaring debt burdens of U.S. households.

Is physician implicit bias associated with differences in care by patient race for metastatic cancer-related pain?

RationaleImplicit racial bias affects many human interactions including patient-physician encounters. Its impact, however, varies between studies. We assessed the effects of physician implicit, racial bias on their management of cancer-related pain using a randomized field experiment.MethodsWe conducted an analysis of a randomized field experiment between 2012 and 2016 with 96 primary care physicians and oncologists using unannounced, Black and White standardized patients (SPs)who reported uncontrolled bone pain from metastatic lung cancer. We assessed implicit bias using a pain-adaptation of the race Implicit Association Test. We assessed clinical care by reviewing medical records and prescriptions, and we assessed communication from coded transcripts and covert audiotapes of the unannounced standardized patient office visits. We assessed effects of interactions of physicians’ implicit bias and SP race with clinical care and communication outcomes. We conducted a slopes analysis to examine the nature of significant interactions.ResultsAs hypothesized, physicians with greater implicit bias provided lower quality care to Black SPs, including fewer renewals for an indicated opioid prescription and less patient-centered pain communication, but similar routine pain assessment. In contrast to our other hypotheses, physician implicit bias did not interact with SP race for prognostic communication or verbal dominance. Analysis of the slopes for the cross-over interactions showed that greater physician bias was manifested by more frequent opioid prescribing and greater discussion of pain for White SPs and slightly less frequent prescribing and pain talk for Black SPs with the opposite effect among physicians with lower implicit bias. Findings are limited by use of an unvalidated, pain-adapted IAT.ConclusionUsing SP methodology, physicians’ implicit bias was associated with clinically meaningful, racial differences in management of uncontrolled pain related to metastatic lung cancer. There is favorable treatment of White or Black SPs, depending on the level of implicit bias.

A Prescriptive Analytics Framework for Optimal Policy Deployment Using Heterogeneous Treatment Effects

We define a prescriptive analytics framework that addresses the needs of a constrained decision-maker facing, ex ante, unknown costs and benefits of multiple policy levers. The framework is general in nature and can be deployed in any utility-maximizing context, public or private. It relies on randomized field experiments for causal inference, machine learning for estimating heterogeneous treatment effects, and on the optimization of an integer linear program for converting predictions into decisions. The net result is the discovery of individual-level targeting of policy interventions to maximize overall utility under a budget constraint. The framework is set in the context of the four pillars of analytics and is especially valuable for companies that already have an existing practice of running A/B tests. The key contribution of this work is to develop and operationalize a framework to exploit both within- and between-treatment arm heterogeneity in the utility response function in order to derive benefits from future (optimized) prescriptions. We demonstrate the value of this framework as compared to benchmark practices—i.e., the use of the average treatment effect, uplift modeling, as well as an extension to contextual bandits—in two different settings. Unlike these standard approaches, our framework is able to recognize, adapt to, and exploit the (potential) presence of different subpopulations that experience varying costs and benefits within a treatment arm while also exhibiting differential costs and benefits across treatment arms. As a result, we find a targeting strategy that produces an order of magnitude improvement in expected total utility for the case where significant within- and between-treatment arm heterogeneity exists.

Limits to third-party reporting: Evidence from a randomized field experiment in Norway

Third-party reporting and employers’ tax withholding are powerful compliance mechanisms, as long as the employer and employee do not collude to evade. In cooperation with the Norwegian Tax Administration we designed a randomized field experiment with unannounced on-site audits. Matching audit data to administrative registers, we provide evidence of collusive tax evasion. We find that firms assigned to be audited increased their subsequent wage reporting on behalf of their employees by 18 percent relative to firms assigned to the control group. The effect is more pronounced among small firms with few employees. Our results document limitations of third-party reporting, but also that these limitations can be counteracted by minor on-site audits.

Yes, You Can! Effects of Transparent Admission Standards on High School Track Choice: A Randomized Field Experiment

Abstract High school track choice determines college access in many countries. We hypothesize that some qualified students avoid the college-bound track in high school simply because they overestimate admission requirements. To test this hypothesis, we conducted a randomized field experiment that communicated the admission standards of local secondary schools on the academic track to students in Hungary before the application deadline. We targeted the subset of students (“seeds”) who occupied the most central position in the classroom-social networks, aiming to detect both direct effects on the track choice of targeted seeds and spillover effects on their untreated peers. We found neither a direct effect nor a spillover effect on students’ applications or admissions on average. Further analyses, however, revealed theoretically plausible heterogeneity in the direct causal effect of the intervention on the track choice of targeted seeds. Providing information about admission standards increased applications and admissions to secondary schools on the academic track among seeds who had a pre-existing interest in the academic track but were unsure of their chances of admission. This demonstrates that publicizing admissions standards can set students on a more ambitious educational trajectory. We discuss the implications for theory and policy.