Randomized Survey Experiment Research Articles

Patient trust in the use of machine learning-based clinical decision support systems in psychiatric services: A randomized survey experiment.

Clinical decision support systems (CDSS) based on machine-learning (ML) models are emerging within psychiatry. If patients do not trust this technology, its implementation may disrupt the patient-clinician relationship. Therefore, the aim was to examine whether receiving basic information about ML-based CDSS increased trust in them. We conducted an online randomized survey experiment in the Psychiatric Services of the Central Denmark Region. The participating patients were randomized into one of three arms: Intervention=information on clinical decision-making supported by an ML model; Active control=information on a standard clinical decision process, and Blank control=no information. The participants were unaware of the experiment. Subsequently, participants were asked about different aspects of trust and distrust regarding ML-based CDSS. The effect of the intervention was assessed by comparing scores of trust and distrust between the allocation arms. Out of 5800 invitees, 992 completed the survey experiment. The intervention increased trust in ML-based CDSS when compared to the active control (mean increase in trust: 5% [95% CI: 1%; 9%], p=0.0096) and the blank control arm (mean increase in trust: 4% [1%; 8%], p=0.015). Similarly, the intervention reduced distrust in ML-based CDSS when compared to the active control (mean decrease in distrust: -3%[-1%; -5%], p=0.021) and the blank control arm (mean decrease in distrust: -4% [-1%; -8%], p=0.022).No statistically significant differences were observed between the active and the blank control arms. Receiving basic information on ML-based CDSS in hospital psychiatry may increase patient trust in such systems.

The impact of specific health warning messages on Chinese young people's perception of smoking risks and quitting intentions.

Although a substantial body of research has analyzed the effectiveness of cigarette package warning labels in tobacco control, the very general health warnings messages (HWMs) on cigarette packaging in China have shown limited effectiveness in deterring youth from smoking. Therefore, this study investigates the impact of specific and more detailed warning text messages on Chinese young people's risk perception of smoking and their intention to quit. We employed a randomized survey experiment to examine the impact of specific text-based warning labels on Chinese young people's risk perception of smoking and intention to quit. The total effective sample size was 1064 participants. The subjects were divided into three groups: the first group served as the control group, which was shown the existing cigarette package warning labels; the second group was shown cigarette package warning labels related to cardiovascular, digestive, and respiratory diseases; and the third group was shown cigarette package warning labels related to sexual dysfunction. The respiratory disease-related warnings significantly increased young people's awareness of smoking-related respiratory risks (p<0.01). The impact of warning labels for the three common diseases on enhancing young people's overall risk perception of smoking (p<0.05) and their intention to quit exhibited only weak statistical significance (p<0.05). In contrast, warning labels related to sexual dysfunction significantly increased young people's risk perception of smoking (p<0.001) and their intention to quit (p<0.001), with a much higher level of statistical significance compared to those related to the other three common diseases. Detailed descriptions of the risks associated with all four diseases were positively correlated with awareness of smoking-related harm and the intention to quit. However, warnings related to sexual dysfunction had a greater level of statistical significance compared to those related to the other three common diseases. This stronger significance may be attributed to young people's heightened concern about sexual dysfunction.

The Effects of Behaviorally Informed Messages on COVID-19 Vaccination Intentions and Behavior: Evidence from Randomized Survey Experiments in South Africa

With COVID-19 vaccination rates remaining below optimal levels, scalable interventions to shift vaccination intentions are needed. We embedded two randomized experiments in South Africa’s COVID-19 Vaccine Survey (CVACS) to investigate the potential to change vaccine intentions and behavior. In Experiment 1, 3510 unvaccinated South African adults were randomly assigned to a no-message control group, a social norm message, or a message highlighting that vaccines were free, available and easy to obtain. The free and easy message significantly increased vaccine intentions but did not increase other outcome measures. The social proof message was associated with an increase in self-reported vaccination status at follow-up (not significant at traditional statistical thresholds). In Experiment 2, 3608 unvaccinated South African adults were randomly assigned to a no-message control group, a message highlighting gaining greater freedoms, or a message highlighting being part of the solution to the pandemic. Neither value proposition message increased vaccination intentions. Light-touch and scalable messages informed by behavioral science and social marketing principles may increase vaccination intentions and uptake, However, more attention should be paid to understanding the behavioral barriers experienced by different population segments, and to tailoring and targeting messaging to those barriers and segments.

The German Business Panel: Firm-Level Data for Accounting and Taxation Research

ABSTRACT The German Business Panel (GBP) periodically surveys key decision-makers via a large sample of companies in Germany. The survey questions investigate managers’ accounting and taxation choices as well as the expected and perceived outcomes of those decisions. To obtain causal evidence, the survey supports the use of randomized survey experiments. The evidence from the GBP can meaningfully advance our understanding of issues that require data on internal processes as well as expectations, perceptions, and objectives behind ex-post reported accounting figures. The target population comprises the universe of legal entities included in the official German Statistical Business Register. We show that the dominance of small and medium-sized entities is a feature that Germany shares with many European countries, implying that GBP findings will be reasonably generalizable to European settings. We illustrate the usefulness of GBP data by presenting evidence from the initial waves of the GBP during the COVID-19 pandemic. The findings show how government aid programs contributed unevenly to the solvency of the most negatively affected companies, and how companies, rather than consumers, benefited from a temporary reduction of VAT rates. The paper also demonstrates how the scientific community can use the GBP data.

Service satisfaction among a language minority: a randomized survey experiment on the satisfaction of Swedish-speaking Finns with early childhood education

Service satisfaction among a language minority: a randomized survey experiment on the satisfaction of Swedish-speaking Finns with early childhood education

Tyranny-of-the-Minority Regression Adjustment in Randomized Experiments

Regression adjustment is widely used in the analysis of randomized experiments to improve the estimation efficiency of the treatment effect. This article reexamines a weighted regression adjustment method termed tyranny-of-the-minority (ToM), wherein units in the minority group are given greater weights. We demonstrate that ToM regression adjustment is more robust than Lin’s regression adjustment with treatment-covariate interactions, even though these two regression adjustment methods are asymptotically equivalent in completely randomized experiments. Moreover, ToM regression adjustment can be easily extended to stratified randomized experiments and completely randomized survey experiments. We obtain the design-based properties of the ToM regression-adjusted average treatment effect estimator under such designs. In particular, we show that the ToM regression-adjusted estimator improves the asymptotic estimation efficiency compared to the unadjusted estimator, even when the regression model is misspecified, and is optimal in the class of linearly adjusted estimators. We also study the asymptotic properties of various heteroscedasticity-robust standard errors and provide recommendations for practitioners. Simulation studies and real data analysis demonstrate ToM regression adjustment’s superiority over existing methods. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Public officials’ motivated reasoning and their interpretation of policy information

ABSTRACT Evidence based policy making is premised on the idea that policymakers use policy information in an accurate and unbiased way. However, the interpretation and application of policy information is a cognitive process open to misinterpretation and bias, especially in politically salient policy environments. In this generalization and extension replication, we conduct two randomized survey experiments to test the effects of motivated reasoning and political salience on public officials’ ability to accurately interpret policy information. The results indicate limited support for motivated reasoning, and we find no evidence suggesting that political salience affects officials’ accurate and unbiased interpretation of policy information.

How does information about inequality shape voting intentions and preferences for redistribution? Evidence from a randomized survey experiment in Indonesia

We test the elasticity of people's voting intentions and preferences for redistribution to information about inequality through a large-scale, randomised survey experiment in Indonesia. Respondents received information about either (1) the level of national inequality, (2) the level of national inequality in combination with the degree of intergenerational mobility, (3) their position in the national income distribution, or no information. The first two treatments raised people's concern about inequality and mobility. The first treatment also increased the likelihood they would vote against the President. The third treatment lowered richer respondents’ support for redistribution. These findings provide new insights about the challenges of increasing public support for government-led redistribution, such as tax increases and greater spending on social protection, in middle-income country settings.

The Role of Salience and Memory in Fertility Decisions: Experimental Evidence

Public policies use communication campaigns to affect individual behavior. We analyze how providing women with information on the beneficial effects of using formal childcare may affect their realized fertility. We argue that cues in the messages are particularly salient for mothers and women with fertility intensions, since they activate these women’s past memories. Hence, cues induce these women to create mental representations of future actions, such as realized fertility. We exploit a randomized survey experiment run in 2011, which provides information on the positive effects that attending daycare may have on the children’ future cognitive development. Using a follow-up survey run six-year later we show that the treatment increases realized fertility among mothers and women with declared fertility intentions, for whom the communication was more salient. Yet, the treatment did not affect the individual knowledge nor recall of the information provided in the message. Our results carry important policy implications: persuading individuals is difficult, but communication can be effective if salient.

Validation of the Very Efficient Short‐Form Need for Cognition Scale in Chinese: Detecting the effect of item orientation

AbstractUsing a large‐scale randomized survey experiment, this study examined the dimensionality, item functioning, instrument properties and associations with validity‐related variables of the Very Efficient Short‐Form Need for Cognition Scale (NCS‐6) in Chinese and investigated whether its psychometric properties were affected by item orientation. For dimensionality, results indicated that a two‐factor model demonstrated a superior fit to the data compared to the one‐factor model when the NCS‐6 contained mixed‐worded items. The one‐factor model showed excellent model fit when the NCS‐6 contained only nonreversed or reversed items. For item functioning, factor loadings were invariant across experimental conditions, but item thresholds were not. For instrument properties, although the internal consistency reliabilities were excellent in each experimental condition, mean scores of the NCS‐6 showed a decreasing pattern as the number of reversed items increased. For associations with validity‐related variables, the structural relationships between NCS‐6 and validity‐related variables were not significantly different across experimental conditions. Overall, the dimensionality and instrument properties of the Chinese version NCS‐6 were affected when it contained mixed‐worded items.

Presidents and vaccines: head of state inoculation as a tool for vaccine promotion.

Vaccine hesitancy, an important threat to global health, has increased since the onset of the COVID-19 pandemic. The public vaccination of high-profile figures, such as heads of state, has been touted as a potential tool for increasing vaccine acceptance among the general population. However, systematic information on such role modelling is lacking and existing studies focus on a small number of high-income countries. We take advantage of the COVID-19 pandemic to fill this gap. Through a systematic search of internet sources, we first document that most global leaders supported the vaccination campaign and actively communicated their vaccination status to the public. We then turn to a case study to provide experimental evidence on vaccine role modelling for a country in Africa - the region that is most lagging behind in achieving universal immunization coverage. We rely on a randomized survey experiment with 600 citizens in the Democratic Republic of Congo and take advantage of the fact that the Congolese President publicly received a COVID-19 vaccine during the survey period. Our findings demonstrate that the impact of political leader's role modelling is moderated by trust and depends on media outreach and access. When trust in leaders is lacking, or news on their actions is inaccessible, alternative ambassadors and effective communication methods become crucial in motivating and informing the public. This may be especially relevant in fragile states and remote regions.

Informing the recruitment crisis in policing: Evaluating which incentives can entice the best candidates among college students

AbstractResearch SummaryScrutiny over the quality of police work continues to increase, at the same time that a recruitment crisis may hinder policing services. Moreover, this crisis exists despite numerous work incentives for officers, which are potentially competitive salaries, early pensions, and legal protections. Using a sample of 604 undergraduate students—a natural recruiting pool for police agencies—this study explores the predictors of students’ interest in a career in policing. In addition, we conducted a randomized survey experiment to identify the effect of work incentives on students’ willingness to apply to be a police officer upon graduation. Specifically, we manipulated the starting salary, years of service until pension eligibility, and the protection afforded against lawsuits. Finally, we tested whether the effect of these incentives differs across the quality of applicants (i.e., grade point average [GPA], ability to pass existing application criteria, and subjective qualities that comprise honesty, teamwork, and compassion) and their preexisting interest in a career in policing. We found that students are generally uninterested in becoming police officers, particularly when their GPA is high. Though the starting salary was the most impactful of all incentives, some lawsuit protection is key for enticing the best students. In contrast, conditions for pension had only modest if any effects.Policy ImplicationsRecruiting is key to improving the quality of policing, particularly when many potential candidates are overlooking the career. However, research on the causes and consequences of police recruitment strategies is limited. This study implements rigorous research methods to investigate how to increase the pool of high‐quality applicants to police agencies. Findings support sensible and potentially cost‐effective policy decisions in regard to police recruiting, which should focus on increasing the starting salary, and on more effectively communicating the value of other existing benefits of the career that comprise lawsuit protection and attractive pension plans.

Do online pension dashboards affect pension knowledge and expectations? Evidence from a randomized survey experiment

Do online pension dashboards affect pension knowledge and expectations? Evidence from a randomized survey experiment

Public Opinions on Removing Disincentives and Introducing Incentives for Organ Donation: Proposing a European Research Agenda.

The shortage of organs for transplantations is increasing in Europe as well as globally. Many initiatives to the organ shortage, such as opt-out systems for deceased donation and expanding living donation, have been insufficient to meet the rising demand for organs. In recurrent discussions on how to reduce organ shortage, financial incentives and removal of disincentives, have been proposed to stimulate living organ donation and increase the pool of available donor organs. It is important to understand not only the ethical acceptability of (dis)incentives for organ donation, but also its societal acceptance. In this review, we propose a research agenda to help guide future empirical studies on public preferences in Europe towards the removal of disincentives and introduction of incentives for organ donation. We first present a systematic literature review on public opinions concerning (financial) (dis)incentives for organ donation in European countries. Next, we describe the results of a randomized survey experiment conducted in the United States. This experiment is crucial because it suggests that societal support for incentivizing organ donation depends on the specific features and institutional design of the proposed incentive scheme. We conclude by proposing this experiment's framework as a blueprint for European research on this topic.

Comparing Student and Writing Instructor Perceptions of Academic Dishonesty When Collaborators Are Artificial Intelligence or Human

It remains unclear if perceptions of academic dishonesty concerning artificial intelligence writing technologies (AIWTs) present new challenges or if they reflect prior, non-AI concerns. To structure this problem, we used a randomized control survey experiment. We compared student ( n = 603) and instructor ( n = 312) attitudes toward dishonesty in collaborations involving humans versus AIWT in 10 writing-related scenarios. Results suggest similar perception patterns among students and instructors, with both populations expressing significant differences in perceived dishonesty between AI and human collaborators in some scenarios. This experiment structures the problem of AI writing and academic dishonesty for future research in this emerging field.

The effectiveness of between-wave mailings and tailored material incentives on response rates: results from a young adolescent longitudinal survey

ABSTRACT Mailing campaigns are a way to keep longitudinal survey respondents engaged. While mailings usually include a survey answer request, sometimes respondents are contacted between-waves to update contact information or simply to keep respondent contact. Research on the actual impact of these between-wave contacts on response rates is scarce. This paper addresses three research questions: (1) Do between-wave campaigns impact response rate? (2) Does the addition of non-monetary incentives to between-wave campaigns further impact response rates? (3) Do between-wave campaigns also affect other members of the household? Following a one-year break between survey waves, we design a randomized survey experiment assigning respondents to five treatment conditions. Results show that keeping-in-touch mailings did not significantly increase the likelihood of an interview in the National Educational Panel Study for either children or their parents in general. However, when we focus on specific characteristics of the targets, significant differences are found.

Hindsight Bias and Trust in Government

Abstract We empirically assess whether hindsight bias affects citizens' evaluation of their political actors. Using an incentivized elicitation technique, we demonstrate that people systematically misremember their past policy preferences regarding how to best fight the Covid-19 pandemic. At the peak of the first wave in the United States, the average respondent mistakenly believed that they supported significantly stricter restrictions at the onset of the first wave than they actually did. Exogenous variation in the extent of hindsight bias, induced through a randomized survey experiment, indicates that hindsight bias has a negative causal impact on the change in trust in government.

Open Voice or Private Message? The Hidden Tug-of-War on Social Media Customer Service

Firms use social media as a great marketing tool and a convenient platform to deliver customer service. However, due to its public and social nature, social media tends to amplify a brand's successes as well as failures. Reluctant to subject their customer service to public scrutiny, firms are increasingly turning to private messaging on their social media channels for customer service conversations, which amounts to a reincarnation of traditional customer service in the social media era. Nonetheless, whether customers are willing to relinquish their newfound power is unclear. In this paper, we analyze a natural experiment where the inconvenience of the private channel with the treated firm is suddenly eliminated, and we find evidence that customers prefer to complain through the public channel. A randomized survey experiment further confirms this insight. Overall, firms’ and customers’ diverging preferences toward public or private channel reveal a hidden tug-of-war between the traditional mode of customer service featuring firm control and the recently emerged mode of customer service featuring shared control. These findings have important implications for firms’ customer service operations.

The impact of price and image warnings on the social perception of gifting cigarettes in China.

The tobacco gift-giving culture in China poses a significant challenge to public health; however, there is limited research on effectively curbing the tobacco gift-giving culture and its associated tobacco gift consumption. This study examines the potential impact of two tobacco control measures that the Chinese government may consider adopting on cigarette gifting behavior in the future in Chinese society. This study employed a randomized survey experiment to examine the effects of cigarette price treatment and pictorial health warning labels (HWLs) on cigarette gifting. The total sample size of this study is 1035. Four groups of participants were presented with representative cigarettes categorized into high-, medium-, and low-priced products, along with different prices (normal and double) or external packaging imagery (normal and pictorial HWL versions). The price of cigarettes for personal consumption forms an L-shaped distribution, and the price of cigarette gifts forms a W-shaped distribution. Increasing cigarette prices reduces smokers' willingness to gift high-priced cigarettes but stimulates the consumption of low-price cigarettes as gifts. Pictorial HWLs do not directly influence smokers' intentions to gift cigarettes, but they enhance the effectiveness of price regulation concerning medium-priced cigarette products. If the price variance of cigarettes is not reduced, the effect of price regulation will be very limited. Implementing combined interventions of pictorial HWLs and price regulation or modifying the pricing structure of tobacco products may yield stronger control outcomes.

Managing crises as if no one is watching? Governance dilemmas from a public perspective

AbstractIn the midst of ongoing crises, understanding how citizens perceive administrative crisis management is more relevant than ever. Combining organizational literature with insights from legitimacy research, this article scrutinizes how the public evaluates governance decisions concerning prominent crisis management dilemmas: flexibility versus stability, inclusion versus exclusion, and equity‐based versus needs‐based resource distribution. The paper argues that flexible, inclusive, and equity‐based governance decisions are generally perceived as more legitimate. However, governance decisions are also associated with adverse effects that can mitigate any initially positive effect on legitimacy. The argument is tested in a large‐scale randomized survey experiment in the context of a migration crisis, where governance decisions were manipulated. The findings support the expectations for inclusive crisis management and equity‐based resource distribution, which are perceived as the most legitimate governance alternatives. Internal adaptations of administrative practices toward more flexible and adaptive solutions, however, are perceived less legitimate than stable governmental action.