A novel experimental vignette methodology: SMART vignettes

Abstract

We motivate and present the methodology of vignette studies. The primary contribution of this paper is our proposal of a novel vignette study design: “SMART vignettes.” Our design has two notable features: the first is its use of sequential randomization, which conceptually originates from the sequential multiple assignment randomization trial (SMART) design developed by Murphy (2004). The second feature is adaptive allocation. These new features in vignette studies offer unique advantages not offered by traditional vignettes: (1) valid causal inferences on the conditional distributions of the primary outcome of interest, given other factors, (2) balanced allocations across groups, and (3) a greater degree of interactivity for the survey respondent. We illustrate the utility of our method using a case example of a vignette study used to probe physicians’ attitudes toward an AI-embedded clinical system. In this example, a SMART vignette was used to randomize hypothetical scenarios to gain a better understanding of the causal impact of physician attitudes, given emerging evidence that a range of factors including previous decisions, play a role in influencing clinical decisions. We simulated hypothetical vignette studies under both SMART and conventional (i.e. single randomization at baseline) designs. We varied the number of factors for each study and fixed each factor to have two levels. Relative loss was used to compare the degree of imbalance between groups. Both designs had smaller relative losses with larger sample sizes. The SMART study design had lower loss than its conventional counterpart for all values of [Formula: see text] for all studies, indicating better balance. As demonstrated by the relative loss in our simulations, our proposed SMART vignette design has an advantage over the conventional design. This method holds promise in generating new knowledge in decision making scenarios occurring over multiple and discrete time points.