Implementing planned missingness in stimulus sampling designs: Strategies for optimizing statistical power and precision while limiting participant burden

Abstract

The stimulus sampling design (SSD) is a widely applied research paradigm in which participants evaluate a series of visual or auditory stimuli. Researchers are often interested in phenomena reflected by differences across stimuli, and recent simulation work suggests that more than 100 stimuli may be necessary to detect smaller effects. Unfortunately, administering a large number of stimuli to each participant may compromise the validity of responses due to individual differences in susceptibility to fatigue and distraction. The present work describes the application of planned missingness design strategies to maximize statistical power, while minimizing participant burden in SSD studies. A Monte Carlo simulation study was conducted to determine the number of stimuli per rater (SPR) and average raters per stimuli (RPS) needed to ensure unbiased estimates of model parameters, as well as the desired statistical power and interval coverage. Findings suggest that two commonly used statistical estimation techniques, restricted maximum likelihood and the Markov Chain Monte Carlo algorithm, provide reliable estimates of model parameters and maintain adequate statistical power under substantial levels of planned missingness. Incorporating these methods will allow researchers to design more efficient and powerful experiments by reducing redundancies and minimizing sources of methodological error. Recommendations for incorporating planned missingness strategies in SSDs are provided.