Comparing cross-classified mixed effects and Bayesian structural equations modeling for stimulus sampling designs: A simulation study

Abstract

Researchers examining a wide range of psychological phenomena, including interpersonal perception, attitude formation, and stereotype activation, apply stimulus sampling designs (SSD). The standard SSD study requires participant raters to provide evaluations of a series of target stimuli (e.g., photographs, media clips, vignettes), and the constituent responses are simultaneously nested within participants and stimuli, yielding a cross-classified data structure. Prior methodological work has illustrated the application of both cross-classified mixed effects, and cross-classified structural equation modeling to accommodate the corresponding dependency structure. Despite their widespread application, little is known about how sample size for both participants and stimuli is associated with inferential power and coverage in SSDs. Even less is known about the feasibility of random slopes, or whether the frequentist (maximum likelihood) and Bayesian (MCMC) estimators differ in accuracy or efficiency under the design conditions typically observed in SSD studies. We conducted a Monte Carlo simulation study to better understand parameter bias, statistical power, and confidence or credible interval coverage, as a function of the number of participant raters and target stimuli, effect size, as well as the presence of random slopes, and modeling framework. Recommendations for future research are provided.