Teaching simulation-based inference: Random sampling vs. random assignment? What instructors should know

Abstract

ABSTRACT “Simulation-based Inference” is often considered a pedagogical strategy for helping students develop inferential reasoning, e.g., giving them a visual and concrete reference for deciding whether the observed statistic is unlikely to happen by chance alone when the null hypothesis is true. In this paper, we highlight for teachers some implications of different simulation strategies when analyzing two variables. In particular, does it matter whether the simulation models random sampling or random assignment? We present examples from comparing two means and simple linear regression, highlighting the impact on the standard deviation of the null distribution. We also highlight some possible extensions that simulation-based inference easily allows.