An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials

Abstract

Background: Prior research has examined design parameters for assessing average program impacts on achievement outcomes with cluster randomized trials (CRTs). Little is known about parameters for assessing differential impacts for student subgroups. Objectives: This article develops the framework for optimally designing CRTs to detect differential impacts of programs by student subgroups and presents initial estimates of the critical parameters.Research Design: Estimates of intraclass correlation coefficients (ICCs) for the slope of the student-level moderator and Minimum Detectable Effect Sizes (MDESs) for average and differential impacts are calculated before and after conditioning on effects of covariates.Subjects: Data from six cluster randomized trials are analyzed with student outcomes in grades three through ten. Measures: Achievement in math, science, reading and writing is assessed with CRTs conducted in six states. Results: In figuring MDES for differential impact, the “ICC for the slope” – the between-cluster variation in the slope of the moderator divided by total variance – replaces the usual ICC. Estimates of median ICCs for the slope were .02 for gender and .04 for socioeconomic. Lower values of ICC for slope, compared to the usual ICC, contend with inflated influence of within-cluster variance in determining power to detect differential impact. For studies considered, after conditioning on covariates there was similar power to detect average and differential impacts (for socioeconomic status and gender) of the same size; and estimates of differential impact were often larger than of average impacts. Conclusions: Measuring differential impacts in CRTs is important for addressing questions of equity, generalizability, and to guide interpretation of subgroup impact findings. Adequate power to detect these effects appears reachable with CRTs designed to measure average impact. Ongoing collection of parameters for assessing differential impacts is the next step.