Sensitivity analyses for clustered data: An illustration from a large-scale clustered randomized controlled trial in education

Abstract

In this paper, we demonstrate the importance of conducting well-thought-out sensitivity analyses for handling clustered data (data in which individuals are grouped into higher order units, such as students in schools) that arise from cluster randomized controlled trials (RCTs). This is particularly relevant given the rise in rigorous impact evaluations that use cluster randomized designs across various fields including education, public health and social welfare. Using data from a recently completed cluster RCT of a school-based teacher professional development program, we demonstrate our use of four commonly applied methods for analyzing clustered data. These methods include: (1) hierarchical linear modeling (HLM); (2) feasible generalized least squares (FGLS); (3) generalized estimating equations (GEE); and (4) ordinary least squares (OLS) regression with cluster-robust (Huber–White) standard errors. We compare our findings across each method, showing how inconsistent results – in terms of both effect sizes and statistical significance – emerged across each method and our analytic approach to resolving such inconsistencies.