The Impacts of Small Teacher-Level Sample Sizes in Cluster-Randomized Trials

Abstract

Cluster-randomized trials have been widely used to evaluate the treatment effects of interventions on student outcomes. When interventions are implemented by teachers, researchers need to account for the nested structure in schools (i.e., students are nested within teachers nested within schools). Schools usually have a very limited number of teachers in each grade. This limited teacher-level sample size may pose potential challenges for study design, analysis, and reporting. The current study aims to evaluate the impacts of small teacher-level sample sizes on study design, analysis, and reporting in cluster-randomized trials. Through a simulation study, we show that the intraclass correlation coefficients at the teacher and school levels are prone to biased estimates in many cases, especially when the student-level sample size is also small. One design strategy we recommend is to sample a large number of students per teacher as this will alleviate the impacts of small teacher-level sample sizes. We recommend that researchers use the total variance, if available, rather than school-level variance to report standardized treatment effects for a meta-analysis. When the total variance is not available for all studies, researchers should use a school-level variance in two-level models as the base for standardization to reduce biases.