The effects of attrition on baseline comparability in randomized experiments in education: A meta-analysis.

Abstract

Using meta-analysis, randomized experiments in education that either clearly did or clearly did not experience student attrition were examined for the baseline comparability of groups. Results from 35 studies suggested that after attrition, the observed measures of baseline comparability of groups did not differ more than would be expected given sampling error. The degree of either overall or differential attrition did not relate to baseline comparability, a finding that held under sensitivity analyses. Also, both overall and differential attrition rates were unrelated to posttest effect sizes. All of these analyses, however, lacked sufficient statistical power to detect small but potentially meaningful effects. Results suggest caution is warranted when applying quality scales and other blanket rules pertaining to attrition that are meant to either serve as inclusion-exclusion criteria or in scoring study quality. Much greater attention is needed to both the reporting of attrition in primary studies and to the development of conceptual and empirical models of the attrition process. These developments would aid further investigation of the relation between attrition and study outcomes.