Inefficiency, Subsample Selection Bias, and Nonrobustness

Abstract

This article responds to Peterson and Howell’s (PH’s) defense of their voucher research. We show that PH’s rationales for excluding some 815 students—more than 40% of their sample—because they lacked baseline data in a randomized experiment are unconvincing. For example, PH’s assertion that students with missing baseline scores had a greater risk of mismatch is based on a misunderstanding of the way the experiment was conducted; the organization that conducted the experiment believes the procedures actually used precluded such a problem. Including students with missing baseline data renders the effect of vouchers insignificant for Black students, even using PH’s nonstandard racial classification scheme. PH also give the misleading impression that our reanalysis shows their results to be robust. In fact, only 7 out of the 27 of our estimates that did not simply reuse PH’s original, incomplete sample of Black students are positive and statistically significant.