Multiple Testing and Heterogeneous Treatment Effects: Re-Evaluating the Effect of PROGRESA on School Enrollment

Abstract

The effect of a program or treatment may vary according to observed characteristics. In such a setting, it may not only be of interest to determine whether the program or treatment has an effect on some sub-population defined by these observed characteristics, but also to determine for which sub-populations, if any, there is an effect. This paper treats this problem as a multiple testing problem in which each null hypothesis in the family of null hypotheses specifies whether the program has an effect on the outcome of interest for a particular sub-population. We develop our methodology in the context of PROGRESA, a large-scale proverty-reduction program in Mexico. In our application, the outcome of interest is the school enrollment rate and the sub-populations are defined by gender and highest grade completed. Under weak assumptions, the testing procedure we construct controls the familywise error rate – the probability of even one false rejection – in finite samples. Similar to earlier studies, we find that the program has a significant effect on the school enrollment rate, but only for a much smaller number of sub-populations when compared to results that do not adjust for multiple testing.