Finding the Strength in a Weak Instrument in a Study of Cognitive Outcomes Produced by Catholic High Schools

Abstract

Summary We show that the strength of an instrument is incompletely characterized by the proportion of compliers, and we propose and evaluate new methods that extract more information from certain settings with comparatively few compliers. Specifically, we demonstrate that, for a fixed small proportion of compliers, the presence of an equal number of always-takers and never-takers weakens an instrument, whereas the absence of always-takers or, equivalently, the absence of never-takers strengthens an instrument. In this statement, the strength of an instrument refers to its ability to recognize and reject a false hypothesis about a structural parameter. Equivalently, the strength of an instrument refers to its ability to exclude from a confidence interval a false value of a structural parameter. This ability is measured by the Bahadur efficiency of a test that assumes that the instrument is flawless, or the Bahadur efficiency of a sensitivity analysis that assumes that the instrument may be somewhat biased. When there are few compliers, the outcomes for most people are unaffected by fluctuations in the instrument, so most of the information about the treatment effect is contained in the tail of the distribution of the outcomes. Exploiting this fact, we propose new methods that emphasize the affected portion of the distribution of outcomes, thereby extracting more information from studies with few compliers. Studies of the effects of Catholic high schools on academic test performance have used ‘being Catholic’ as an instrument for ‘attending a Catholic high school’, and the application concerns such a comparison using the US National Educational Longitudinal Study. Most Catholics did not attend Catholic school, so there are few compliers, but it was rare for non-Catholics to attend Catholic school, so there are very few always-takers.