Robust Small Area Estimation for Generalization

Abstract

Many generalization studies in education are typically based on a sample of 30–70 schools while the inference population is at least twenty times larger. This small sample to population size ratio limits the precision of design-based estimators of the population average treatment effect. Prior work has shown the potential of small area estimation methods to improve generalizations from small samples, specifically within the subclassification framework. However, small area estimation methods are model-based so that the validity of the estimates depends on the model assumptions. In this study, we explore a type of robust small area estimator and assess its performance in settings when core model assumptions are violated. We use a simulation study to compare the robust estimator with a small area estimator that is commonly used in practice and identify the conditions, if any, under which the robust estimator provides improvement. We illustrate the methods using an empirical example and discuss the implications for generalization studies with small samples.