Redefining Populations of Inference for Generalizations from Small Studies

Abstract

Over the past decade, propensity score-based methods have made an important contribution to the improvement of generalizations from educational studies. However, an important limitation is that many studies are based on small sample sizes in which the study comprises 3-5% of the inference population. In response, one suggestion that has been made in practice is to redefine the inference population and identify a smaller subpopulation to which the results are potentially more generalizable. While redefinition is potentially useful in improving bias reduction and precision, the method also entails tradeoffs associated with estimation. Additionally, the resulting redefined population may still be difficult to describe, which can have implications for communication with policymakers. The current study explores the implications of redefinition and proposes a framework to identify and describe redefined subpopulations. For the latter, we combine an approach from observational studies with existing methods for redefinition and assess the advantages and tradeoffs of the proposed framework. We illustrate and assess the tradeoffs of redefinition and of the proposed framework with a simulation study and an empirical example in education.