Evaluating the Substantive Significance of Linear Fixed Effects Regression Results

Abstract

The counterfactuals researchers commonly use to assess the substantive significance of linear fixed effects regression results do not account for the manner in which these models are estimated. Importantly, including fixed effects in a regression means that the model is estimated based on the often narrow within-unit distribution of an independent variable. Despite this, counterfactuals are often motivated with features of an independent variable's overall distribution (e.g., its range or standard deviation). Using simulated data and two case studies, we show this approach has two consequences. First, it inflates the substantive significance of any variables assessed in this way. Second, these counterfactuals are unreliable and exhibit a high degree of model dependence. We recommend instead that researchers assess the substantive significance of fixed effects regression results with counterfactuals based on the within-unit distribution of an independent variable. We provide an R function to implement this recommendation.