Aberrant distortion of variance components in multilevel models under conflation of level-specific effects.

Abstract

Methodologists have often acknowledged that, in multilevel contexts, level-1 variables may have distinct within-cluster and between-cluster effects. However, a prevailing notion in the literature is that separately estimating these effects is primarily important when there is specific interest in doing so. Consequently, in practice, researchers uninterested in disaggregating these effects (or unaware of their difference) routinely fit models that conflate them. Furthermore, even researchers who properly disaggregate the fixed components in a model (avoid fixed conflation) may still inadvertently and unknowingly conflate the random effects (fail to avoid random conflation). The purpose of this article is to elucidate an unappreciated consequence of such fixed or random conflation, namely, that it can cause systematic distortion in all variance components, yielding uninterpretable variances that adversely affect the entire model. In this article, I provide novel mathematical derivations, simulations, and pedagogical illustrations of such variance distortion, showing how it leads to several aberrant consequences: (1) error variances at level-1 and level-2 can systematically increase (in the population) with the addition of predictors; (2) there can be a large apparent degree of between-cluster random-effect variability in cases in which there is actually no between-cluster outcome variability; (3) R-squared measures of explained variance can be severely biased, uninterpretable, and well below the logical bound of 0; and (4) inference for all fixed components of the model-not just the conflated slopes themselves-can be compromised. I conclude with recommendations for practice, including cautionary notes on interpreting results from prior research that had specified conflated slopes. (PsycInfo Database Record (c) 2022 APA, all rights reserved).