The problem of effect size heterogeneity in meta-analytic structural equation modeling.

Abstract

Scholars increasingly recognize the potential of meta-analytic structural equation modeling (MASEM) as a way to build and test theory (Bergh et al., 2016). Yet, 1 of the greatest challenges facing MASEM researchers is how to incorporate and model meaningful effect size heterogeneity identified in the bivariate meta-analysis into MASEM. Unfortunately, common MASEM approaches in applied psychology (i.e., Viswesvaran & Ones, 1995) fail to account for effect size heterogeneity. This means that MASEM effect sizes, path estimates, and overall fit values may only generalize to a small segment of the population. In this research, we quantify this problem and introduce a set of techniques that retain both the true score relationships and the variability surrounding those relationships in estimating model parameters and fit indices. We report our findings from simulated data as well as from a reanalysis of published MASEM studies. Results demonstrate that both path estimates and overall model fit indices are less representative of the population than existing MASEM research would suggest. We suggest 2 extension MASEM techniques that can be conducted using online software or in R, to quantify the stability of model estimates across the population and allow researchers to better build and test theory. (PsycINFO Database Record