Quantifying the influence of partial scalar invariance on mean comparisons: two proposed effect sizes

Abstract

Simulation studies have evaluated methodologies for detecting factor invariance, with the majority of work focused on best practices for identifying non-invariance. Fewer studies have examined the impact of partial factorial invariance on statistical analyses involving scores derived from observed indicator variables, particularly in the context of mean comparisons. In particular, partial scalar invariance can influence group comparisons using observed indicators. Currently, no statistical tools quantify effects of such partial invariance on observed data analyses, leading researchers to make not fully informed decisions regarding the accuracy of obtained results under non-invariant conditions. This study introduces two effect sizes designed to quantify the influence of partial invariance (especially but not exclusively scalar invariance) on group mean comparisons. The effect sizes are based on the DFIT methodology for differential item functioning. Results demonstrate that each effect size is sensitive to a lack of invariance, and can correctly differentiate low from high levels of partial invariance. Interpretation of the results and implications are discussed.