Do method effects generalize across traits (and what if they don't)?

Abstract

Multitrait-multimethod (MTMM) data can be analyzed with single-indicator confirmatory factor analysis (CFA-MTMM) models. Most single-indicator CFA-MTMM models imply-but do not allow testing-the restrictive assumption that method biases generalize (correlate) perfectly across different traits for a given method. To examine the validity of this assumption, we identified and reviewed 20 published applications of multiple-indicator CFA-MTMM models, which allow testing this assumption. Based on simulated data, we demonstrate the consequences of violating the assumption of perfectly general method effects based on the CT-C(M-1) approach. We extracted 111 heterotrait-monomethod method factor correlation estimates, which varied between |.01| and |1.0| (mean=.52) with most correlations being substantially smaller than |1|. The results of our review and simulations show that violations of the assumption of perfectly general method effects (a) are very common, (b) are difficult to detect based on model fit statistics, and (c) can lead to considerable bias in estimates of convergent validity, method specificity, reliability, and method factor correlations in single-indicator models. We recommend that researchers abandon the use of single-indicator CFA-MTMM models and that they use multiple-indicator CFA-MTMM models whenever possible.