MTMM Matrices Research Articles

A Reevaluation of Assessment Center Construct-Related Validity

Recent Monte Carlo research (Lance, Woehr, & Meade, 2007) has questioned the primary analytical tool used to assess the construct-related validity of assessment center post-exercise dimension ratings (PEDRs) – a confirmatory factor analysis of a multitrait-multimethod (MTMM) matrix. By utilizing a hybrid of Monte Carlo data generation and univariate generalizability theory, we examined three primary sources of variance (i.e., persons, dimensions, and exercises) and their interactions in 23 previously published assessment center MTMM matrices. Overall, the person, dimension, and person by dimension effects accounted for a combined 34.06% of variance in assessment center PEDRs (16.83%, 4.02%, and 13.21%, respectively). However, the largest single effect came from the person by exercise interaction (21.83%). Implications and suggestions for future assessment center research and design are discussed.

The Multitrait-Multimethod Matrix at 50!

Fifty years ago, in 1959, Campbell and Fiske published one of the most influential papers in psychology. In their article Convergent and discriminant validation by the multitraitmultimethod matrix, they argued that it is not sufficient to consider one single operationalization of one construct for purposes of test validation but that multiple measures of multiple constructs are necessary. Campbell and Fiske recommended using at least two methods that are as different as possible for measuring the constructs. Moreover, Campbell and Fiske made clear that it is not possible to get a measure of a trait that is free of method-specific influences. Whenever, in science, we measure a construct (a trait) we have to use a specific measurement method. Therefore, it is the trait and the method that influence the observed score simultaneously. In order to separate methodfrom traitspecific influences, it is thus always necessary to consider more than one trait and more than one method in the validation process. Campbell and Fiske proposed the multitraitmultimethod (MTMM) matrix for analyzing the convergent and discriminant validity. The MTMM matrix consists of the correlations between all multiple measures representing the different traits measured by the different methods. These correlations can be evaluated by several criteria that have been developed by Campbell and Fiske. If the different measures of the same construct are highly correlated, this proves convergent validity. If the different measures of one construct are not correlated with the measures of another construct, this indicates discriminant validity. Campbell and Fiske’s article had and has an enormous influence on psychology (Eid & Diener, 2006). It is the most often cited paper that has ever been published in Psychological Bulletin (Sternberg, 1992). To date, it has been cited 4,735 times (Social Science Citation Index, February 27, 2009, 3:41 pm), and its citation rate is increasing. Their article does not only have an important impact on test validation studies but also has a strong impact on methodological research as many researchers have developed new approaches for analyzing MTMM data and tried to overcome some of the problems and limitations that are related to former approaches of analyzing MTMM matrices. This special issue is dedicated to honoring Campbell and Fiske’s influential work. It presents three different modern approaches for analyzing MTMM data. All contributors use the same data set illustrating their approaches. This enables readers to concentrate on the comparison of the different approaches with respect to the way convergent and discriminant validity can be analyzed as well as how traitand method-specific influences can be identified and quantified. The data consists of three personality traits (extraversion, neuroticism, and conscientiousness) assessed by three raters (one selfand two peer raters). Each scale consists of four items (adjectives such as talkative, conscientious, etc.) that were rated on a five-point scale. The sample size is n = 481. Maas, Lensvelt-Mulders, and Hox show in their contribution A multilevel multitrait-multimethod analysis how multilevel models can be applied to analyze MTMM data. Oort presents an application of Three-mode models for multitrait-multimethod data, and Nussbeck, Eid, Geiser, Courvoisier, and Lischetzke show how the data can be analyzed by A CTC(M-1) model for different types of raters. Finally, Hofling, Schermelleh-Engel, and Mossbrugger compare these approaches in their contribution Analysing multitrait-multimethod (MTMM) data: A comparison of three approaches. We hope that the readers will enjoy seeing how this important field of methodological research has developed over the last years.

Linear Confirmatory Factor Models to Evaluate Multitrait-Multimethod Matrices: The Effects of Number of Indicators and Correlation Among Methods

Two models for confirmatory factor analysis of multitrait-multimethod data (MTMM) were assessed, the correlated traits-correlated methods (CTCM), and the correlated traits-correlated uniqueness models (CTCU). Two Monte Carlo experiments (100 replications per cell) were performed to study the behavior of these models in terms of magnitude and direction of bias, and accuracy of estimates. Study one included a single indicator per trait-method combination, and it manipulated three independent variables: matrix type, from three traits-three methods to six traits-six methods; correlation among method factors, from zero to .6; and model type (CTCM and CTCU). Study two included simulated MTMM matrices with two or more indicators per trait-method combination. Again, three independent variables were manipulated: number of indicators per trait-method combination, from 2 to 5; correlation among methods; and model type, CTCM and CTCU. The results from study one showed that the CTCU model performed very well for MTMM designs with a single indicator per trait-method combination, and consistently better than the CTCM model. However, the results from study two showed that the CTCM model worked reasonably well and better than the CTCU model when more than two indicators per trait-method combination were available. Despite the CTCM model's allowance for correlation between methods, results pointed to better estimates when methods were orthogonal. The main conclusion of the present article is that the use of CTCU models in the situations described in study one and the use of CTCM models in those represented in study two could be recommended.

Msultiplicative Models and MTMM Matrices

The rationale for multiplicative models in the context of MTMM covariance matrices, as developed by Swain and Browne, is described and illustrated with several sets of empirical data. Comparisons with restricted factor analysis models are made, and certain criticisms of this approach are noted. An extension of the multiplicative model to three-facet data that includes occasions, methods, and traits is also presented.

Multiplicative Models and MTMM Matrices

Multiplicative Models and MTMM Matrices

Methodology Review: Analysis of Multitrait-Multimethod Matrices

Procedures for analyzing multitrait-multimethod (MTMM) matrices are reviewed. Confirmatory factor analysis (Jöreskog, 1974) is presented as a general model allowing evaluation of the discriminant and convergent validity of MTMM matrices, both as a whole and in individual trait-method units. However, it is noted that this model is deficient with regard to analysis of trait-method interactions of the type de scribed by Campbell and O'Connell (1967, 1982). Composite direct product models described by Browne (1984) are one possible solution to this problem. Fur ther, more systematic use of hypothesis testing regard ing convergent and discriminant validity in nested hi erarchical models is recommended (Widaman, 1985), as well as the use of a procedure to cross-validate models of MTMM matrices described by Cudeck and Browne (1983).

Assessment of the Multitrait-Multimethod Matrix Validity of Likert Scales Via Confirmatory Factor Analysis.

Two samples of adults were administered nine attitude scales: three constructs (attitudes toward discipline of children, mathematics, and the law) each measured by three methods (Likert, Thurstone and Semantic Differential techniques). For one sample a dichotomous Likert response format was used, in the other a nine-point Likert continuum. Each of the two resulting 9 x 9 MTMM matrices was assessed for evidence of validity using confirmatory factor analysis, supplemented by an estimation of variance components analysis, and the results compared across matrices. The results showed that the three methods converged across the three constructs but that there was an interaction effect such that it did make a difference which method was chosen to measure any particular construct. There was more shared variance among variables in the matrix where the nine-point Likert scales had been used. The results indicated (a) greater validity of Likert scales with the use of a larger number of scale steps, in the sense that total scores more highly estimated the construct underlying the item set, and (b) the value of conceptualizing "types" of measurement techniques which are nevertheless not interchangeable in any particular context.

QUESTIONNAIRE VALIDATION BY CONFIRMATORY FACTOR ANALYSIS: AN IMPROVEMENT OVER MULTITRAIT‐MULTIMETHOD MATRICES*

ABSTRACTThe Campbell and Fiske criteria for assessing the construct validity of multitrait‐multimethod (MTMM) matrices has had a long history of use. While various statistical techniques, including ANOVA, have attempted to provide rigor to the MTMM matrix design, numerous problems still remain unsolved. Part of the problem in using an MTMM matrix is the assumption of measurement independence. This study attempts to illustrate the misleading inferences that often occur from MTMM analysis when method variance overlap is not accurately assessed. Three questionnaires were designed that were not method independent. Traditional procedures for assessing MTMM matrices suggested the three scaling formats used were not burdened with unusual method variance. A reanalysis of the MTMM matrix employing a Confirmatory Factor Analysis technique illustrated that method variance was a problem. Finally, the need for studies that concentrate on the nature of method variance, its causes and effects, is discussed.