Abstract
Saris, Satorra, and Coenders (2004) proposed a new approach to estimate the quality of survey questions, combining the advantages of 2 existing approaches: the multitrait–multimethod (MTMM) and the split-ballot (SB) ones. Implemented in practice, this new approach led to frequent problems of nonconvergence and improper solutions. This article uses Monte Carlo simulations to understand why the SB-MTMM is working well in some cases but not in others. The number of SB groups is a crucial element: The 3-group design is performing better. However, the 2-group design can also perform well: The analyses suggest that the interaction between the absolute values of the correlations between the traits and the relative values of the different correlations between traits plays an important role.
Highlights
The idea of repeating questions using different methods started with Campbell and Fiske (1959, p. 81): “In order to examine discriminant validity, and in order to estimate the relative contributions of trait and method variance, more than one trait as well as more than one method must be employed in the validation process [...] it will be convenient to achieve this through a multitrait-multimethod matrix
- the sample size: are the problems solved if the sample size is large enough? - the closeness of the true values to boundaries: are the Heywood cases (HC) occurring because the true values are close to zero? - the similarities between different true values: are there more non convergence problems because of these similarities?
If the true method variance is .05, the estimated value will more probably be negative than if the true value is .40. Another example is suggested by Saris et al (2004): they mentioned that the 2group design is not empirically identified if there is no correlation between the traits
Summary
The idea of repeating questions using different methods started with Campbell and Fiske (1959, p. 81): “In order to examine discriminant validity, and in order to estimate the relative contributions of trait and method variance, more than one trait as well as more than one method must be employed in the validation process [...] it will be convenient to achieve this through a multitrait-multimethod matrix. When considering three traits and methods, the true score model can be defined by the following system of equations (Saris and Andrews, 1991): Yij = rij Tij + eij for i = 1,2,3 and j = 1,2,3 (1) Tij = vij Fi + mij Mj for i = 1,2,3 and j = 1,2,3 (2). Since at least 20 minutes of similar questions (Van Meurs and Saris, 1990) should separate one question from its repetition in order to avoid memory effects, quite long interviews are needed to implement the MTMM approach Because of these limitations, Saris et al (2004) propose to combine MTMM and split-ballot (SB) designs. In Saris et al ́s words (2004, p.313): “It enables researchers to evaluate measurement reliability and validity, and does so while reducing the response burden.” They suggest that the estimates can be obtained by Maximum Likelihood (ML) estimation for multiple-group (MG) analyses.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
