Unbalanced 3-Group Split-Ballot Multitrait–Multimethod Design?

Abstract

A common way of estimating measurement quality is the split-ballot multitrait–multimethod (SB-MTMM) approach. However, this approach leads often to non-convergence or improper solutions when using a 2-group design, whereas the 3-group design performs better. Nevertheless, the 3-group design is rarely implemented because it makes it complicated for applied researchers to use the data. Therefore, we propose to draw groups of unequal sample sizes: two larger groups and one third group as small as possible. Using Monte Carlo simulations and real data analyses, we investigate how well such a design works and which size is needed for the third group. Our results suggest that a 3-group SB-MTMM design with smaller size for the third group (reducing till 5–10%) leads to similar levels of accuracy and no large changes in the model or quality estimates.