Design efficiency for imbalanced multilevel data

Abstract

The importance of accurate estimation and of powerful statistical tests is widely recognized but has rarely been acknowledged in practice in the social and behavioral sciences. This is especially true for estimation and testing when one is dealing with multilevel designs, not least because approximating accuracy and power is more complex due to having multiple variances and research units at several levels. The complexity further increases for imbalanced designs, often necessitating simulation studies that perform accuracy and power calculations. However, we show, using such simulation studies, that the distortion of balance can be ignored in most cases, making efficiency studies simpler and the use of existing software valid. An exception is suggested for imbalanced data from a large majority of small groups. Furthermore, an empirical sampling distribution of variance parameters may show substantial skewness and kurtosis, depending on the number of groups and, for the random slope, depending also on the group's size, adding another caveat to the recommendation to ignore imbalance.