Bayesian Model Selection Methods for Multilevel IRT Models: A Comparison of Five DIC‐Based Indices

Abstract

AbstractModel selection is important in any statistical analysis, and the primary goal is to find the preferred (or most parsimonious) model, based on certain criteria, from a set of candidate models given data. Several recent publications have employed the deviance information criterion (DIC) to do model selection among different forms of multilevel item response theory models (MLIRT). The majority of the practitioners use WinBUGS for implementing MCMC algorithms for MLIRT models, and the default version of DIC provided by WinBUGS focused on the measurement‐level parameters only. The results herein show that this version of DIC is inappropriate. This study introduces five variants of DIC as a model selection index for MLIRT models with dichotomous outcomes. Considering a multilevel IRT model with three levels, five forms of DIC are formed: first‐level conditional DIC computed from the measurement model only, which is the index given by many software packages such as WinBUGS; second‐level marginalized DIC and second‐level joint DIC computed from the second‐level model; and top‐level marginalized DIC and top‐level joint DIC computed from the entire model. We evaluate the performance of the five model selection indices via simulation studies. The manipulated factors include the number of groups, the number of second‐level covariates, the number of top‐level covariates, and the types of measurement models (one‐parameter vs. two‐parameter). Considering the computational viability and interpretability, the second‐level joint DIC is recommended for MLIRT models under our simulated conditions.