Online Calibration in Multidimensional Computerized Adaptive Testing with Polytomously Scored Items

Abstract

AbstractOnline calibration is a key technology for item calibration in computerized adaptive testing (CAT) and has been widely used in various forms of CAT, including unidimensional CAT, multidimensional CAT (MCAT), CAT with polytomously scored items, and cognitive diagnostic CAT. However, as multidimensional and polytomous assessment data become more common, only a few published reports focus on online calibration in MCAT with polytomously scored items (P‐MCAT). Therefore, standing on the shoulders of the existing online calibration methods/designs, this study proposes four new P‐MCAT online calibration methods and two new P‐MCAT online calibration designs and conducts two simulation studies to evaluate their performance under varying conditions (i.e., different calibration sample sizes and correlations between dimensions). Results show that all of the newly proposed methods can accurately recover item parameters, and the adaptive designs outperform the random design in most cases. In the end, this paper provides practical guidance based on simulation results.