The Automated Test Assembly and Routing Rule for Multistage Adaptive Testing with Multidimensional Item Response Theory

Abstract

AbstractDesigning a multidimensional adaptive test (M‐MST) based on a multidimensional item response theory (MIRT) model is critical to make full use of the advantages of both MST and MIRT in implementing multidimensional assessments. This study proposed two types of automated test assembly (ATA) algorithms and one set of routing rules that can facilitate the development of an M‐MST. Different M‐MST designs were developed based on the proposed ATA algorithms and routing rules and were evaluated through two sets of simulation studies. Study 1 used simulated item banks and considered a variety of testing factors, and results from which can inform us the theoretical performance of the proposed M‐MST designs. Study 2 was designed based on a real multidimensional assessment. In addition, a MCAT was simulated as a baseline design to demonstrate the advantage of the proposed M‐MST design in each study. Our simulation results indicate that the proposed ATA algorithms and routing rule can generate M‐MSTs with a good‐quality control and the same or even better ability estimation results than the MCAT given the same condition. This demonstrates the advantage of using M‐MST for multidimensional assessment especially for the one that needs to satisfy many nonstatistical constraints.