Application Study on Online Multistage Intelligent Adaptive Testing for Cognitive Diagnosis

Abstract

“On-the-fly assembled multistage adaptive Testing (OMST)” provides some unique advantages for both Computerized Adaptive Testing (CAT) and Multistage Testing (MST). In OMST, not one but multiple items are assembled on the fly into one unit in each stage. We apply the idea of OMST to Cognitive Diagnosis CAT (CD-CAT), name it as Online Multistage Intelligent Adaptive Testing (OMIAT), which aims to accurately estimate both examinees’ latent ability level and their knowledge state (KS) simultaneously. A simulation study was conducted to five different item selection methods in CD-CAT: OMIAT method, Shannon Entropy (SHE) method, Aggregate standardized information (ASI) method, Maximum Fisher Information (MFI) method, and random method. The result shows that: (1) both the OMIAT and the ASI methods can not only measure the ability level with precision, but also classify the examinee’s KS with accuracy. In most cases, the OMIAT method is superior to the ASI method in terms of the evaluation criteria, especially when the number of attributes, which is required to respond correctly to the item, is small (<=2). (2) The pattern classification correct rate of the SHE method is always the highest and that of the OMIAT method is always second, but the item exposure rate and the time consumption of the OMIAT method is far superior to those of the SHE method.