Item Parameter Drift in Computer Adaptive Testing Due to Lack of Content Knowledge

Abstract

This study was conducted to analyze potential item parameter drift (IPD) impact on person ability estimates and classification accuracy when drift affects an examinee subgroup. Using a series of simulations, three factors were manipulated: (a) percentage of IPD items in the CAT exam, (b) percentage of examinees affected by IPD, and (c) item pool targeting. IPD impact on ability estimation was evaluated using bias and root mean square error (RMSE). A multivariate general linear model treating three factors as fixed effects and bias and RMSE as outcome variables are used to investigate both main and interaction effects. The impact on classification accuracy was examined based on number and percentages of misclassifications, their significance, and rank order correlations. The findings revealed that IPD exposed to a sub-group of examinees can affect classification accuracy of those examinees substantially, but IPD impact on average ability estimation was small. The study provides useful information to schools, states, and countries planning to implement or are currently implementing CAT as part of their assessments by emphasizing IPD in subgroups may violate fairness and consistency principles of CAT applications.