Two Approaches to Estimation of Classification Accuracy Rate Under Item Response Theory

Abstract

Within the framework of item response theory (IRT), there are two recent lines of work on the estimation of classification accuracy (CA) rate. One approach estimates CA when decisions are made based on total sum scores, the other based on latent trait estimates. The former is referred to as the Lee approach, and the latter, the Rudner approach, each after its representative contributor. In this article, the two approaches are delineated in the same framework to highlight their similarities and differences. In addition, a simulation study manipulating IRT model, sample size, test length, and cut score location was conducted. The study investigated the empirical CA that can be achieved using either the total scores or the latent trait estimates. It also evaluated the performances of the two approaches in estimating their respective empirical CAs. Results on the empirical CA suggest that when the model fits, classifications made with the latent trait estimate shall be equally or more accurate than classifications made with total score. The magnitude of difference was governed by divergence from the one-parameter logistic (1PL) model. Both Lee and Rudner approaches provided good estimates of their respective empirical CAs for every condition that was simulated. Practical implications of the simulation results are discussed.