An Option-Based Partial Credit Item Response Model

Abstract

Multiple-choice (MC) tests have been criticized for allowing guessing and the failure to credit partial knowledge, and alternative scoring methods and response formats (Ben-Simon et al., Appl Psychol Meas 21:65–88, 1997) have been proposed to address this problem. Modern test theory addresses these issues by using binary item response models (e.g., 3PL) with guessing parameters, or with polytomous IRT models. We propose an option-based partial credit IRT model and a new scoring rule based on a weighted Hamming distance between the option key and the option response vector. The test taker (TT)’s estimated ability is based on information from both correct options and distracters. These modifications reduce the TT’s ability to guess and credit the TT’s partial knowledge. The new model can be tailored to different formats, and some popular IRT models, such as the 2PL and Bock’s nominal model, are special cases of the proposed model. Markov Chain Monte Carlo (MCMC) analysis was used to estimate the model parameters and it provides satisfactory estimates of the model parameters. Simulation studies show that the weighted Hamming distance scores have the highest correlation with TTs’ true abilities, and their distribution is also less skewed than those of the other scores considered.