Fitting testlets with polytomous IRT models

Abstract

Introduction In Chapter 5, we described three different approaches to the scoring of tests made up of testlets. The aim of that chapter was to illustrate that blindly choosing an off-the-shelf test scoring model may not always be the wisest way to proceed. It is sensible to first consider the goals of the testing program before deciding how to score – one must understand what is the question being asked before trying to figure out what is the best answer. Nevertheless, we have found that a latent trait approach to test scoring is almost always useful and often may be close to optimal. There are two broad avenues that have been frequently used to solve practical testing problems. In this chapter, we will explore one of them, and the rest of this book will be devoted to the other. The first latent trait approach leans heavily on Rosenbaum's theorem of item bundles using a polytomous item response theory (IRT) model to score the locally independent bundles (testlets). The key idea is that the items that form each testlet may have excessive local dependence, but that once the entire testlet is considered as a single unit and scored polytomously (e.g., if it is a sevenitem testlet, the examinee's score for that testlet can range from 0 to 7), these local dependencies may disappear. If the testlets themselves are conditionally independent, they can be scored with a traditional polytomous IRT model.