Applying Rasch Model and Generalizability Theory to Study Modified-Angoff Cut Scores

Abstract

The traditional approach to scale modified-Angoff cut scores transfers the raw cuts to an existing raw-to-scale score conversion table. Under the traditional approach, cut scores and conversion table raw scores are not only seen as interchangeable but also as originating from a common scaling process. In this article, we propose an alternative approach to scale cut scores that first explores the interchangeability between raw cuts and raw scores and then verifies that both sets of scores share a common metric. The two approaches are compared with Rasch measurement model and Generalizability theory using real data from a standard setting study. The two approaches show nontrivial differences in their locations of scaled cut scores, but variability of their scaled cut scores remains comparable. The results illustrate the importance of verifying that panelists’ judgment of item difficulties and empirical item difficulties are on a common metric prior to the scaling of standard setting cuts.