Setting Cut-Scores for Complex Performance Assessments: A Critical Examination of the Analytic Judgment Method

Abstract

The purpose of this article is to promote an increased awareness of the processes for setting cut-scores for complex performance assessments by (a) describing the Analytic Judgment Method (AJM) for setting cut-scores, and (b) critically evaluating the technical adequacy and practicability of the AJM by focusing on one investigation where the AJM was used by Plake and Hambleton (2001) for setting standards on the Pennsylvania Grade 8 Mathematics Achievement Test. Although Plake and Hambleton (1998, 2001) demonstrate that the AJM is an attractive iterative procedure that uses independent judgments of actual student work, more research is necessary to replicate the results and determine whether the AJM would produce high interrater reliability with more traditionally sized panels of 20 or more representatives.