Objective Standard Setting for Judge-Mediated Examinations

Abstract

Judge-mediated examinations are defined as those for which expert evaluation (using rubrics) is required to determine correctness, completeness, and reasonability of test-taker responses. The use of multifaceted Rasch modeling has led to improvements in the reliability of scoring such examinations. The establishment of criterion-referenced standards for these assessments has, however, remained a labor-intensive practice. This article explores a method for incorporating the establishment of minimal competency within the measurement model itself. Two high-stakes testing boards participated in a controlled experiment comparing a multifaceted Rasch modeled approach to a traditional modified borderline approach often used for standard setting in judge-mediated situations. Criterion points from each were compared to assess differences and support validity. Results of the experiment suggest that modeling standard setting as part of the performance evaluation is an effective and efficient way to define minimal competence.