A validity study of expert judgment procedures for setting cutoff scores on high-stakes credentialing examinations using cluster analysis.

Abstract

This study compares an expert judgment process--minimal performance levels (MPL) using the Nedelsky and Ebel procedures--for setting cutoff scores for pass/fail on licensure examinations with an empirical approach--cluster analysis. Data from all three components of the Canadian Standard Assessment in Optometry (CSAO) examinations (knowledge, clinical judgment, and clinical skills) from 243 candidates were obtained. Results indicate that for the written components of the exams employing the Nedelsky method of MPL setting, there was a mean agreement of pass/fail of 81% with the cluster analysis approach on pass/fail categorization. For the performance exams using the Ebel method, the mean agreement of pass/fail with the cluster analysis was 93%. Thus the subjective approaches to setting cutoff scores (i.e., expert judgment methods) converge with the objective method (i.e., cluster analysis) of classifying test takers in the same categories.