Interrater Reliability Reconsidered

Abstract

The purpose of this study was to determine if adequate levels of interrater reliability could be attained on a national standardizedperformance test, using one examiner per observation. Traditionally, performance assessments have relied on two examiners because of the subjectivity involved in grading. However, for nationally administered health-professions performance tests involving process evaluation, cost and logistical issues render two examiners unrealistic. Datafor this study were obtained during a national clinical skills examination for optometry. Surplus examiners were available to serve as secondary examiners in four evaluation stations, yielding 101 paired candidate observations. The results indicated high levels of interrater reliability. At each station, the agreement between examiners in their dichotomous yes-no observations was 90% or greater The agreement among examiner-determined scores based on the differentially weighted items was greater These results indicate that adequate, psychometrically sound examiner judgments and scores can be attained with one examiner per observation.