Reliability of different grading systems used in evaluating surgical students

Abstract

Inter-rater agreement in assigning grades using five different grading systems was determined. The performance of 16 students in a surgery clerkship was rated by 21 faculty raters using a pass-fail grading system, a pass-fail-honors system, a letter grade system, a number grade scale from 1 to 10, and a number grade scale from 1 to 100. Inter-rater agreement coefficients were used to assess relative and absolute reliabilities, respectively. Both the letter grade and 1 to 10 number grade systems provided good discrimination, had high to moderate reliability, and required only five raters to achieve a mean rating with the commonly recommended reliability of 0.80. Using the letter grade system, however, a majority of raters agreed on a specific grade assignment for 14 of 16 students, in contrast to the 1 to 10 scale, for which this was true for only 4 of 16 students. The results of this reliability study favor the use of a letter grading system.