Evaluator Agreement in Medical Student Assessment Across a Multi-Campus Medical School During a Standardized Patient Encounter.

Abstract

Class rank and clerkship grades impact a medical student's residency application. The variability and inter-rater reliability in assessment across multiple clinical sites within a single university system is unknown. We aimed to determine if medical student assessment across medical school campuses is consistent when using a standardized scoring rubric. Attending physicians who participate in assignment of clerkship grades for neurology from three separate clinical campuses of the same medical school observed 10 identical standardized patient encounters completed by third year medical students during the 2017-2018 academic year. Scoring was completed using a standardized rubric. Descriptive analysis and intra-rater comparisons were completed. Evaluations as a part of this study were completed in 2018. Of 50 possible points for the patient encounter, the median score among all medical students and all evaluators was 43 (IQR 40, 45.5). Evaluator number 1 provided a statistically significant lower overall score as compared to evaluators 2 and 3 (p = 0.0001 and p = 0.0006, respectively), who were consistently similar in their overall medical student assessment (p = 0.46). Overall agreement between evaluators was good (ICC = 0.805, 95% CI 0.36-0.95) and consistency was excellent (ICC = 0.91, 95% CI 0.75-0.97). Medical student evaluation across multiple clinical campus sites via observation of identical standardized patient encounters and use of a standardized scoring rubric generally demonstrated good inter-rater agreement and consistency, but the small variation seen may affect overall clerkship scores.