Assessment of patient communication skills during OSCE: Examining effectiveness of a training program in minimizing inter-grader variability

Abstract

ObjectiveTo assess effectiveness of a training program in reducing inter-grader variability in grading communication skills during an objective structured clinical exam (OSCE). MethodsGlobal communication (GC) skills are assessed by standardized participants (SP) and faculty during each OSCE using a 6 item rubric. Despite criteria delineated in the GC rubric, inter-grader variability was observed. During 2008–2009 academic year, a training program was implemented before each OSCE to achieve more consistent interpretation and grading in GC skills. GC grades between SP and faculty for 2nd and 3rd level student OSCEs during 2008–2009 (post-training) were compared to 2007–2008 (pre-training). Data was analyzed using repeated measures ANOVA. Results274 and 281 students participated in OSCEs during 2007–2008 and 2008–2009 academic years, respectively. Training significantly (P<.001) decreased grader variability between SPs and faculty. There was a greater mean difference between faculty vs. SP before training (faculty 14.68, SP 15.87) compared to after training (faculty 13.51, SP 13.78). Mean GC scores for both faculty and SPs also decreased significantly after training. Conclusion and practice implicationsA training program may be necessary to reduce inter-rater variability in assessment of OSCE communication skills if it is to be truly helpful to student pharmacists learning to counsel patients.