Education Research: A Long-term Faculty Development Initiative Improves Specificity and Usefulness of Narrative Evaluations of Clerkship Students

Abstract

Background and Objectives Narrative-based evaluations are increasingly used to discriminate between levels of trainee performance, yet barriers to high-quality narratives remain. Prior evidence shows mixed results regarding the effectiveness of faculty development efforts on improving narrative evaluation quality. Methods We used a quasi-experimental study incorporating a historical control group to examine the effectiveness of a pragmatic, multipronged, 4-year faculty development initiative on narrative evaluation quality in a neurology clerkship. We evaluated narrative evaluation quality using the narrative evaluation quality instrument (NEQI) in random samples of narrative evaluations from a historical control and intervention group. We used multilevel modeling to compare NEQI scores (and subscale scores) across groups. Informed by the theory of deliberate practice, our faculty development initiative included (1) annual grand rounds sessions focused on developing high-quality narratives and reporting evaluation metrics, (2) restructuring the clerkship assessment form to simplify and prioritize narratives, (3) recruiting key faculty to rotate on the clerkship grading committee to gain experience with and practice developing quality narratives, and (4) instituting a narrative evaluation excellence award to faculty and residents. Results The faculty development initiative was associated with improvements in the quality of students' narrative evaluations. Specifically, the intervention group was a significant predictor of NEQI score, with means of 6.4 (95% CI 5.9–6.9) and 7.6 (95% CI 7.2–8.1) for the historical control and intervention groups, respectively. In addition, the intervention group was associated with significant improvement in the specificity and usefulness NEQI subscale scores, but not the performance domain subscale score. Discussion A long-term, multipronged faculty development initiative can facilitate improvements in narrative evaluation quality. We attribute these findings to 2 factors: (1) pragmatic, solution-oriented efforts that balance focused didactics with programmatic shifts that promote deliberate practice and skill improvement and (2) departmental resources that prioritize and convey a commitment to improving trainee assessment.