Use the right words: evaluating the effect of word choice and word count on quality of narrative feedback in ophthalmology competency-based medical education assessments

Abstract

Background: The purpose of this study was to investigate the effect of word choice on the quality of narrative feedback in ophthalmology resident trainee assessments following the introduction of competency-based medical education at Queen’s University. Methods: Assessment data from July 2017-December 2020 were retrieved from ElentraTM (Integrated Teaching and Learning Platform) and anonymized. Written feedback was assigned a Quality of Assessment for Learning (QuAL) score out of five based on this previously validated rubric. The correlation between QuAL score and specific coaching words was determined using a Spearman’s Rho analysis. Independent samples t-tests were used to compare the QuAL score when a specific word was used, and when it was absent. Results: A total of 1997 individual assessments were used in this analysis. The number of times the identified coaching words were used within a comment was significantly and positively associated with the total QuAL score, with the exception of “next time” (rho=0.039, p=0.082), “read” (rho = 0.036, p = 0.112), “read more” (rho = -0.025, p = 0.256) and “review” (rho = -0.017, p = 0.440). The strongest correlations were for “continue” (rho = 0.182, p < 0.001), “try(ing)” (rho = 0.113, p < 0.001) and “next step” (rho = 0.103, p < 0.001). The mean value of the QuAL score increased when coaching words were used vs. not used with the largest mean difference of 1.44 (p < 0.001) for “reflect”. A clear positive relationship was demonstrated between word count and QuAL score (rho = .556, p < 0.001). Conclusions: The use of certain coaching words in written comments may improve the quality of feedback.