Reading between the lines: faculty interpretations of narrative evaluation comments.

Abstract

Narrative comments are used routinely in many forms of rater-based assessment. Interpretation can be difficult as a result of idiosyncratic writing styles and disconnects between literal and intended meanings. Our purpose was to explore how faculty attendings interpret and make sense of the narrative comments on residents' in-training evaluation reports (ITERs) and to determine the language cues that appear to be influential in generating and justifying their interpretations. A group of 24 internal medicine (IM) faculty attendings each categorised a subgroup of postgraduate year 1 (PGY1) and PGY2 IM residents based solely on ITER comments. They were then interviewed to determine how they had made their judgements. Constant comparative techniques from constructivist grounded theory were used to analyse the interviews and develop a framework to help in understanding how ITER language was interpreted. The overarching theme of 'reading between the lines' explained how participants read and interpreted ITER comments. Scanning for 'flags' was part of this strategy. Participants also described specific factors that shaped their judgements, including: consistency of comments; competency domain; specificity; quantity, and context (evaluator identity, rotation type and timing). There were several perceived purposes of ITER comments, including feedback to the resident, summative assessment and other more socially complex objectives. Participants made inferences based on what they thought evaluators intended by their comments and seemed to share an understanding of a 'hidden code'. Participants' ability to 'read between the lines' explains how comments can be effectively used to categorise and rank-order residents. However, it also suggests a mechanism whereby variable interpretations can arise. Our findings suggest that current assumptions about the purpose, value and effectiveness of ITER comments may be incomplete. Linguistic pragmatics and politeness theories may shed light on why such an implicit code might evolve and be maintained in clinical evaluation.