Reading Between the Lines: Understanding How Faculty Interpret Language Cues to Reliably Rank Trainees Using Narrative Comments

Abstract

Background: Narrative comments on in-training evaluation reports (ITERs) can be reliably used to rank-order residents but it is not clear how faculty attendings interpret comments and which language cues are important in this process. Methods: 24 internal medicine (IM) faculty at 1 school each categorized a subgroup of PGY-1 and PGY-2 IM residents based solely on ITER comments, then were interviewed to determine how they made their judgments. Constructivist grounded theory was used to analyze the interviews and develop a framework to understand how ITER language was interpreted. Results: The overarching theme “reading between the lines” explained how participants read and interpreted ITER comments. Scanning for “flags” was part of this strategy. Participants also described specific factors that shaped their judgments: consistency of comments, competency domain, specificity, quantity, and context (evaluator identity, rotation type, and timing). There were several perceived purposes of ITER comments, including feedback to the resident, summative assessment, and other more socially complex uses. Conclusions: Participants made inferences based on what they thought evaluators intended by their comments, and seemed to share an understanding of a “hidden code.” Participants' ability to “read between the lines” explains how comments can be effectively used to categorize and rank-order residents. Our findings suggest that current assumptions about what makes for “good” ITER comments may be incomplete, and variable beliefs about the purpose of ITERs can create challenges for faculty development. Linguistic pragmatics and politeness theories may shed light on why such an implicit code might evolve and be maintained.