Relationship Between Narrative Comments and Ratings for Entry-Level Performance on the Clinical Performance Instrument: A Call to Rethink the Clinical Performance Instrument

Abstract

Introduction. Clinical instructors (CIs) use narratives to qualify anchor points for 18 performance criteria on the Clinical Performance Instrument: Version 2006 (CPI'06) when assessing students' performance during clinical experiences. Directors of Clinical Education (DCE) final narratives and anchor points on the CPI'06 to make sure that the narratives and anchor points are in alignment with each other. CIs and DCEs should have a mutual understanding of the terminology used on the CPI'06. The purpose of this study was to determine the level of agreement (concordance) between raters on whether examples of narratives supported the term, “entry-level performance” (ELP), before and after focus group discussions. Methods. Fifty-four CIs discussed the CPI'06 terminology. A pretest/posttest design was used with the intervention being focus group discussions. Participants compared the definition of ELP with sample narratives and agreed, disagreed, or remained undecided about whether the narratives supported ELP. Statistical tests were analyzed to determine participants' concordance about whether the narratives supported ELP. Their rationale to support their decisions was analyzed using qualitative methodology. Results. Participants' percent of pairwise agreement about whether the narratives supported ELP for testing conditions were highest for safety (65.2–95.3%) and lowest for professional behaviors (34.4–44.0%). Intraclass correlation for multiple raters using ICC(2,1) model indicated interrater reliability for absolute agreement on each test, across five performance criteria, was fair to poor. Qualitative assessment for participants' rationale about decisions indicated inconsistencies in the interpretation of CPI'06 terminology. Discussion and Conclusion. Agreement among professionals about the interpretation of the language used on the CPI'06 to ensure that students graduate with the requisite skills and behaviors is essential. This study indicates that there was a lack of participants' concordance with the interpretation of ELP on the CPI'06.