64. ASSESSING THE QUALITY OF PEER FEEDBACK AMONGST PEDIATRIC RESIDENTS

Abstract

Background An Accreditation Council for Graduate Medical Education Common Program requirement states that a training program must evaluate resident performance in a timely manner and use multiple evaluators. Peer evaluation is one tool used by many programs to fulfill this requirement. Peer evaluation is critical to the development and training of residents because it allows for the assessment of cognitive and noncognitive skills not necessarily observed by faculty. A literature review showed that effective feedback should be performance-focused, specific, based directly on observable, or objective data, neutral, identify a plan for improvement. The purpose of this study was to evaluate the quality of open-ended feedback provided to pediatric residents by their peers throughout their clinical rotations. Methods At our institution, pediatric residents are asked to complete a peer evaluation form on all residents that they have worked with during their rotation. The residents received no formal training on giving peer feedback. The form is a series of Likert scaled questions and an open-ended feedback section. Based on our literature review, we created a rubric that assessed the degree to which the open-ended comments were specific (low, moderate, or high), actionable (yes or no), and objective (low, moderate, or high). Two blinded reviewers retrospectively analyzed 1368 peer feedback comments received by 26 graduating pediatric residents during their 3-year training period. Results Cohen's kappa for each criterion demonstrated high inter-rater reliability. Our analyses determined that, on average, 59% of the comments were rated moderately or highly specific (?=0.834), 3% were rated actionable (?=0.969), and 92% were rated moderately or highly objective (?=0.935) when evaluated separately against each criterion in the rubric (e.g. specificity alone). When analyzing the overall quality of each comment, less than 5% of comments (?=0.958) were determined to be specific (moderate or high), actionable, AND objective (moderate or high). While a majority of comments were evaluated to be specific and/or objective, we found the greatest disparity in feedback quality was a lack of actionable feedback. Discussion These results indicate residents-in-training need development on how to improve the quality of feedback they provide to their peers, particularly in the area of actionable information. Future efforts should focus on creating and evaluating strategies that elicit higher quality feedback.