16. Reliability of a Case-Based Oral Examination by Case and Competency for Evaluation of Plastic Surgery Residents

Abstract

Background: An oral examination was developed for administration to integrated and independent residents in their final three years of training at multiple Plastic Surgery residency programs. Questions were designed to assess specific skills as outlined in the ACGME six core competencies. Using the combined scores for each case, residents were able to be given feedback on their performance. Our aim is to assess the reliability of our examination by case and by competency. Methods: A unique 8-case oral examination was administered yearly from 2013 to 2019 at multiple Plastic Surgery residency programs. We evaluated one specific examination that was administered in Spring 2019 in 3 states to 38 Plastic Surgery residents from 8 programs. Each item in each case was labeled to target a specific core competency. There was a total of 48 patient care items, 10 medical knowledge items, 5 professionalism items, 4 practice-based learning and improvement items, 4 systems-based practice items, and 1 communication skills item. Reliability was assessed by cronbach’s alpha coefficient calculated for the entire exam, for each unique case, and for each competency. Results: The entire 72-item examination was highly reliable (cronbach’s alpha = 0.91). The cronbach’s alpha coefficients were higher by case than by competency. 7 of the 8 cases were reliable (cronbach’s alpha > 0.70, range 0.45 - 0.82). The competency with the highest reliability was patient care (cronbach’s alpha = 0.87) and the lowest reliability was systems-based practice (cronbach’s alpha = 0.14). Communication skills could not be assessed as there was only one item. Conclusion: The reliability of the examination in total is excellent and it shows promise to be used on a broader scale in a standardized fashion. The reliability of the cases is good and may be used to assess resident knowledge of specific case content. While patient care and medical knowledge are reliably assessed in this exam, further work needs to be done to adequately evaluate for the other 4 core competencies.