Board #139 - Research Abstract Development of a Multi-Station Assessment of Senior Pediatric Residents (Submission #8630)

Abstract

Hypothesis The Accreditation Council for Graduate Medical Education (ACGME) has encouraged programs to move away from traditional assessment methodologies, such as faculty assessments and In-Training Examination (ITE), and instead incorporate many types of assessment from a toolbox. Objective Structured Clinical Examinations (OSCEs) may incorporate the use of patient simulators or standardized patients or both. While much of the existing OSCE literature relates to evaluation of medical student performance, there is limited work pertaining to assessment of pediatric residents’ clinical skills. To our knowledge, OSCEs have not been used to assess pediatric residents prior to entering their final year of residency. We developed an OSCE using multiple different testing modalities to assess residents finishing their second year of training. This study aims to describe the process involved in developing the assessment program and to evaluate the tools being used to assess resident performance. Methods Five OSCE cases were developed to assess the residents over the course of an afternoon session. Three cases, focusing on apnea, ventricular tachycardia and status epilepticus, focused on medical management and were performed using high-fidelity simulations. A fourth case involved a patient who had a febrile seizure and was designed specifically to look at a resident giving anticipatory guidance to a standardized parent to examine communication skills. A final case was developed with the idea of assessing residents while they aNew South Walesered pages about various patients and was designed to examine their skills communicating with interns, nurses and consults. Results Assessments were completed annually in the spring from 2012-2014 for all pediatric second-year residents. The first year of the assessment in 2012 was used as a pilot year to develop the cases and assessment tools. The second and third years of the assessment allowed for studies to examine the reliability and validity, as well as further adjustments based on these studies. Tools showed good inter-rater reliability, with an intra-class coefficient above 0.75 for all three tools. Using a generalizability study, checklist tools showed good reliability with one rater for the apnea and status epilepticus cases and two raters for the ventricular tachycardia case. The assessment of the febrile seizure case was changed after the first year to a modified Kalamazoo communication tool. Conclusion The first year of this assessment program was focused on developing case scenarios and assessment tools, as well as looking at feasibility of running the assessment. The assessment required a large amount of staffing and facility time and may not be feasible in other institutions without significant financial and institutional support. The checklist tools for the simulation stations are reasonably reliable and valid, and may be used for assessment of pediatric resident resuscitation skills.