66. A NOVEL MILESTONE-BASED SHIFT ASSESSMENT TOOL OF PEDIATRIC EMERGENCY MEDICINE (PEM) FELLOWS

Abstract

Background Feedback is critical in training PEM fellows but often difficult in the emergency department (ED). We identified a need to develop a milestone-based feedback form to assess fellows’ performance after an ED shift. Objectives: Describe the development and implementation of an ED shift assessment tool including the frequency of assessments completed. Methods Using qualitative and quantitative data, a formative assessment tool was developed through a consensus survey completed by PEM faculty. We developed an on-demand feedback form on REDCapTM using the high-scored milestones results from the survey. After a 4-week pilot, the tool was implemented on smartphones from July 1-December 31, 2019. We used convenience sampling with the following steps: 1) fellow chose a milestone to be evaluated and submitted a request to a faculty member for feedback, 2) the faculty received an email in real-time to complete the form. Then, the fellow received an email with relative real-time feedback. Quantitative data was analyzed using descriptive statistics. Qualitative data was analyzed using thematic content analysis and partially applying open axial & selective coding procedures using grounded theory principles. Results Twenty-two (68.8%) faculty completed the consensus survey. Ten milestones scored as “Very Important” for the assessment tool with those related to Patient Care (PC) domain scoring the highest. Barriers to feedback included time management, faculty training, and difficulty in direct fellow observation. Fellows completed 57 assessments, and 17 faculty members provided feedback 46 times. The majority (52.6%) of assessments were completed by 1st year fellows. PC 4 was the most common milestone selected by fellows. Fellows most often self-assessed at an expected level of training, whereas faculty rated more favorably (expected level: 76.1%, above level: 23.9%). Conclusions The shift assessment tool was well-received by PEM faculty and fellows. We plan to investigate the impact of this tool in its practical value in providing formative feedback to our fellows as well as ways to improve this tool. Feedback is critical in training PEM fellows but often difficult in the emergency department (ED). We identified a need to develop a milestone-based feedback form to assess fellows’ performance after an ED shift. Objectives: Describe the development and implementation of an ED shift assessment tool including the frequency of assessments completed. Using qualitative and quantitative data, a formative assessment tool was developed through a consensus survey completed by PEM faculty. We developed an on-demand feedback form on REDCapTM using the high-scored milestones results from the survey. After a 4-week pilot, the tool was implemented on smartphones from July 1-December 31, 2019. We used convenience sampling with the following steps: 1) fellow chose a milestone to be evaluated and submitted a request to a faculty member for feedback, 2) the faculty received an email in real-time to complete the form. Then, the fellow received an email with relative real-time feedback. Quantitative data was analyzed using descriptive statistics. Qualitative data was analyzed using thematic content analysis and partially applying open axial & selective coding procedures using grounded theory principles. Twenty-two (68.8%) faculty completed the consensus survey. Ten milestones scored as “Very Important” for the assessment tool with those related to Patient Care (PC) domain scoring the highest. Barriers to feedback included time management, faculty training, and difficulty in direct fellow observation. Fellows completed 57 assessments, and 17 faculty members provided feedback 46 times. The majority (52.6%) of assessments were completed by 1st year fellows. PC 4 was the most common milestone selected by fellows. Fellows most often self-assessed at an expected level of training, whereas faculty rated more favorably (expected level: 76.1%, above level: 23.9%). The shift assessment tool was well-received by PEM faculty and fellows. We plan to investigate the impact of this tool in its practical value in providing formative feedback to our fellows as well as ways to improve this tool.