Measuring Success: Development of a Standardized Outcomes Scorecard for CHEST Live Learning Courses

Abstract

SESSION TITLE: Medical Education SESSION TYPE: Original Investigation Slide PRESENTED ON: Monday, October 30, 2017 at 07:30 AM - 08:30 AM PURPOSE: Measurement of educational outcomes is essential to ensure CHEST course participants receive a high quality training experience and the knowledge, skills, and attitudes that they can implement in their clinical practice. Although CHEST Live Learning (simulation) courses have been popular and effective for more than a decade, evaluation and performance improvement processes have been course specific. The purpose of this project was to develop a standardized scorecard to measure all Live Learning program outcomes and provide actionable feedback to course chairs and faculty, the Education Committee, and CHEST leadership. METHODS: The CHEST Educational Outcomes Subcommittee defined 3 outcome domains: Learners (demand, learner feedback), Quality and Performance (Q&P, instructional design and outcomes measures), and Sustainability. Common goals for each domain were defined using historical attendance data, ACCME and SSIH accreditation standards, and best evidence when available. Data were retrospectively collected for each of the 12 Live Learning courses in 2016. Courses received feedback for each domain as a percentage of goal. A Total Score was also calculated with an emphasis on Q&P (70%). Data were expressed as mean+/-SD. RESULTS: Mean Total Score for all Live Learning courses was high at 96 +/-11 % of goal. For the domain of Q&P, the average score was 101+/-13%, with the highest metrics for individual courses being concentrated in skills testing (8 courses) and pre/post-test improvement (7 courses). In contrast, the lowest scores for individual courses were most commonly observed in the metrics for curriculum mapping (9 courses), pre/post-test improvement (4 courses), and question reliability (4 courses). The lowest Q&P scores were in updating curriculum maps (mean % of goal: 71+/-26) and the most variability was in pre/post-test improvement (mean % of goal: 115+/-69), though the result was always positive. Because scores for the separate Learner’s domain included course attendee evaluation completion rates (mean % of goal: 65 +/-17), this domain was the most variable (mean % of goal: 74 +/- 17) of the 3 domains. CONCLUSIONS: Using a systematically developed, standardized Outcomes Scorecard we found that CHEST Live Learning courses conducted in 2016 scored highly in all three domains of Learners, Quality and Performance, and Sustainability, with most variability in the Learners’ domain. For the domain of Q&P, the highest scores for courses tended to be concentrated in skills testing and pre/post-test improvement while the lowest scores tended to occur in the metrics for curriculum map, pre/post-test improvement, and question reliability. These scorecard results provide an objective assessment of both the many strengths of CHEST programs and actionable areas for ongoing quality improvement. CLINICAL IMPLICATIONS: These Outcomes Scorecard results demonstrate the value and effectiveness of CHEST’s Live Learning courses, and help target resources towards important areas for improvement such as psychometrics of test questions. DISCLOSURE: The following authors have nothing to disclose: Greg Lasko, Robert Ramillosa, Alexander Niven, William Kelly, J Madison No Product/Research Disclosure Information