LO40: Designing for the future: machine learning software in the age of competency-based medical education

Abstract

Introduction/Innovation Concept: Background: Competency based medical education (CBME) is a method of assessing resident performance through standardized tasks and milestones. The Royal College of Physicians and Surgeons of Canada has started phasing in CBME as the preferred training method, but no tool support exists to process this data. Approximately 400 data points are collected per resident per year at McMaster’s Division of Emergency Medicine. This is an unwieldy amount of data to analyze. Objective: Recognizing that collection and analysis of resident data is an important facet to postgraduate medical education, McMaster University began developing a program to provide predictive automated data analysis of resident performance. Methods: To achieve the stated objective, we adapted a design thinking methodology, which emphasizes the importance of human-centered design. By interviewing stakeholders, we collected user requirements and “pain points” that allowed us to build and evaluate multiple prototypes addressing their problems, such as the ability to process data into reports, real-time reporting, and predictive analytics. We solicited feedback from our stakeholders to iteratively refine the prototypes, ensuring that it was user intuitive and met user needs. Curriculum, Tool, or Material: We developed a software platform that collects, aggregates, reports, and has the possibility of analyzing resident data in real time. It also can present performance data via a real-time dashboard. Having automated the report generating process, administrative workload is reduced to a monitoring capacity. Quantitative data on resident performance has been analysed using artificial Neural Network to identify patterns in resident performance. It performs with a sensitivity of 81% and a specificity of 43%, and accurately predict which residents require remedial support 43% of the time. When built into a learning management system, this allows for the provision of additional support to residents-at-risk. Conclusion: Combining machine learning with resident assessment data has allowed us to build a promising predictive model to predict resident outcomes. This gives us the potential to decrease administrative workload and improve data quality by providing real-time performance dashboards and eliminating the redundancies of manual data processing. If scaled, this innovation might assist program directors in determining competency of residents and human resource planning for the healthcare systems at large.