Challenges Of The Clinical Machine Learning Implementation For The Prospective System For High Intensity Evaluation During Radiation Therapy (SHIELD-RT) Study

Abstract

Patients undergoing outpatient cancer radiation therapy (RT) or chemoradiation (CRT) may require evaluation in the emergency department (ED) or hospitalization. We previously developed a machine learning (ML) algorithm utilizing electronic health records (EHR) to predict risk of such events, and this algorithm was used to stratify high risk patients for randomization as part of the recently completed prospective System for High Intensity Evaluation During Radiation Therapy (SHIELD-RT) study. This study demonstrated that ML-directed mandatory supplemental evaluations reduced the rate of acute care visits during RT and CRT. Here we describe the workflow report on the operational challenges encountered while using the oncology information system (OIS) ARIA (Varian Medical Systems, Palo Alto) for clinical implementation of the ML algorithm. In this single institution randomized QI study (NCT04277650), all outpatient adult courses of RT and CRT started from January 7, 2019 to June 30, 2019 were evaluated during their first week of treatment by the ML algorithm. New courses of treatment in the prior week were identified via the OIS, using a preexisting OIS designation of patient appointments as “new start” (new patient beginning new course), “old start” (previously treated patient starting new course) or “final treatment” (either final fraction of multi-fraction treatment or single fraction treatment). To ensure that the ML algorithm was run on newly started courses, “old start” and “final treatment” appointments were checked to verify the appointment corresponded to new treatment courses or to new single fraction treatment courses, respectively. The final patient list was used to query pre-treatment EHR histories and treatment prescription information. Weekly time requirements by the physics staff were documented over the course of the study. Manual review of patient and treatment prescription data from the OIS required on average 4.9 hours per week. Patient appointments labeled “old start” sometimes corresponded to the start of a new treatment plan (i.e. boost plans) during a treatment course that was already underway and previously evaluated for the study. At the time of the treatment start, some sequential boost prescriptions were entered as drafts that required systematic confirmation. Ideally, extraction of patient and radiation therapy treatment plan data would happen on a rolling basis as patients started treatment, but the need for review of the data limited data extraction to once weekly. The workflow for implementation will be presented at the conference. Our study demonstrated a successful clinical implementation of an ML-based algorithm, but manual review of treatment course identification for algorithm assessment was required. Full operationalization beyond the SHIELD-RT study would necessitate changes to data management practices in the OIS centering on identifying new starts.