Derivation and implementation of a machine learning approach to prompt serious illness conversations among outpatients with cancer.

Abstract

131 Background: Machine learning (ML) algorithms can accurately identify patients with cancer at risk of short-term mortality and facilitate timely conversations about treatment and end-of-life preferences. We developed, validated, and implemented a ML algorithm to predict mortality in a general oncology setting, using electronic health record (EHR) data prior to a clinic visit. Methods: Our cohort consisted of patients aged ≥18 years who had an encounter in outpatient oncology practices within a large academic health system between February 1st and July 1st, 2016. We randomly split the sample into training (70%) and validation (30%) cohorts at the patient-encounter level. We trained three ML algorithms to predict 180-day mortality and describe performance in the holdout validation cohort. From October 2018 to February 2019, we used the best-performing algorithm to generate weekly lists of high-risk patients at a single community oncology practice and studied the impact on rates of documented serious illness conversations (SICs). Results: Among 62,377 encounters used to train the algorithms, 7.4% involved a patient who died within 180 days. Gradient boosting and/or random forest outperformed logistic regression in all metrics (Table), and the gradient boosting model had superior discrimination and calibration. In the gradient boosting model, observed 180-day mortality was 45.5% (95% CI 39.0-52.3%) in the high-risk group vs. 3.3% (95% CI 2.9-3.7%) in the low-risk group. In a survey of oncology clinicians, 59% of patients flagged as high-risk were appropriate for a serious illness conversation in the upcoming week (response rate 52%). Five months after implementing the intervention, average monthly documented SICs increased by 23% (31.7 to 39). Conclusions: A ML algorithm based on EHR data accurately identified patients with cancer at risk of short-term mortality, was concordant with oncologists’ assessments, and was associated with more SICs. [Table: see text]