Prospective validation of machine learning-based approaches to predict potentially preventable emergency visits and hospitalizations.

Abstract

404 Background: Patients undergoing cancer treatment are at risk for unplanned acute care. Early identification of at-risk patients could enable preventative interventions, reducing costs and treatment delays. To address this, the Centers for Medicare & Medicaid Services developed the Chemotherapy Measure (OP-35) to monitor potentially preventable acute care utilization during outpatient treatment. We previously developed machine-learning (ML) models using three approaches: least absolute shrinkage selection operator (LASSO), random forest (RF), and gradient boosted trees (GBT). The models predict risk of an OP-35 qualifying acute care event in the 30-days following a systemic therapy infusion and had good performance in the internal validation cohort, with GBT demonstrating the best predictive ability (receiver operating characteristic area under the curve (ROC-AUC) = 0.805). The aim of this study was to prospectively validate these models in patients undergoing systemic therapy at a single institution. Methods: All three models are being prospectively validated on systemic therapy infusions, including chemo, immuno, biologic, hormone, research and targeted therapies. Assuming a 2% event rate based on our prior data, with an alpha = 0.05 and 84% power, to detect an ROC-AUC of 0.75, validation will run for a total of 8000 infusions, from May 1 to August 21, 2023. We present early findings from May 1 to May 14, 2023 for this prospective validation. Model performance was assessed for calibration based on brier score and predictive ability based on ROC-AUC. Sensitivity and specificity were calculated for the model with the highest ROC-AUC based on a previously determined Youden’s J statistic. Results: This study included 1096 systemic therapy treatments across 957 patients. 21 (1.9%) infusions resulted in an OP-35 qualifying acute care event (3 emergency department visits and 18 hospitalizations). Most events were due to pain, anemia, sepsis, and/or dehydration. All models had good performance, with GBT demonstrating the greatest predictive ability (ROC-AUC = 0.78 [0.67 - 0.88], compared to LASSO (0.76 [0.65-0.85]) and RF (0.70 [0.56-0.81]). GBT also had good calibration (brier score = 0.017 [0.011 - 0.025]) followed closely by LASSO (0.018 [0.011-0.026]) and RF (0.019 [0.012-0.026]). Youden-based cut-off of 0.0249 corresponded to a validation sensitivity of 77.6% and specificity of 61.9%. Conclusions: Early prospective validation of ML models demonstrates accurate predictions of OP-35 qualifying acute care events on a per-infusion basis. The use of computational tools to identify patients at risk for unplanned acute care would enable preventative interventions, reducing costs and treatment disruptions. Prospective validation is ongoing and more comprehensive results will be presented at the meeting.