Integration of a machine learning model-generated surgical risk score into the EHR and preoperative workflow.

Abstract

11163 Background: We previously developed a machine learning model (MLM) that accurately predicts postoperative complications (POC) for cancer inpatients. As laborious manual entry of patient data is a barrier for risk calculator use, and risk constantly changes depending on clinical status, we sought to integrate our MLM into the electronic health record (EHR) and preoperative workflow as a readily available surgical risk score (SRS) that is generated using real-time data. Here, we report on the integration process and survey results regarding its functionality and impact on patient care. Methods: A MLM that predicts severe POC (CD Grade ≥3) for cancer inpatients undergoing same-hospitalization operations was previously developed using EHR data. To integrate our MLM into the Epic EHR, we developed a real-time infrastructure (RTI) using the cloud-based platform Azure. EHR data is inputted to RTI and processed in real-time to generate model outputs. These outputs are then sent back to the EHR through the Epic Cognitive Computing Platform to generate each patient’s specific SRS. The SRS is dichotomized as “high” or “low” risk based upon a previously determined threshold and is displayed as a “Best Practice Advisory (BPA)” when a surgical case request is entered. In addition to the automated BPA, the SRS is sent to the primary surgeon as an InBasket Case Message. To elicit feedback of our EHR-integrated SRS, a survey was distributed to surgical providers at our institution. Results: After integration of our MLM into the EHR preoperative workflow, 185 surgeries were completed between November 16, 2021 to July 31, 2023. The 30-day severe POC rate was 27.6%, and our MLM had precision of 44.7% and recall of 74.5%, resulting in AUROC of 0.78 and AUPRC of 0.53. 35 surgical providers (40% primary surgeons, 34% surgical fellows, 26% advanced practice providers) from the divisions of surgical oncology, colorectal surgery, and gynecologic oncology completed our survey in July 2022 and July 2023. 28 (80%) of providers stated that they have interacted with the SRS through either BPA or InBasket Case Messages. Of these providers, 21 (75%) strongly agreed or agreed that the SRS was visible at appropriate times, and 18 (64%) strongly agreed or agreed that the SRS contains relevant information to assess surgical risk. 8 (29%) strongly agreed or agreed that the SRS was used to facilitate preoperative discussion. The 2023 survey additionally asked whether the SRS impacted the decision to operate; among providers who had interacted with the SRS, 4 (36%) strongly agreed or agreed with this statement. Conclusions: Integration of a real-time, MLM-generated SRS into the EHR and preoperative workflow can be successfully performed. While our integrated SRS is visible to surgeons and contains risk assessment felt relevant to the operation, the impact of the SRS on clinical care and decision making remains unclear.