Automated Clinical Prognostic Modelling for ICU Triage Prediction of In-Hospital Mortality of COVID-19 Patients: A Retrospective Cohort Study in the Netherlands

Abstract

Background: ICU capacity has been under extreme pressure throughout the COVID-19 pandemic. Prediction models that estimate the risk of in-hospital mortality of COVID-19 patients could support decision making on ICU admittance and treatment. We aimed to assess the predictive performance of automated in-hospital mortality prediction modelling (AutoML) methods to triage COVID-19 patients at ICU admission versus 24 hours after ICU admission. Methods: We conducted an observational study of all COVID-19 patients admitted to Dutch ICUs between February and July 2020. Data was analyzed of 2,690 COVID-19 patients from 70 ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry. The main outcome measure was in-hospital mortality. We asessed model performance (at admission and after 24 hours, respectively) of AutoML versus the more traditional approach with expert-based predictor pre-selection followed by logistic regression. Findings: Predictive performance of the autoML models with variables available at admission shows fair discrimination (average AUROC = 0·75-0·76 (sdev = 0·03), PPV = 0·70-0·76 (sdev = 0·1) at cut-off = 0·3 (the observed mortality rate), and good calibration. This performance is on par with a logistic regression model with selection of patient variables by three experts (average AUROC = 0·78 (sdev = 0·03) and PPV = 0·79 (sdev = 0·2)). The obtained best AutoML model was a Linear Discriminant Analysis model. Extending the models with variables that are available at 24 hours after admission resulted in models with higher predictive performance (average AUROC = 0·77-0·79 (sdev = 0·03) and PPV = 0·79-0·80 (sdev = 0·10-0·17)). Interpretation: Automated clinical prognostic modelling delivers prediction models with fair discriminatory performance, and good calibration and accuracy. The performance of the automated models is shown to be as good as models that were developed using regression analysis with expert-based predictor pre-selection. In the context of the restricted availability of data in an ICU quality registry, extending the models with variables that are available at 24 hours after admission showed small (but significantly) increase in predictive performance. Funding Statement: This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. Declaration of Interests: The authors declare that they have no competing interests. Ethics Approval Statement: The study protocol was reviewed by the Medical Ethics Committee of the Amsterdam Medical Center, the Netherlands. This committee provided a waiver from formal approval (W20_273 # 20.308) and informed consent since this trial does not fall within the scope of the Dutch Medical Research (Human Subjects) Act.