External validation of a predictive model for reintubation after cardiac surgery: A retrospective, observational study

Abstract

Study objectiveExplore validation of a model to predict patients' risk of failing extubation, to help providers make informed, data-driven decisions regarding the optimal timing of extubation. DesignWe performed temporal, geographic, and domain validations of a model for the risk of reintubation after cardiac surgery by assessing its performance on data sets from three academic medical centers, with temporal validation using data from the institution where the model was developed. SettingThree academic medical centers in the United States. PatientsAdult patients arriving in the cardiac intensive care unit with an endotracheal tube in place after cardiac surgery. InterventionsReceiver operating characteristic (ROC) curves and concordance statistics were used as measures of discriminative ability, and calibration curves and Brier scores were used to assess the model's predictive ability. MeasurementsTemporal validation was performed in 1642 patients with a reintubation rate of 4.8%, with the model demonstrating strong discrimination (optimism-corrected c-statistic 0.77) and low predictive error (Brier score 0.044) but poor model precision and recall (Optimal F1 score 0.29). Combined domain and geographic validation were performed in 2041 patients with a reintubation rate of 1.5%. The model displayed solid discriminative ability (optimism-corrected c-statistic = 0.73) and low predictive error (Brier score = 0.0149) but low precision and recall (Optimal F1 score = 0.13). Geographic validation was performed in 2489 patients with a reintubation rate of 1.6%, with the model displaying good discrimination (optimism-corrected c-statistic = 0.71) and predictive error (Brier score = 0.0152) but poor precision and recall (Optimal F1 score = 0.13). Main resultsThe reintubation model displayed strong discriminative ability and low predictive error within each validation cohort. ConclusionsFuture work is needed to explore how to optimize models before local implementation.