Development and validation of the HMS-Sepsis mortality model

Abstract

When comparing outcomes after sepsis, it is essential to account for patient case mix to make fair comparisons. We developed a model to assess risk-adjusted 30-day mortality in the Michigan Hospital Medicine Safety's sepsis initiative (HMS-Sepsis). Can HMS-Sepsis registry data adequately predict risk of 30-day mortality? Do performance assessments using adjusted vs unadjusted data differ? Retrospective cohort of community-onset sepsis hospitalizations in HMS-Sepsis registry (4/2022-9/2023), with split derivation (70%) and validation (30%) cohorts. We fit a risk-adjustment model (HMS-Sepsis mortality model) incorporating acute physiology, demographic, and baseline health data and assessed model performance using c-statistics, Brier's scores, and comparisons of predicted vs observed mortality by deciles of risk. We compared hospital performance (1st quintile, middle quintiles, 5th quintile) using observed versus adjusted mortality to understand the extent to which risk-adjustment impacted hospital performance assessment. Among 17,514 hospitalizations from 66 hospitals during the study period, 12,260 (70%) were used for model derivation and 5,254 (30%) for model validation. 30-day mortality for the total cohort was 19.4%. The final model included 13 physiologic variables, two physiologic interactions, and 16 demographic and chronic health variables. The most significant variables were age, metastatic solid tumor, temperature, altered mental status, and platelet count. The model c-statistic was 0.82 for the derivation cohort, 0.81 for the validation cohort, and ≥0.78 for all subgroups assessed. Overall calibration error was 0.0% and mean calibration error across deciles of risk was 1.5%. Standardized mortality ratios yielded different assessments than observed mortality for 33.9% of hospitals. The HMS-Sepsis mortality model had strong discrimination, adequate calibration, and reclassified one-third of hospitals to a different performance category from unadjusted mortality. Based on its strong performance, the HMS-Sepsis mortality model can aid in fair hospital benchmarking, assessment of temporal changes, and observational causal inference analysis.