Evidence-Based Determination of Cut-Off Points for Increased Cardiac-Surgery Mortality Risk With EuroSCORE II and STS: The Best-Performing Risk Scoring Models in a Single-Centre Australian Population

Abstract

Risk scoring models (RSMs) are commonly used for estimation of postoperative-mortality risk in patients undergoing cardiac surgery, but their prediction accuracy may vary in different populations and clinical situations. The prognostic accuracies of some RSMs have not yet been fully evaluated in the Australian population. In this retrospective observational study, our aims were to assess the performance of four contemporary RSMs, to identify the best RSMs for prediction of postoperative-mortality in the single-centre cohort, and to determine a statistical threshold for classification of patients with increased or "higher" mortality risk. The study population included patients who underwent cardiac surgery at Liverpool Hospital between January 2013 and December 2014. Demographic information was collected, and mortality risks were estimated with the ES2 (EuroSCORE II), STS (Society of Thoracic Surgeons Score), AS (AusSCORE total) and ASMR (AusSCORE multi-risk) RSMs. (Additive EuroSCORE) (AES) and LES (logistic EuroSCORE) were included for historical interest. Discrimination, the ability to stratify patients between mortality and no mortality outcomes, and calibration, the comparison of risk score estimated and observed outcome in the population, were evaluated for each RSM, to determine their predictive accuracy in the study population. Discrimination was assessed by the AUC (area under the receiver operating characteristic curve), and acceptable calibration by the p-value greater than 0.05 for the Hosmer-Lemeshow (H-L) test. The best AUCs in contempory models were compared using the DeLong test. For ES2 and STS risk scores, cut-off points, or thresholds, for patients at increased risk of mortality were derived using Youden's J-statistics, calculated from sensitivity and specificity of models in predicting mortality. From a total study population of 898 patients, 738 had scores for all six RSMs. The three EuroSCORE risk models and Youden's J-statistics analysis included the total population. Of the models in contemporary use, ES2 had higher discrimination (AUC=0.850) in this population than ASMR (AUC=0.767, p=0.024) and AS (AUC=0.739) and non-significantly higher discrimination than STS (AUC=0.806, p=0.19). All contemporary models had acceptable calibration but the older LES (H-L p=0.024) did not. Estimated mortality was closest to observed mortality with the ES2 model. Both AES and LES over predicted mortality. The RSM with the highest discrimination in isolated coronary artery bypass graft surgery (CAGs) (AUC=0.847), isolated valves (AUC=0.830), and females (AUC=0.784) was the ES2 model. STS discrimination was highest in CAGs plus valve procedures (AUC 0.891), and males (STS AUC=0.891). Cut-off points for risk scores to define increased risk populations were 3.0% for ES2 and 1.7% for STS. Similar proportions of patients in each RSM (ES2-26% to STS-32%) were defined as higher risk by the model threshold score depending on type of procedure. Among RSMs in contemporary use, ES2 and STS showed the best discrimination and acceptable calibration. Caution is recommended in specific subgroups. Increased mortality risk score cut-off points could be identified for these two RSMs in this single-centre cohort.