Earned Outcomes Correlate with Reliability-Adjusted Surgical Mortality after Abdominal Aortic Aneurysm Repair and Predict Future Performance

Abstract

Cumulative, probability-based metrics are regularly used to measure quality in professional sports, but these methods have not been applied to health care delivery. These techniques have the potential to be particularly useful in describing surgical quality, where case volume is variable and outcomes tend to be dominated by statistical "noise". The established statistical technique used to adjust for differences in case volume is reliability-adjustment which emphasizes statistical "signal"but has several limitations. We sought to validate a novel measure of surgical quality based on earned outcomes methods (deaths above average, DAA) against reliability-adjusted mortality rates, using abdominal aortic aneurysm (AAA) repair outcomes to illustrate the measure's performance. Earned outcomes methods were used to calculate the outcome of interest for each patient: deaths above average (DAA). Hospital-level DAA were calculated for non-ruptured open AAA repair and EVAR in the Vascular Quality Initiative (VQI) database from 2016-2019. DAA for each center is the sum of observed - predicted risk of death for each patient; predicted risk of death was calculated using established multivariable logistic regression modeling. Correlations of DAA with reliability-adjusted mortality rates and procedure volume were determined. Because an accurate quality metric should correlate with future results, outcomes from 2016-2017 were used to categorize hospital quality based on (1) risk-adjusted mortality, (2) risk- and reliability-adjusted mortality, and (3) DAA. The best performing quality metric was determined by comparing the ability of these categories to predict 2018-2019 risk-adjusted outcomes. During the study period, 3,734 patients underwent open repair (106 hospitals), and 20,680 patients underwent EVAR (183 hospitals). DAA was closely correlated with reliability-adjusted mortality rates for open repair (r=0.94, P<0.001) and EVAR (r=0.99, P<0.001). DAA also correlated with hospital case volume for open repair (r=-.54, P<0.001), but not EVAR (r=0.07, P=0.3). In 2016-2017, most hospitals had 0% mortality (55% open repair, 57% EVAR), making it impossible to evaluate these hospitals using traditional risk-adjusted mortality rates alone. Further, zero mortality hospitals in 2016-2017 did not demonstrate improved outcomes in 2018-2019 for open repair (3.8% vs 4.6%, P=0.5) or EVAR (0.8% vs 1.0%, P=0.2) compared to all other hospitals. In contrast to traditional risk-adjustment, 2016-2017 DAA evenly divided centers into quality quartiles which predicted 2018-2019 performance with increased mortality rate associated with each decrement in quality quartile (Q1 3.2%, Q2 4.0%, Q3 5.1%, Q4 6.0%). There was a significantly higher risk of mortality at worst quartile open repair hospitals compared to best quartile hospitals (OR 2.01, [95% CI 1.07-3.76], P=0.03). Using 2016-2019 DAA to define quality, highest quality quartile open repair hospitals had lower median DAA compared to lowest quality quartile hospitals (-1.18 DAA vs +1.32 DAA, P<0.001), correlating with lower median reliability-adjusted mortality rates (3.6% vs 5.1%, P<0.001). Adjustment for differences in hospital volume is essential when measuring hospital-level outcomes. Earned outcomes accurately categorize hospital quality and correlate with reliability-adjustment but are easier to calculate and interpret. From 2016-2019, highest quality open AAA repair hospitals prevented >40 perioperative deaths compared to the average hospital, and >80 perioperative deaths compared to lowest quality hospitals.