Abstract 005: A Comparison of Methods for Identifying Hospital Performance Outliers in Cardiac Surgery

Abstract

Listen

Translate article

Introduction: A number of statistical approaches have been advocated and implemented to estimate adjusted hospital outcomes for public reporting or reimbursement. Nonetheless, the ability of these methods to identify hospital performance outliers in support of quality improvement has not yet been fully investigated. Methods: We leveraged data from patients undergoing coronary artery bypass grafting surgery between 2012-2015 at 33 hospitals participating in a statewide quality collaborative. We applied 5 different statistical approaches (1: indirect standardization with standard logistic regression models, 2: indirect standardization with fixed effect models, 3: indirect standardization with random effect models, 4: direct standardization with fixed effect models, 5: direct standardization with random effect models) to estimate hospital post-operative pneumonia rates adjusting for patients’ risk. Unlike the standard logistic regression models, both fixed effect and random effect models accounted for hospital effect. We applied each method to each year, and subsequently compared methods in their ability to identify hospital performance outliers. Results: Pneumonia rates ranged from 0 % to 24 %. The standard logistic regression models for 2013-2015 had c-statistics of 0.73-0.75, fixed effect models had c-statistics of 0.81-0.83, and random effect models had c-statistics of 0.80-0.83. Each method differed in its ability to identify performance outliers (Figure 1). In direct standardization, random effect models stabilized the hospital rates by moving the estimated rates toward the average rate, fixed effect models produced larger standard errors of hospital effect (particularly for hospitals with low case volumes). In indirect standardization, the three models showed high agreement on their derived observed: expected ratio (intraclass correlation =0.95). Indirect standardization with fixed effect or random effect models, identified similar hospital performance outliers in each year. Conclusion: The five statistical approaches varied in their ability to identify performance outliers. Given its higher sensitivity to outlier hospitals, indirect standardization methods with fixed or random effect models, may be best suited to support quality improvement activities.