Consumer Rankings and Health Care: Toward Validation and Transparency

Abstract

Differences between the Centers for Medicare & Medicaid Services (CMS)-measured rates of safety events for Rush University Medical Center (RUMC; Chicago) and the U. S. News & World Report (USNWR)-deter mined patient safety score were evaluated in an attempt to validate the USNWR patient safety score-based ranking. The USNWR findings for Patient Safety Indicators (PSIs) were compared with findings derived from RUMC internal billing data, and sensitivity analyses were conducted using a simulated data set derived from the Healthcare Cost and Utilization Project (HCUP) state inpatient data sets. Discrepancies were found for PSIs 3 (Pressure Ulcer Rate), 9 (Perioperative Hemorrhage or Hematoma Rate), and 11 (Postoperative Respiratory Failure Rate)-an excess of 0.72, 0.63, and 0.26 cases/1,000 admissions, in USNWR versus RUMC, respectively). The sensitivity analysis, which included missing present on admission (POA) flags and dates, resulted in an increase of rates by 1.83 (95% confidence interval [CI] = 1.10-2.56) cases/1,000 hospital- izations, 2.72 (CI = 0.00-5.90) cases/1,000 hospitalizations, and 3.89 (CI = 1.60-6.20) cases/1,000 hospitalizations for PSI 3, 9, and 11, respectively. Regression modeling showed that each 1% increase in transfers was associated with an in- crease of 0.06 cases of PSI 3/1,000 admissions; each 1,000 increase in admissions was associated with an increase of 0.04 cases of PSI 9/1,000 admissions. The USNWR data set produced inaccurate PSI rates for RUMC, and false-positive event rates were more common among high-transfer and high-volume hos- pitals. More transparency and validation is needed for con- sumer-based benchmarking methods. In response to these findings and concerns raised by others, in 2016 USNWR made changes to its methodology and data sources and reported them in announcing its 2016-17 Best Hospitals.