Hospital Report Cards Research Articles

Accuracy of Hospital Report Cards Based on Administrative Data

Many of the publicly available health quality report cards are based on administrative data. ICD-9-CM codes in administrative data are not date stamped to distinguish between medical conditions present at the time of hospital admission and complications, which occur after hospital admission. Treating complications as preexisting conditions gives poor-performing hospitals "credit" for their complications and may cause some hospitals that are delivering low-quality care to be misclassified as average- or high-performing hospitals. To determine whether hospital quality assessment based on administrative data is impacted by the inclusion of condition present at admission (CPAA) modifiers in administrative data as a date stamp indicator. Retrospective cohort study based on 648,866 inpatient admissions between 1998 and 2000 for coronary artery bypass graft (CABG) surgery, coronary angioplasty (PTCA), carotid endarterectomy (CEA), abdominal aortic aneurysm (AAA) repair, total hip replacement (THR), acute MI (AMI), and stroke using the California State Inpatient Database which includes CPAA modifiers. Hierarchical logistic regression was used to create separate condition-specific risk adjustment models. For each study population, one model was constructed using only secondary diagnoses present at admission based on the CPAA modifier: "date stamp" model. The second model was constructed using all secondary diagnoses, ignoring the information present in the CPAA modifier: the "no date stamp model." Hospital quality was assessed separately using the "date stamp" and the "no date stamp" risk-adjustment models. Forty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 40 percent of the THR hospitals, and 33 percent of the AMI hospitals identified as low-performance hospitals by the "date stamp" models were not classified as low-performance hospitals by the "no date stamp" models. Fifty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 50 percent of the CEA hospitals, and 36 percent of the AMI hospitals identified as low-performance hospitals by the "no date stamp" models were not identified as low-performance hospitals by the "date stamp" models. The inclusion of the CPAA modifier had a minor impact on hospital quality assessment for AAA repair, stroke, and CEA. This study supports the hypothesis that the use of routine administrative data without date stamp information to construct hospital quality report cards may result in the mis-identification of hospital quality outliers. However, the CPAA modifier will need to be further validated before date stamped administrative data can be used as the basis for health quality report cards.

Do report cards influence hospital choice? The case of kidney transplantation.

The national program to report hospital-level outcomes for transplantation has been in place since 1991, yet it has not been addressed in the existing literature on hospital report cards. We study the impact of reported outcomes on demand at kidney transplant centers. Using a negative binomial regression with hospital fixed effects, we estimate the number of patients choosing each center as a function of reported outcomes. Parameters are identified by the within-hospital variation in outcomes over five successive report cards. We find some evidence that report cards influence younger and college-educated patients, but, overall, report cards do not affect demand.

Administrative Data Feedback for Effective Cardiac Treatment

Hospital report cards are increasingly being implemented for quality improvement despite lack of strong evidence to support their use. To determine whether hospital report cards constructed using linked hospital and prescription administrative databases are effective for improving quality of care for acute myocardial infarction (AMI). The Administrative Data Feedback for Effective Cardiac Treatment (AFFECT) study, a cluster randomized trial. Patients with AMI who were admitted to 76 acute care hospitals in Quebec that treated at least 30 AMI patients per year between April 1, 1999, and March 31, 2003. Hospitals were randomly assigned to receive rapid (immediate; n = 38 hospitals and 2533 patients) or delayed (14 months; n = 38 hospitals and 3142 patients) confidential feedback on quality indicators constructed using administrative data. Quality indicators pertaining to processes of care and outcomes of patients admitted between 4 and 10 months after randomization. The primary indicator was the proportion of elderly survivors of AMI at each study hospital who filled a prescription for a beta-blocker within 30 days after discharge. At follow-up, adjusted prescription rates within 30 days after discharge were similar in the early vs late groups (for beta-blockers, odds ratio [OR], 1.06; 95% confidence interval [CI], 0.82-1.37; for angiotensin-converting enzyme inhibitors, OR, 1.17; 95% CI, 0.90-1.52; for lipid-lowering drugs, OR, 1.14; 95% CI, 0.86-1.50; and for aspirin, OR, 1.05; 95% CI, 0.84-1.33). In addition, adjusted mortality was similar in both groups, as were length of in-hospital stay, physician visits after discharge, waiting times for invasive cardiac procedures, and readmissions for cardiac complications. Feedback based on one-time, confidential report cards constructed using administrative data is not an effective strategy for quality improvement regarding care of patients with AMI. A need exists for further studies to rigorously evaluate the effectiveness of more intensive report card interventions.

Impact of the choice of benchmark on the conclusions of hospital report cards

Abstract Background Hospital report cards for outcomes following acute myocardial infarction (AMI) are being produced with increasing frequency. Implicit in the statistical methods used is the fact that hospitals are being compared with an average hospital. Prior research has demonstrated that institutional characteristics such as a high annual volume of AMI patients and academic status are associated with improved outcomes. This raises the important issue of what is an appropriate benchmark against which hospitals should be compared. The objective of the current study was to determine whether the number of hospitals identified as mortality outliers depended upon the benchmark against which hospitals are compared. Methods We examined all patients discharged with a diagnosis of AMI from 163 Ontario hospitals between April 1, 2000, and March 30, 2001. Logistic regression models that incorporated random provider effects were used to identify hospitals with a mortality rate significantly higher than average. The initial model included only patient characteristics, whereas additional models incorporated both patient and hospital characteristics. Results After adjusting for patient characteristics only, 3 hospitals had significantly higher mortality compared to an average-mortality hospital, while 4 hospitals had significantly lower mortality than an average-mortality hospital. However, after further adjusting for peer group, only 1 hospital was identified as having significantly lower mortality than an average-mortality institution in its peer group. Conclusions The use of peer-group-defined rather than overall benchmarks has a substantial impact on the identification of mortality outliers. The choice of the appropriate benchmark is related to the underlying purpose of the comparison.

Optimal Statistical Decisions for Hospital Report Cards

Hospital report cards provide information designed to help patients and providers to make decisions. The purpose of this study was to place the design of hospital report cards into a decision-theoretic framework. The authors' objectives were 2-fold: 1st, to determine what the choice of significance level implies about the relative value of the different types of misclassifications that can arise. Second, to determine optimal significance levels for specific cost functions describing the relative costs associated with different types of misclassifications. Using a previously published theoretical model for hospital mortality, the authors computed false positive (i.e., falsely classified as providing poor-quality care) and false negative (falsely classified as providing good-quality care) rates. First, they determined the cost functions for false negatives and false positives that are implicitly associated with the use of significance levels of 0.05 and 0.01 for identifying hospitals with higher than average mortality. Second, they determined the levels of statistical significance that should be chosen to minimize predefined cost functions, thus minimizing costs associated with misclassifying hospitals. The lower the statistical significance level required for identifying hospitals with higher than average mortality, the lower the implicit cost of false negatives compared to false positives. For a given significance level, the greater the number of patients treated at each hospital or the greater the proportion of truly poorly performing hospitals, the lower the value of the implicit cost incurred by a false negative compared to that for a false positive. For cost functions that put a high relative penalty on false negatives compared to false positives, the use of significance levels of 0.05 or 0.01 does not result in optimal decisions across expected number of patients treated at each hospital or proportions of truly poor-quality care. Hospital report cards that use significance levels of either 0.05 or 0.01 to identify hospitals that have statistically significantly higher than average mortality make implicit assumptions about cost functions, and the values of the optimal cost function vary across scenarios.

Impact of the choice of benchmark on the conclusions of hospital report cards

Background Hospital report cards for outcomes following acute myocardial infarction (AMI) are being produced with increasing frequency. Implicit in the statistical methods used is the fact that hospitals are being compared with an average hospital. Prior research has demonstrated that institutional characteristics such as a high annual volume of AMI patients and academic status are associated with improved outcomes. This raises the important issue of what is an appropriate benchmark against which hospitals should be compared. The objective of the current study was to determine whether the number of hospitals identified as mortality outliers depended upon the benchmark against which hospitals are compared. Methods We examined all patients discharged with a diagnosis of AMI from 163 Ontario hospitals between April 1, 2000, and March 30, 2001. Logistic regression models that incorporated random provider effects were used to identify hospitals with a mortality rate significantly higher than average. The initial model included only patient characteristics, whereas additional models incorporated both patient and hospital characteristics. Results After adjusting for patient characteristics only, 3 hospitals had significantly higher mortality compared to an average-mortality hospital, while 4 hospitals had significantly lower mortality than an average-mortality hospital. However, after further adjusting for peer group, only 1 hospital was identified as having significantly lower mortality than an average-mortality institution in its peer group. Conclusions The use of peer-group-defined rather than overall benchmarks has a substantial impact on the identification of mortality outliers. The choice of the appropriate benchmark is related to the underlying purpose of the comparison.

Hospital Report Cards: Intent, Impact, and Illusion

The public is currently being bombarded with a growing number of hospital report cards revealing hospital-specific outcome data. The original intent of these report cards was to provide information to the public to enable them to make a more educated choice in selecting health care services and to stimulate provider activities to improve outcomes of care. Serious issues have been raised about the accuracy, validity, meaningfulness, and interpretability of these reports, which has affected both provider and consumer response. With public image and financial implications on the line, hospitals need to be aware of these reports and take the necessary steps to improve outcomes of care.

The reliability and validity of Bayesian measures for hospital profiling: a Monte Carlo assessment

There is a growing trend towards the production of “hospital report-cards” in which hospitals with higher than acceptable mortality rates are identified. Several commentators have advocated for the use of Bayesian methods for health care report cards. Earlier research has demonstrated that there is poor concordance between different Bayesian methods. The current study used Monte Carlo simulation methods to examine the reliability and validity of four different Bayesian measures of hospital performance. Estimates of the reliability of the different measures ranged from a low of 0.89 to a high of 0.99. Estimates of the validity of the four measures ranged from a low of 0.58 to a high of 0.65. Thus, while the four measures of hospital performance demonstrated high reliability, the validity of each method was at most moderate. It is hypothesized that the low validity is due in part to the limited sample sizes that are typically available for hospital report cards.

Mortality trends during a program that publicly reported hospital performance.

It is unclear whether publicly reporting hospitals' risk-adjusted mortality leads to improvements in outcomes. To examine mortality trends during a period (1991-1997) when the Cleveland Health Quality Choice program was operational. Time series. Medicare patients hospitalized with acute myocardial infarction (AMI; n = 10,439), congestive heart failure (CHF; n = 23,505), gastrointestinal hemorrhage (GIH; n = 11,088), chronic obstructive pulmonary disease (COPD; n = 8495), pneumonia (n = 23,719), or stroke (n = 14,293). Risk-adjusted in-hospital mortality, early postdischarge mortality (between discharge and 30 days after admission), and 30-day mortality. Risk-adjusted in-hospital mortality declined significantly for all conditions except stroke and GIH, with absolute declines ranging from -2.1% for COPD to -4.8% for pneumonia. However, the mortality rate in the early postdischarge period rose significantly for all conditions except COPD, with increases ranging from 1.4% for GIH to 3.8% for stroke. As a consequence, the 30-day mortality declined significantly only for CHF (absolute decline 1.4%, 95% CI, -2.5 to -0.1%) and COPD (absolute decline 1.6%, 95% CI, -2.8-0.0%). For stroke, risk-adjusted 30-day mortality actually increased by 4.3% (95% CI, 1.8-7.1%). During Cleveland's experiment with hospital report cards, deaths shifted from in hospital to the period immediately after discharge with little or no net reduction in 30-day mortality for most conditions. Hospital profiling remains an unproven strategy for improving outcomes of care for medical conditions. Using in-hospital mortality rates to monitor trends in outcomes for hospitalized patients may lead to spurious conclusions.

Consumers and hospital report cards: first efforts at definition.

Consumers and hospital report cards: first efforts at definition.

Grading the graders: how hospitals in California and New York perceive and interpret their report cards.

Concerns about quality of care are increasing as hospitals struggle to lower costs. Hospital report cards are controversial, but little is known about their impact. To determine whether recent hospital report cards are viewed more favorably than pioneering federal efforts; whether a report based on clinical data is viewed more favorably than one based on administrative data; and whether attitudes toward report cards are related to hospital characteristics. Mailed survey of chief executives at 374 California hospitals and 31 New York hospitals listed in report cards on myocardial infarction and coronary bypass mortality. Two-hundred-and-seventy-four hospitals in California (73.3% response) and 27 in New York (87.1% response). California hospitals were categorized on ownership, size, occupancy, risk-adjusted mortality, teaching status, patient volume, and surgical capability. Number of hospital units that received or discussed the report card, ratings of its quality, perceptions of its usefulness, and knowledge of its methods. In both states, report cards were widely disseminated within hospitals. The mean quality rating was higher (P = 0.0074) in New York than in California; New York respondents appeared to be more knowledgeable about key methods. One or more hospital characteristics was associated with each outcome measure. Leaders at high-mortality hospitals were especially critical and did not find the report useful, despite limited understanding of its methods. Recent hospital report cards were rated better than pioneering federal efforts. A report based on clinical data was rated better, understood better, and disseminated more often to key staff than one that was based on administrative data. Barriers to constructive use of outcomes data persist, especially at high mortality hospitals.

Assessing Health Care Report Cards

Report cards have become a prominent feature of the health policy landscape (F. Thompson 1994; Epstein 1995; Corrigan and Rogers 1995; Brennan and Berwick 1996). Although the federal government no longer distributes annual hospital report cards, many state governments do. Some state governments also have begun to distribute report cards on health maintenance organizations (HMOs). In addition, one state government releases periodic reports on nursing homes, based on state inspections.

The California Hospital Outcomes Project: How Useful Is California’s Report Card for Quality Improvement?

Hospital report cards have proliferated in the 1990s but remain controversial because risk-adjusted outcomes measures are complex and have uncertain validity. Despite this controversy, little is known about their value and impact. A two-stage survey of hospital leaders in California was undertaken in September 1996 and July 1997 to explore how the 1996 reports and data from the California Hospital Outcomes Project (CHOP) were used to improve organizations' performance. In the first stage, a questionnaire was mailed to the chief executive officer of each hospital in the report. In the second stage, a stratified random sample of the respondents who indicated a willingness to provide further information was interviewed. Thirty-nine interviews were completed, representing 87% yield after replacing informants who failed to return six messages. About three-quarters of the interviewees found some aspect of the CHOP report to be useful, especially for benchmarking performance, improving ICD-9-CM (International Classification of Diseases, 9th Revision, Clinical Modification) coding, and educating physicians about documentation and clinical pathways. The most common criticisms were that the reports were not timely and described death rates without providing practical information about the process of care. Although the 1996 CHOP reports and data were widely disseminated within hospitals, most reported uses did not directly affect the process of care for patients with acute myocardial infarction. This finding reflects two critical weaknesses of the project--nontimely data and lack of information about the process of care. Nevertheless, hospital quality managers recognize that public report cards are here to stay, and some carefully studied their outcomes data to identify areas for improvement.

Assessing quality using administrative data.

Administrative data result from administering health care delivery, enrolling members into health insurance plans, and reimbursing for services. The primary producers of administrative data are the federal government, state governments, and private health care insurers. Although the clinical content of administrative data includes only the demographic characteristics and diagnoses of patients and codes for procedures, these data are often used to evaluate the quality of health care. Administrative data are readily available, are inexpensive to acquire, are computer readable, and typically encompass large populations. They have identified startling practice variations across small geographic areas and-supported research about outcomes of care. Many hospital report cards (which compare patient mortality rates) and physician profiles (which compare resource consumption) are derived from administrative data. However, gaps in clinical information and the billing context compromise the ability to derive valid quality appraisals from administrative data. With some exceptions, administrative data allow limited insight into the quality of processes of care, errors of omission or commission, and the appropriateness of care. In addition, questions about the accuracy and completeness of administrative data abound. Current administrative data are probably most useful as screening tools that highlight areas in which quality should be investigated in greater depth. The growing availability of electronic clinical information will change the nature of administrative data in the future, enhancing opportunities for quality measurement.

Developing a hospital report card to demonstrate value in healthcare.

The Sisters of Charity Health Care Systems, Inc., (SCHCS) developed methods to collect more than 290 data elements from its 15 hospitals to measure community benefit, illness prevention, patient satisfaction, severity of illness, appropriateness, traditional quality measures, outcome measures, maternal/child services, psychiatric services, efficiency, financial performance, and risk management. The data were compiled to produce a report card called the Hospital Quality Profile, which SCHCS hospitals now use for strategic quality planning, assessment, and monitoring. Collaboration between quality management, information systems, and financial department personnel is critical to the success of such a quality database.