Ranking institutions

Abstract

Jenkins and Gauvreau1Jenkins KJ Gauvreau K Center-specific differences in mortality: preliminary analyses using the Risk Adjustment in Congenital Heart Disease (RACHS-1) method.J Thorac Cardiovasc Surg. 2002; 124: 97-104Abstract Full Text Full Text PDF PubMed Scopus (195) Google Scholar illustrated the use of a novel risk adjustment method in congenital heart surgery and chose to present their results largely in terms of institutional rankings. However, ranks are a notoriously inaccurate comparator for performance—someone always has to be bottom and top of a league table, no matter how much the play of chance may have contributed to their performance. Figure 1 shows the risk-adjusted standardized mortality ratios (SMRs) with 95% confidence intervals, as ranked by Jenkins and Gauvreau1Jenkins KJ Gauvreau K Center-specific differences in mortality: preliminary analyses using the Risk Adjustment in Congenital Heart Disease (RACHS-1) method.J Thorac Cardiovasc Surg. 2002; 124: 97-104Abstract Full Text Full Text PDF PubMed Scopus (195) Google Scholar according to outcomes from 22 institutions in 1996.We first note that a formal test that all the centers have SMRs of precisely 1 is barely significant (χ2 = 35.6, df = 22, P = .03, after transformation of all values to power 0.3 to bring to approximate normality), so there is not even strong evidence of any heterogeneity among centers. We can also estimate the “true rank” of each center. This requires the methodology described by Marshall and Spiegelhalter,2Marshall EC Spiegelhalter DJ Reliability of league tables of in-vitro fertilisation clinics: retrospective analysis of live birth rates.BMJ. 1998; 317: 1701-1704Crossref PubMed Google Scholar in which the “true SMRs” are repeatedly simulated from the confidence intervals in Figure 1 and then ranked at each iteration of the simulation. The resulting estimated true ranks and their 95% confidence intervals are shown in Figure 2.Fig. 2Estimated true ranks (data points) with 95% confidence intervals (horizontal bars) for 22 centers from Figure 1, showing great uncertainty associated with ranks ascribed to individual institutions.View Large Image Figure ViewerDownload Hi-res image Download (PPT)There is considerable uncertainty about all the centers' true ranks, which naturally arises from the high degree of overlap of the confidence intervals in Figure 1. We can only state with confidence that center E is in the top half (despite being ranked fourth) and center B is in the bottom half; any further attempt at detailed ranking is spurious. Table 1 presents the probabilities that centers near the top or bottom of the league table truly are the best or worst centers.Table 1Probabilities of being “true best” and “true worst” centers for the 8 highest and lowest ranking centersCenterProbability that “true best” centerCenterProbability that “true worst” centerC0.27J0.01D0.16Q0.01H0.20L0.06E0.06A0.15M0.03O0.09F0.11N0.05G0.00P0.27S0.12B0.21 Open table in a new tab No center receives more than 30% chance of being either the winner or loser, although center P turns out most likely to be the worst by a small margin. Such an analysis illustrates the grave dangers of institutional ranking unless there is clear heterogeneity among centers. It also explains why there are generally such radical changes in rankings from year to year when profiling institutions. Presentations that do not emphasize rankings, such as the funnel plots of Stark and colleagues,3Stark J Gallivan S Lovegrove J Hamilton JR Monro JL Pollock JC et al.Mortality rates after surgery for congenital heart defects in children and surgeons' performance.Lancet. 2000; 355: 1004-1007Abstract Full Text Full Text PDF PubMed Scopus (89) Google Scholar are thus to be preferred.