The reliability and validity of Bayesian measures for hospital profiling: a Monte Carlo assessment

Abstract

There is a growing trend towards the production of “hospital report-cards” in which hospitals with higher than acceptable mortality rates are identified. Several commentators have advocated for the use of Bayesian methods for health care report cards. Earlier research has demonstrated that there is poor concordance between different Bayesian methods. The current study used Monte Carlo simulation methods to examine the reliability and validity of four different Bayesian measures of hospital performance. Estimates of the reliability of the different measures ranged from a low of 0.89 to a high of 0.99. Estimates of the validity of the four measures ranged from a low of 0.58 to a high of 0.65. Thus, while the four measures of hospital performance demonstrated high reliability, the validity of each method was at most moderate. It is hypothesized that the low validity is due in part to the limited sample sizes that are typically available for hospital report cards.