How can Health Care Organizations be Reliably Compared?

Abstract

Patient experience is increasingly used to assess organizational performance, for example in public reporting or pay-for-performance schemes. Conventional approaches using 95% confidence intervals are commonly used to determine required survey samples or to report performance but these may result in unreliable organizational comparisons. We analyzed data from 2.2 million patients who responded to the English 2009 General Practice Patient Survey, which included 45 patient experience questions nested within 6 different care domains (access, continuity of care, communication, anticipatory care planning, out-of-hours care, and overall care satisfaction). For each question, unadjusted and case-mix adjusted (for age, sex, and ethnicity) organization-level reliability, and intraclass correlation coefficients were calculated. Mean responses per organization ranged from 23 to 256 for questions evaluating primary care practices, and from 1454 to 2758 for questions evaluating out-of-hours care organizations. Adjusted and unadjusted reliability values were similar. Twenty-six questions had excellent reliability (≥0.90). Seven nurse communication questions had very good reliability (≥0.85), but 3 anticipatory care planning questions had lower reliability (<0.70). Reliability was typically <0.70 for questions with <100 mean responses per practice, usually indicating questions which only a subset of patients were eligible to answer. Nine questions had both excellent reliability and high intraclass correlation coefficients (≥0.10) indicating both reliable measurement and substantial performance variability. High reliability is a necessary property of indicators used to compare health care organizations. Using the English General Practice Patient Survey as a case study, we show how reliability and intraclass correlation coefficients can be used to select measures to support robust organizational comparisons, and to design surveys that will both provide high-quality measurement and optimize survey costs.