ICH-CAHPS: What Signal on the Chadburn?

Abstract

Related Article, p. 753 Related Article, p. 753 In this issue of AJKD, Weidmer and colleagues1Weidmer B.A. Cleary P.D. Keller S. et al.Development and evaluation of the CAHPS (Consumer Assessment of Healthcare Providers and Systems) survey for in-center hemodialysis patients.Am J Kidney Dis. 2014; 64: 753-760Abstract Full Text Full Text PDF Scopus (35) Google Scholar describe developing and testing the In-Center Hemodialysis Consumer Assessment of Healthcare Provider and Systems (ICH-CAHPS) survey. The story begins in 2000, when the US Office of the Inspector General recommended implementing a standardized survey allowing public comparison of dialysis facility patient experience. Two years later, the Centers for Medicare & Medicaid Services (CMS) asked the Agency for Healthcare Research and Quality (AHRQ) to create a standardized CAHPS survey for in-center hemodialysis patients, to guide patient facility selection and stimulate provider quality improvement. In 2007, the National Quality Forum endorsed ICH-CAHPS,2National Quality Forum. CAHPS In-Center Hemodialysis Survey. http://www.qualityforum.org/QPS/0258. Accessed June 4, 2014.Google Scholar and the 2008 ESRD Facility Conditions for Coverage recommended its use.3US Department of Health and Human Services. Centers for Medicare & Medicaid Services. Medicare and Medicaid Programs; Conditions for Coverage for End-Stage Renal Disease Facilities; Final Rule. 42 CFR Parts 405, 410, 413, 414, 488, and 494; 2008.Google Scholar However, widespread use did not occur until its inclusion in the 2014 and 2015 ESRD Quality Incentive Program (QIP) offered 10 bonus points for attesting to administering the ICH-CAHPS. Currently, the 2016 QIP (for calendar year 2014) requires attestation and reporting scores to CMS4US Department of Health and Human Services. Centers for Medicare & Medicaid Services. Medicare Program; End-Stage Renal Disease Prospective Payment System, Quality Incentive Program, and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies, Final Rule. 42 CFR Parts 412 and 4103; 2013.Google Scholar while the recently released 2017 Proposed Rule indicates CMS’s intention to use ICH-CAHPS scores in future QIP equations.5US Department of Health and Human Services. Centers for Medicare & Medicaid Services. Medicare Program; End-Stage Renal Disease Prospective Payment System, Quality Incentive Program, and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies; Proposed Rule. 42 CFR Parts 405, 411m 413, et al; 2014.Google Scholar Fourteen years since inception, there is now increased attention and importance to measuring and improving patient experience. Weidmer and colleagues utilized structured and well-accepted survey development processes to design the 58-item ICH-CAHPS survey to allow: 1) consumers and patients to make comparisons among dialysis facilities; 2) dialysis facilities to benchmark their performance; 3) CMS to monitor facility performance; and 4) facilities to gather information for internal quality improvement purposes.6Crawley B. CAHPS In-Center Hemodialysis (ICH) Survey Overview. https://cahps.ahrq.gov/news-and-events/events/past-events/UGM10/DAY1_d_3_CrawleyOverview.pdf. Accessed August 5, 2014.Google Scholar Convenience samples of patients participating in focus groups in Palo Alto, California and Raleigh-Durham, North Carolina confirmed areas of concern identified by the literature review. The cognitive interview process also used a convenience sample. While Weidmer and colleagues conclude that ICH-CAHPS can compare experience of care at dialysis facilities, they acknowledge that additional assessment is needed for subpopulations not included in their study. Generalizability is a concern given the convenience sampling and is particularly disconcerting for a survey being nationally implemented for public reporting, facility comparison, and determining payment. Once they had developed the survey, Weidmer and colleagues field tested it with 1,454 in-center hemodialysis patients at 32 facilities using 2 administration methods: telephone calls only and mailed surveys with telephone follow-up to those who didn’t return a mail survey. The optional 2012 and 2013 ICH-CAHPS administrations were conducted using similar, AHRQ-defined procedures. In calendar year 2014, CMS began to certify third-party vendors. These vendors will use less robust survey administration procedures than were used during survey development and validation.7Minimum Survey Vendor Business Requirements for the National Implementation of the In-Center Hemodialysis CAHPS (ICH CAHPS) Survey. https://ichcahps.org/ICHCAHPS_VendorMinimumBusinessRequirements.doc. Accessed June 4, 2014.Google Scholar The change in procedures complicates future comparisons with this study. The authors report ICH-CAHPS item-total correlations, a widely used psychometric method to ensure homogeneity of questions that are combined in a scale. Questions belong in a scale if they achieve an item-total correlation of 0.40 or greater. Originally, 9 multi-item scales were hypothesized, but confirmatory factor analysis suggested that the data fit better into 3 or 4 scales. Item-total correlations for 6 of 17 items in the Dialysis Facility Care scale and 4 of the 9 items in the Providing Information to Patients scale were below the accepted cut point of 0.40. Overall, the 3 composite scales (Nephrologist Caring and Concern, Quality of Dialysis Center Care and Operations, and Providing Information to Patients) showed acceptable (α > 0.70) internal consistency reliability, ranging from 0.75 to 0.93. Wood and colleagues8Wood R. Paoli C.J. Hays R.D. Taylor-Stokes G. Piercy J. Gitlin M. Evaluation of the Consumer Assessment of Healthcare Providers and Systems In-Center Hemodialysis survey.Clin J Am Soc Nephrol. 2014; 9: 1099-1108Crossref PubMed Scopus (24) Google Scholar recently identified several item-total correlations less than 0.40, most in the Providing Information to Patients scale, and reported the internal consistency reliability of the Providing Information to Patients scale to be unacceptably low (α = 0.55). The lack of demographic and clinical data in both studies limits comparisons and hypothesis generation to explain the disparate findings. There are many barriers to adopting the ICH-CAHPS survey results as a quality measure. Both Weidmer and colleagues and Wood et al identified items on the ICH-CAHPS survey that do not meet psychometric standards for surveys. Taken together, the results of these 2 studies suggest an opportunity to refine and potentially eliminate items from the ICH-CAHPS survey. Moreover, in its current form, the ICH-CAHPS survey is long and its administration imposes a substantial respondent burden. Patients are the only data source and it is vital to minimize their burden. In addition, mailing cost limits the number of pages over which the survey can be spread; this leads to a small font size that makes the survey inaccessible to patients whose comorbid conditions and/or age reduce their vision. CMS allows for telephone interviews in this circumstance; however, in our experience, hemodialysis patient telephone interviews are fraught with problems. CMS requires that the interview occur while the patient is outside the facility, but acceptable call hours are limited and many don’t answer unless they recognize the number on caller ID. Furthermore, even well-conducted telephone interviews frustrate patients: the necessary scripting is stilted, response choices can be confusing, questions are perceived as repetitive, and calls are lengthy, on average 30 minutes. Although Weidmer and colleagues state that ICH-CAHPS is reliable and valid, other important considerations exist. One purpose of the survey is to compare dialysis facilities. It is only useful as a comparison tool if the characteristics that influence both the propensity to respond and the substance of responses are identified, and if results are adjusted for these characteristics. Published data9Rubin H.R. Fink N.E. Plantinga L.C. Sadler J.H. Kliger A.A. Powe N.R. Patient ratings of dialysis care with peritoneal dialysis vs hemodialysis.JAMA. 2004; 291: 697-703Crossref PubMed Scopus (243) Google Scholar, 10Alexander G.C. Sehgal A.A. Dialysis patient ratings of the quality of medical care.Am J Kidney Dis. 1998; 32: 284-289Abstract Full Text Full Text PDF PubMed Scopus (16) Google Scholar, 11Paddison C.A. Elliot M.N. Haviland A.M. et al.Experiences of care among Medicare beneficiaries with ESRD: Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey results.Am J Kidney Dis. 2013; 61: 440-449Abstract Full Text Full Text PDF PubMed Scopus (44) Google Scholar suggest that adjustment for race will be necessary, but since clinical data are lacking in the studies by Weidmer and colleagues and Wood et al, whether adjustment for other factors will be necessary remains uncertain. CMS is currently sponsoring a “mode experiment” to determine whether responses differ by survey administration method (mail, telephone, or mail with telephone follow-up) or patient characteristics so that appropriate adjustments (ie, case-mix adjustment) can be made to reported data.12Mode Experiment. https://ichcahps.org/GeneralInformation/ModeExperiment.aspx. Accessed June 2, 2014.Google Scholar This experiment could also inform a system to identify and record patients who cannot complete the survey (eg, cognitive impairment, language barriers). Currently, reasons for nonresponse are recorded and used in response rate calculations only if the patient or representative voluntarily identify these on a mailed survey or inform a telephone interviewer. Facilities’ response rates should reflect their patient population; a systematic process would allow for more meaningful comparisons of response rates and results. The mode experiment could inform some of these questions, yet CMS does not intend to make the results of this investigation public.13Centers for Medicare and Medicaid Services. ICH-CAHPS Survey Administration and Specifications Manual. Version 1.2014.Google Scholar Other unanswered questions include characteristics of nonresponders and bias, associations between survey responses and clinical factors, benchmarks, meaning of scores and differences in scores, and most importantly, the relationship to other outcomes of importance. Of note, a well-conducted, cross-sectional study found that Americans who reported the best experience had greater hospitalization use and overall health care expenditures and increased mortality.14Fenton J.J. Jerant A.F. Bertakis K.D. Franks P. The cost of satisfaction: a national study of patient satisfaction, health care utilization, expenditures, and mortality.Arch Intern Med. 2012; 172: 405-411Crossref PubMed Scopus (647) Google Scholar Until studied, we cannot know if these findings apply to hemodialysis patients. In the 2017 proposed rule,5US Department of Health and Human Services. Centers for Medicare & Medicaid Services. Medicare Program; End-Stage Renal Disease Prospective Payment System, Quality Incentive Program, and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies; Proposed Rule. 42 CFR Parts 405, 411m 413, et al; 2014.Google Scholar CMS announced its intention to use ICH-CAHPS results for QIP calculations. When CMS focuses on a measure, facilities work to improve their performance. We anticipate the same outcome with the ICH-CAHPS survey. Improving patient experience with dialysis care is important. But, at this point, there are few validated strategies for improving that experience and a little-known publication by the ICH-CAHPS development team15Using the CAHPS In-Center Hemodialysis survey to improve quality: lessons learned from a demonstration project. Submitted to the Centers for Medicare & Medicaid Services by: American Institutes for Research, RAND, Harvard Medical School, Westat, Network 15.Google Scholar suggests that facilities need to use other surveys and sources to gather more information (which the ICH-CAHPS Survey Administration and Specifications Manual restricts to the point of rendering them of little use); ICH-CAHPS itself is not conducive to successful quality improvement. Weidmer and colleagues conclude the ICH-CAHPS may be used to compare facilities. However, both they and Wood et al state that further evaluation is needed in areas such as validation in subgroups, psychometrics, and clinically meaningful differences. These areas constitute major investigations and suggest that the ICH-CAHPS survey has not yet reached the level of an acceptable quality metric. Patients intuitively understand and care about 2 items on the Dialysis Facility Compare web site (www.medicare.gov/dialysisfacilitycompare): hospitalization and death. Patient experience will be the third. But until we learn more, we are navigating murky waters in dense fog, on a moonless night, guided only by an uncalibrated instrument. As Mr J. Bruce Ismay and Captain Edward Smith learned too late on the Titanic, the signal to proceed full ahead may have unanticipated consequences. Dr Richardson was an invited member of the ESRD CAHPS Technical Expert Panel convened by AHRQ and CMS. Support: None. Financial Disclosure: The authors declare that they have no relevant financial interests. Development and Evaluation of the CAHPS (Consumer Assessment of Healthcare Providers and Systems) Survey for In-Center Hemodialysis PatientsAmerican Journal of Kidney DiseasesVol. 64Issue 5PreviewThe US Centers for Medicare & Medicaid Services assess patient experiences of care as part of the end-stage renal disease prospective payment system and Quality Incentive Program. This article describes the development and evaluation of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) In-Center Hemodialysis Survey. Full-Text PDF