Simplified Notifications Improve Response Rates to Patient Surveys: Findings from the Medicare Disenrollment Survey Experiment

Abstract

Research ObjectiveResponse rates to patient surveys used to evaluate the quality of care have declined in recent years. We assessed several strategies to improve survey response rates, specifically the use of shorter, simpler, more motivational text as recommended by Dillman (2018) (ie, how completing the survey would help other beneficiaries choose plans) in prenotification and cover letters and streamlining the front‐end screener items that assess survey eligibility.Study DesignIn the context of the 2018 Medicare Advantage (MA) Disenrollment Reasons Survey, a monthly survey of disenrollees from MA plans, we implemented a three‐arm randomized experiment (using randomized block design) blocked by MA plan. (1) The control arm (N = 43,277) received the standard survey with prenotification and cover letters. (2) Experimental arm 1 (N=2,484) received the standard survey with modified prenotification and cover letters. (3) Experimental arm 2 (N = 2,484) received the standard survey with streamlined screener questions and modified prenotification and cover letters. We used regression modeling to compare response rates by arm, adjusting for person‐level characteristics (age, gender, dual eligibility for Medicare and Medicaid or low‐income supplement eligibility, and rurality as defined by Beale Code) and fixed effects for MA plan. The disenrollment survey protocol consists of a prenotification letter followed by a mailed survey with cover letter; nonrespondents receive a second mailed survey with cover letter.Population StudiedRandom sample of 48,245 Medicare beneficiaries who voluntarily disenrolled from a Medicare Advantage plan in December 2018 and were mailed a disenrollment reasons survey to complete in February‐March 2019.Principal FindingsAdjusted response rates were 2.3‐2.5 percentage points higher in each experimental arm (33.2% and 33.0%)) than in the standard arm (30.7%). Thus, overall, the experimental arm modifications increased response rates by 2.4 percentage points (95% CI: [1.1‐3.7], (P < .001). Adjusted response rates did not differ between the two experimental arms (P = 0.895). Response rates were lower for beneficiaries <65 years of age, and for dually eligible beneficiaries, but did not differ by gender or rurality.ConclusionsShorter, simpler, and more motivating prenotification and cover letter language, that provided the clear purpose of the survey and value of providing feedback to assist other beneficiaries like themselves in making plan selections, significantly improved response rates in the disenrollment survey at no additional cost. Such modifications may encourage response by reducing the time and complexity for disenrollees to read and understand the survey material. Streamlined screening questions did not further increase response rates, suggesting format or complexity of screening logic may be a less critical factor in response rates than the initial decision to participate based on the initial materials.Implications for Policy or PracticeImprovements to invitational materials to participate in patient surveys can boost response rates at no additional cost, which is important in an environment of declining response rates and limited resources for data collection.Primary Funding SourceCenters for Medicare and Medicaid Services.