Improving Survey Response Rates Research Articles

The Effects of COVID-19 Pandemic Policy on Social Needs Across the State of Kansas and Western Missouri: Paired Survey Response Testing.

Studying patients' social needs is critical to the understanding of health conditions and disparities, and to inform strategies for improving health outcomes. Studies have shown that people of color, low-income families, and those with lower educational attainment experience greater hardships related to social needs. The COVID-19 pandemic represents an event that severely impacted people's social needs. This pandemic was declared by the World Health Organization on March 11, 2020, and contributed to food and housing insecurity, while highlighting weaknesses in the health care system surrounding access to care. To combat these issues, legislators implemented unique policies and procedures to help alleviate worsening social needs throughout the pandemic, which had not previously been exerted to this degree. We believe that improvements related to COVID-19 legislature and policy have positively impacted people's social needs in Kansas and Missouri, United States. In particular, Wyandotte County is of interest as it suffers greatly from issues related to social needs that many of these COVID-19-related policies aimed to improve. The research objective of this study was to evaluate the change in social needs before and after the COVID-19 pandemic declaration based on responses to a survey from The University of Kansas Health System (TUKHS). We further aimed to compare the social needs of respondents from Wyandotte County from those of respondents in other counties in the Kansas City metropolitan area. Social needs survey data from 2016 to 2022 were collected from a 12-question patient-administered survey distributed by TUKHS during a patient visit. This provided a longitudinal data set with 248,582 observations, which was narrowed down into a paired-response data set for 50,441 individuals who had provided at least one response before and after March 11, 2020. These data were then bucketed by county into Cass (Missouri), Clay (Missouri), Jackson (Missouri), Johnson (Kansas), Leavenworth (Kansas), Platte (Missouri), Wyandotte (Kansas), and Other counties, creating groupings with at least 1000 responses in each category. A pre-post composite score was calculated for each individual by adding their coded responses (yes=1, no=0) across the 12 questions. The Stuart-Maxwell marginal homogeneity test was used to compare the pre and post composite scores across all counties. Additionally, McNemar tests were performed to compare responses before and after March 11, 2020, for each of the 12 questions across all counties. Finally, McNemar tests were performed for questions 1, 7, 8, 9, and 10 for each of the bucketed counties. Significance was assessed at P<.05 for all tests. The Stuart-Maxwell test for marginal homogeneity was significant (P<.001), indicating that respondents were overall less likely to identify an unmet social need after the COVID-19 pandemic. McNemar tests for individual questions indicated that after the COVID-19 pandemic, respondents across all counties were less likely to identify unmet social needs related to food availability (odds ratio [OR]=0.4073, P<.001), home utilities (OR=0.4538, P<.001), housing (OR=0.7143, P<.001), safety among cohabitants (OR=0.6148, P<.001), safety in their residential location (OR=0.6172, P<.001), child care (OR=0.7410, P<0.01), health care access (OR=0.3895, P<.001), medication adherence (OR=0.5449, P<.001), health care adherence (OR=0.6378, P<.001), and health care literacy (0.8729, P=.02), and were also less likely to request help with these unmet needs (OR=0.7368, P<.001) compared with prepandemic responses. Responses from individual counties were consistent with the overall results for the most part. Notably, no individual county demonstrated a significant reduction in social needs relating to a lack of companionship. Post-COVID-19 responses showed improvement across almost all social needs-related questions, indicating that the federal policy response possibly had a positive impact on social needs across the populations of Kansas and western Missouri. Some counties were impacted more than others and positive outcomes were not limited to urban counties. The availability of resources, safety net services, access to health care, and educational opportunities could play a role in this change. Future research should focus on improving survey response rates from rural counties to increase their sample size, and to evaluate other explanatory variables such as food pantry access, educational status, employment opportunities, and access to community resources. Government policies should be an area of focused research as they may affect the social needs and health of the individuals considered in this analysis.

Access to Results of Patient Reported Outcome Surveys Does Not Improve Survey Response Rates

IntroductionIn this embedded substudy of a longitudinal, randomized controlled trial, we sought to evaluate the effects of patient engagement and results feedback on longitudinal patient-reported outcome (PRO) survey completion rates. MethodsNewly diagnosed stage 0-III breast cancer patients seen at an academic breast center between June 2019 and December 2021 were invited to participate in a longitudinal PRO study. Participants were emailed the BREAST-Q survey, a validated PRO scale, preoperatively and at regular intervals during their postoperative course. Patients were randomized into the intervention group, who received survey results upon completion, or the control group, who received no feedback. The primary endpoint was postoperative survey completion rate. An intention to treat analysis was performed and a quasi-Poisson regression was used to compare rates of longitudinal survey completion between the two groups. ResultsOf the 253 patients offered the preoperative survey, 115 were in the intervention group and 138 were in the control group. Postoperative survey completion rate was 54% for the intervention group and 47% for the control group. There was no significant difference in longitudinal postoperative survey completion rate between the two groups (rate ratio 1.10; 95% confidence interval [CI] 0.93-1.31). ConclusionsIn this prospective randomized controlled study, patients did not complete surveys at a higher rate when their survey results were shared with them, suggesting that viewing these results without appropriate clinical context does not generate significant enhancement in patient engagement. Effective interventions to improve survey response rate must be identified to better evaluate PROs.

Optimal Timing for Random Digit Dialing

We examine optimal time of day and day of week for conducting random digit dial (RDD) surveys in low- and middle-income countries (LMICs). Different types of survey respondents have competing time demands that influence when they are likely to be able and willing to answer the phone and complete an interview. In this brief, we consider whether there is a best time of day or day of week for improving survey response rates and sample representativeness based on RDD surveys in nine countries. We restrict our analysis to first attempt calls, which function like a randomized experiment. We find that midday calls produce a slightly higher survey completion rate on average than morning calls across the set of nine countries we studied. Evening calls have the lowest survey completion rate. For days of week, there is no evidence of a statistically significant difference in completion rates. We find some evidence that calls earlier in the week have higher contact rates than those made later in the week and that calls made in the evening have lower contact rates than those made earlier in the day. We do not find evidence that the time of day or day of week of the first attempt affects the composition of the sample. It may be meaningfully cost-effective to increase effort during certain time periods when productivity is highest.

Recruitment of women veterans into suicide prevention research: Improving response rates with enhanced recruitment materials and multiple survey modalities

Women veterans (WV) are the fastest growing veteran subpopulation and recognized to be at increased risk for suicide compared with civilians. Improving engagement (e.g., response rates) of WV in survey research is critical to ensuring valid and generalizable findings, which can inform suicide prevention programs tailored for this population. Many factors are known to influence response rates, yet little is known about ways to optimize survey response rates among WV. Three recruitment cohorts (Waves 1 [W1], 2a [W2a], and 2b [W2b]) of WV were invited to participate in an online survey for a national, mixed-methods study examining suicide risk among WV using reproductive health care services paid for or provided by the Veterans Health Administration. To examine the effects of enhanced recruitment efforts, standard recruitment materials were mailed to all three cohorts, with the additions of: a study flyer aiming to build trust between participants and researchers (W2a, W2b) and a paper survey (W2b). Characteristics of responders and non-responders were compared by wave and across survey modalities. Response rates were significantly higher for groups receiving enhanced (W2a = 17.1%; W2b = 24.6%) versus standard (W1 = 12.2%) recruitment materials. WV residing in rural areas were significantly more likely to respond by paper (37.1%) than online (19.8%). Non-respondents were disproportionately racial and ethnic minorities. Disclosure of sensitive information (e.g., military sexual trauma) did not differ by survey modality. Findings suggest that enhanced recruitment materials improve survey response rates among WV, an important consideration for future research with this population.

Simplified Notifications Improve Response Rates to Patient Surveys: Findings from the Medicare Disenrollment Survey Experiment

Research ObjectiveResponse rates to patient surveys used to evaluate the quality of care have declined in recent years. We assessed several strategies to improve survey response rates, specifically the use of shorter, simpler, more motivational text as recommended by Dillman (2018) (ie, how completing the survey would help other beneficiaries choose plans) in prenotification and cover letters and streamlining the front‐end screener items that assess survey eligibility.Study DesignIn the context of the 2018 Medicare Advantage (MA) Disenrollment Reasons Survey, a monthly survey of disenrollees from MA plans, we implemented a three‐arm randomized experiment (using randomized block design) blocked by MA plan. (1) The control arm (N = 43,277) received the standard survey with prenotification and cover letters. (2) Experimental arm 1 (N=2,484) received the standard survey with modified prenotification and cover letters. (3) Experimental arm 2 (N = 2,484) received the standard survey with streamlined screener questions and modified prenotification and cover letters. We used regression modeling to compare response rates by arm, adjusting for person‐level characteristics (age, gender, dual eligibility for Medicare and Medicaid or low‐income supplement eligibility, and rurality as defined by Beale Code) and fixed effects for MA plan. The disenrollment survey protocol consists of a prenotification letter followed by a mailed survey with cover letter; nonrespondents receive a second mailed survey with cover letter.Population StudiedRandom sample of 48,245 Medicare beneficiaries who voluntarily disenrolled from a Medicare Advantage plan in December 2018 and were mailed a disenrollment reasons survey to complete in February‐March 2019.Principal FindingsAdjusted response rates were 2.3‐2.5 percentage points higher in each experimental arm (33.2% and 33.0%)) than in the standard arm (30.7%). Thus, overall, the experimental arm modifications increased response rates by 2.4 percentage points (95% CI: [1.1‐3.7], (P &lt; .001). Adjusted response rates did not differ between the two experimental arms (P = 0.895). Response rates were lower for beneficiaries &lt;65 years of age, and for dually eligible beneficiaries, but did not differ by gender or rurality.ConclusionsShorter, simpler, and more motivating prenotification and cover letter language, that provided the clear purpose of the survey and value of providing feedback to assist other beneficiaries like themselves in making plan selections, significantly improved response rates in the disenrollment survey at no additional cost. Such modifications may encourage response by reducing the time and complexity for disenrollees to read and understand the survey material. Streamlined screening questions did not further increase response rates, suggesting format or complexity of screening logic may be a less critical factor in response rates than the initial decision to participate based on the initial materials.Implications for Policy or PracticeImprovements to invitational materials to participate in patient surveys can boost response rates at no additional cost, which is important in an environment of declining response rates and limited resources for data collection.Primary Funding SourceCenters for Medicare and Medicaid Services.

Improving mail survey response rates in Japan: empirical tests for envelopes, request letters, questionnaires, and schedules

ABSTRACT While many studies examine tools and techniques to improve survey response rates, it remains unclear as to which are effective, and response rates continue to decline. In this study, we examine the effect of four variables expected to influence mail survey response rates: (1) the use of semi-transparent envelopes versus ordinary paper envelopes for delivery, (2) the level of content in the request letter (simple or detailed), (3) the length of the questionnaire, and (4) the survey schedule. We found that the total response rate was 45.4%, while those of the experimental conditions were as follows: (1) semi-transparent versus ordinary paper envelopes, 46.2% versus 44.7%; (2) simple versus detailed request letters, 44.7% versus 46.2%; (3) 4-page versus 8-page questionnaires, 46.0%versus 44.8%; and (4) holiday versus ordinary day, 45.6% versus 45.3%.We found no significant differences in response rates for any of the experimental variables; that is, the logistic regression analysis revealed no relationship between the four experimental conditions and the response rate.

Cost effectiveness of pre- and post-paid incentives for mail survey response

While incentives improve survey response rates, they entail significant costs for the researcher. A mail survey study randomized across three incentive conditions ($2 pre-paid, $10 post-paid, and $20 post-paid), found that households in the $2 pre-paid condition were more likely to respond than the $10 post-paid condition (48.5% vs. 33.5%, β = -0.536, SE = 0.154, p < .001) and the $20 post-paid condition (48.5% vs. 42.4%, β = -0.305, SE = 0.10, p = 0.11). The $2 pre-paid condition was also more cost-effective than the $10 and $20 post-paid conditions (cost-per-complete of $15.35 vs. $26.67 and $34.40, respectively). The results of this study show that a small pre-paid incentive is generally most effective for maximizing both response rates and costs. The data from the study is also used to construct a model to forecast cost-efficiency for variable eligibility conditions, through a factor-adjusted response rate.

Improving Survey Response Rates with Visible Money

AbstractIn mail surveys and in advance letters for surveys in other modes, it is common to include a prepaid incentive of a small amount such as $5. However, when letters are addressed generically (such as to “Resident”), advance letters may be thrown away without being opened, so the enclosed cash is wasted and the invitation or advance letter is ineffective. This research note describes results of an experiment using a nationally representative sample of 4,725 residential addresses to test a new way of letting mail recipients know their letter contains cash and is therefore worth opening: an envelope with a window revealing $5, so the cash is clearly visible from outside the sealed envelope. We also tested the USPS for evidence of theft, and we compared First Class and Priority Mail postage. We found no evidence of theft. We found no difference in response rates between Priority Mail and First Class, making First Class much more cost-effective, and we found that visible money increased the response rate to a mail survey from 42.6 to 46.9 percent, at no significant cost.

Using Response Propensity Modeling to Allocate Noncontingent Incentives in an Address-Based Sample: Evidence from a National Experiment

Abstract Monetary incentives are frequently used to improve survey response rates. While it is common to use a single incentive amount for an entire sample, allowing the incentive to vary inversely with the expected probability of response may help to mitigate nonresponse and/or nonresponse bias. Using data from the 2016 National Household Education Survey (NHES:2016), an address-based sample (ABS) of US households, this article evaluates an experiment in which the noncontingent incentive amount was determined by a household’s predicted response propensity (RP). Households with the lowest RP received $10, those with the highest received $2 or $0, and those in between received the standard NHES incentive of $5. Relative to a uniform $5 protocol, this “tailored” incentive protocol slightly reduced the response rate and had no impact on observable nonresponse bias. These results serve as an important caution to researchers considering the targeting of incentives or other interventions based on predicted RP. While preferable in theory to “one-size-fits-all” approaches, such differential designs may not improve recruitment outcomes without a dramatic increase in the resources devoted to low RP cases. If budget and/or ethical concerns limit the resources that can be devoted to such cases, RP-based targeting could have little practical benefit.

The Use of Text Message Reminders to Increase Completion of Web-Based Surveys: Results of Two Randomized Control Trials

Limited research examines the use of text messaging to improve survey response rates. Drawing on Tailored Panel Management, the Florida Study of Professionals for Safe Families utilized text messages as reminders to complete unfinished web-based surveys, examining if completion varies by (a) message content, and (b) initial level of survey engagement. Overall, message content did not impact completion. However, participants who completed more than half of their survey at the time of a reminder text were more likely to complete their survey than those who had completed less than 50%. We suggest strategies to promote survey completion through text messaging.

Improving Survey Response Rates: The Effect of Embedded Questions in Web Survey Email Invitations

Survey response rate is one of the most critical survey data quality indicators. Many research efforts have been devoted to exploring new ways to improve response rates, especially among web surveys. When inviting participants to web surveys through email, the survey email invitation is the first point of contact made to the survey participants. Several previous studies have examined the impacts of content and design of an email on response rate. However, to our knowledge none of them tested the effect of presenting a survey question in the email. In this study, we report our findings from a web survey experiment focused on embedding the first survey question within the email invitation s . In this condition survey takers are able to see and answer the first question of the survey within the survey email invitation. The results show that as compared to the standard email invite (a link without any survey questions shown), the embedded question email invite improves the email click rate, and survey completion rate, with a small cost to survey drop-out rate. Additionally, the responses to the first question of the survey has shown no difference between the embedded and standard email conditions. The implications of this study and future research directions are also discussed.

Improving survey response rates from parents in school-based research using a multi-level approach.

BackgroundWhile schools can provide a comprehensive sampling frame for community-based studies of children and their families, recruitment is challenging. Multi-level approaches which engage multiple school stakeholders have been recommended but few studies have documented their effects. This paper compares the impact of a standard versus enhanced engagement approach on multiple indicators of recruitment: parent response rates, response times, reminders required and sample characteristics.MethodsParents and teachers were distributed a brief screening questionnaire as a first step for recruitment to a longitudinal study, with two cohorts recruited in consecutive years (cohort 1 2011, cohort 2 2012). For cohort 2, additional engagement strategies included the use of pre-notification postcards, improved study materials, and recruitment progress graphs provided to school staff. Chi-square and t-tests were used to examine cohort differences.ResultsCompared to cohort 1, a higher proportion of cohort 2 parents responded to the survey (76% versus 69%; p < 0.001), consented to participate (71% versus 56%; p < 0.001), agreed to teacher participation (90% versus 82%; p < 0.001) and agreed to follow-up contact (91% versus 80%; p < 0.001). Fewer cohort 2 parents required reminders (52% versus 63%; p < 0.001), and cohort 2 parents responded more promptly than cohort 1 parents (mean difference: 19.4 days, 95% CI: 18.0 to 20.9, p < 0.001).ConclusionThese results illustrate the value of investing in a relatively simple multi-level strategy to maximise parent response rates, and potentially reduce recruitment time and costs.

The Effect of Survey Nonresponse on Quitline Abstinence Rates: Implications for Practice.

Quitline outcome studies are used to maintain and improve the effectiveness of these evidence-based cessation services. Nonresponse has the potential to bias survey results and many US and Canadian quitlines are reporting survey response rates below 50%. This study examines the effect of nonresponse bias on quit rates in three state quitline populations. Results provide implications of nonresponse bias for quitline practice. Quit status, defined as abstinent for 30 days or more 7 months after registering for services, was collected from Minnesota, Hawaii, and Florida quitline participants that responded to a survey. We assigned each responder to a wave based on the number of contacts required to obtain a survey response. The latest two responder groups had the lowest quit rates within each state, although results were not statistically significant. Quit rates in the latest responder wave (Wave 6) were between 4% and 13% points lower than the earliest responders (Wave 1). The cumulative quit rates show what the quit rate would have been had the study ended after the corresponding wave. In all four studies, the cumulative quit rate was lowest in Wave 6. To increase accuracy of quit rates, quitlines should focus on increasing survey response rates. Suggestions for improving survey response rates are provided.

Improving Survey Response Rates in Online Panels

Identifying ways to efficiently maximize the response rate to surveys is important in survey-based research. However, evidence on the response rate effect of donation incentives and especially altruistic and egotistic text appeal interventions is sparse and ambiguous. Via a randomized survey experiment among 6,162 members of an online survey panel, this article shows how low-cost incentives and cost-free text appeal interventions may affect the survey response rate in online panels. The experimental treatments comprise (a) a cash prize lottery incentive, (b) two donation incentives that promise a monetary donation to a good cause in return for survey response, (c) an egotistic text appeal, and (d) an altruistic text appeal. Relative to a control group, we find higher response rates among recipients of the egotistic text appeal and the lottery incentive. Donation incentives yield lower response rates.

Increasing Response Rates to Web-Based Surveys

We review a popular method for collecing data—Web-based surveys. Although Web surveys are popular, one major concern is their typically low response rates. Using the Dillman et al. (2009) approach, we designed, pre-tested, and implemented a survey on climate change with Extension professionals in the Southeast. The Dillman approach worked well, and we generated response rates as high as 79%. However, the method was not problem-free. We share several lessons learned and recommendations for increasing response rates with Web-based surveys and draw attention to the importance of personalized and repeated contact for improving survey response rates.

Assessing the Effectiveness of the Tailored Design Method for Increasing Response Rates of Mail Surveys in Japan

Survey research is commonly conducted in the United States and other Western countries to understand a particular population’s characteristics, behaviors, or opinions; however, few studies have been conducted in Japan to adopt survey methodologies such as the Tailored Design Method developed in the United States. In Japan, mail surveys are generally considered to achieve no more than a 20-40% response rate, and since the mid-1970s, overall response rates in Japanese surveys have declined by about 10 percentage points. Our research is one of the first studies to use the Tailored Design Method in Japan to conduct a mail survey. We compared response rates of our survey about people’s attitudes toward bears and wildlife management to response rates of other mail surveys with similar content to see the effectiveness of the Tailored Design Method. Response rates in two study sites, Nagano, an urban city, and Otari, a rural village were 48.3% and 66.7% respectively, using the Tailored Design Method. The response rate in Otari was significantly higher than those of other rural villages in previous surveys. The response rate for Nagano was comparable to results of rural villages even though response rates in metropolitan areas are usually lower than those of rural areas. The Tailored Design Method may be profitably applied in Japan and other societies in order to improve survey response rates.

The Effect of a College Pen Incentive on Survey Response Rate Among Recent College Graduates

Incentives have shown a variable effect in improving survey response rates, but the effect of a pen from an organization to which the respondent has loyalty has not been studied. Recent college graduates were randomized to receive or not receive a college logo pen accompanying an initial survey mailing. Among 119 total respondents, there were no differences in response rate to the initial mailing, to a second mailing to nonrespondents who did not receive a pen in the initial mailing, or in total response rate. Investigators may save money by not including a pen incentive or may consider stronger incentives.

Assessing the relationship between firm resources and product innovation performance

PurposeThis paper seeks an answer to the question about which of a firm's resources contributes most to product innovation performance (PIP). The paper aims to adopt the resource‐based view (RBV) and consider both tangible and intangible assets.Design/methodology/approachA mail administrated survey was distributed to a randomly selected list of 700 small and medium enterprises (SMEs) in Malaysia. The response rate was 20.1 percent and the usable response rate was 15.4 percent which is favorable for this type of research. As comparable to other SME studies and particularly in Malaysia, this relatively low‐response rate is not surprising for mail administrated questionnaire.FindingsThe findings indicate that, in the Malaysian context, intangible resources are the main drivers of PIP. This is keeping with the expectations of the RBV.Research limitations/implicationsEstablishing a cooperative relationship with the Small and Medium Size Industries Development Corporation of Malaysia plays a significant role in helping us to garner a larger than expected response rate among the SMEs. Alliances such as this may help researchers improve survey response rates among smaller manufacturers.Originality/valueThis paper is unique in that it is the first paper to address the relationship between firm resources and PIP among Malaysian companies. While the paper is primarily descriptive in nature, it does provide some evidence that can be used in the development and testing of hypotheses concerning the relationship between PIP and firm's resources.

Student Engagement: The Key to Improving Survey Response Rates

Over the last decade, universities have paid increasing attention to stakeholder needs, which include the needs and expectations of students. To identify student needs and expectations, tertiary institutions have primarily employed the mechanism of evaluation surveys. In recent years, international research has shown falling survey response rates among students. Monash University, Australia, identified the issue of falling response rates as critical, particularly regarding the graduate destination survey and the course experience questionnaire distributed to Australian university graduates. To address the issue, Monash University adopted a strategy to increase student engagement. The strategy involved setting up a call centre for the purpose of contacting recent Monash University graduates who had not completed the graduate destination survey and course experience questionnaire surveys to explain the significance of the graduate feedback to the institution. This paper highlights the significance of enhancing the engagement of tertiary institutions with students and proposes enhancement of student engagement as one of the more effective and efficient ways of addressing declining student survey response rates.

Improving Survey Response Rates from Chief Executive Officers in Small Firms: The Importance of Social Networks

Social networks are an important source of information for entrepreneurs and small firms. In this study, we consider the influence of social networks on survey response rates from small firms, focusing on effects of trade association endorsement and regional affiliation. Our findings show that trade association endorsement has a positive effect on survey response rate. In addition, the demonstration of the researcher's social ties to the firm's region has a positive effect on survey response rate. Our results lead to several practical implications for survey research on small firms and on industrial populations in general. Targeted personal follow–up with managers in close geographic proximity to the sponsoring university(ies) appears to be a particularly cost–effective strategy to increase response rates.