Nonresponse Follow-up Research Articles

Who Gets Lost, and What Difference Does It Make? Mixed Modes, Nonresponse Follow-up Surveys and the Estimation of Turnout

Abstract This article assesses whether—and to what extent—turnout bias in postelection surveys is reduced by adding a short nonresponse follow-up (NRFU) survey to a mixed-mode survey. Specifically, we examine how the NRFU survey influences response propensities across demographic groups and political factors and whether this affects data quality. We use a rich dataset on validated voter turnout data, collected across two different ballots. In addition to the main survey that comprises computer-assisted telephone interviews (CATI) and web respondents, both studies include a short follow-up mail survey for nonrespondents. The results demonstrate that collecting extra information from additional respondents on so-called “central” questions is worth the effort. In both studies, the NRFU survey substantially increases representativeness with respect to sociodemographic and participation variables. In particular, voters and politically active citizens are more accurately represented in the NRFU survey. This tends to result in better estimates of turnout determinants in the final (combined) sample than is seen from CATI/web respondents only. Moreover, the increase in response rate and the decrease in nonresponse bias comes at almost no price in terms of measurement errors. Vote overreporting is only slightly higher in the mail follow-up survey than in the main CATI/web survey.

A Distance Metric for Modeling the Quality of Administrative Records for Use in the 2020 U.S. Census

Abstract The U.S. Census Bureau is conducting research on using administrative records to reduce the cost while maintaining the quality of the 2020 Census Nonresponse Followup (NRFU). Previous census tests have implemented approaches that use predictive models and optimization procedures to identify vacant and occupied housing units using administrative records. This article details a modification to previous approaches, introducing a simple distance metric to define a quality ranking of housing units to enumerate using administrative records. The distance approach is illustrated, assessed, and compared to a previous approach via a retrospective study of the 2010 U.S. Census.

Adaptive Intervention Methodology for Reduction of Respondent Contact Burden in the American Community Survey

Abstract The notion of respondent contact burden in sample surveys is defined, and a multi-stage process to develop policies for curtailing nonresponse follow-up is described with the goal of reducing this burden on prospective survey respondents. The method depends on contact history paradata containing information about contact attempts both for respondents and for sampled nonrespondents. By analysis of past data, policies to stop case follow-up based on control variables measured in paradata can be developed by calculating propensities to respond for paradata-defined subgroups of sampled cases. Competing policies can be assessed by comparing outcomes (lost interviews, numbers of contacts, patterns of reluctant participation, or refusal to participate) as if these stopping policies had been followed in past data. Finally, embedded survey experiments may be used to assess contact-burden reduction policies when these are implemented in the field. The multi-stage method described here abstracts the stages followed in a series of research studies aimed at reducing contact burden in the Computer Assisted Telephone Interview (CATI) and Computer Assisted Personal Interview (CAPI) modes of the American Community Survey (ACS), which culminated in implementation of policy changes in the ACS.

Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments

Abstract The U.S. Census Bureau is investigating adaptive Nonresponse Follow-Up (NRFU) strategies for single unit businesses in the 2017 Economic Census. These collection protocols require a suite of viable alternative procedures that can be implemented. With business surveys, the majority of cognitive research and nonresponse follow-up procedures focus on collection methods that obtain valid response data from the larger businesses, and there is relatively little quantitative or qualitative research for small businesses. Moreover, the contact methods for small businesses are often constrained by budget limitations. Business programs at the U.S. Census Bureau rely on mailed reminder letters and supplemental promotional materials, with options for certified and bulk mailings. To explore the benefits and disadvantages of the proposed alternative nonresponse follow-up procedures for small businesses, we conducted a field experiment embedded in the 2014 Annual Survey of Manufactures, an annual program that has similar data collection procedures and sampling units as the Economic Census. This article describes the study and presents the results, then discusses how the recommended nonresponse follow-up procedures are implemented in an adaptive collection design test presently being conducted in the 2015 Annual Survey of Manufactures.

Comparison of 2010 Census Nonresponse Follow-Up Proxy Responses with Administrative Records Using Census Coverage Measurement Results

Abstract The U.S. Census Bureau is currently conducting research on ways to use administrative records to reduce the cost and improve the quality of the 2020 Census Nonresponse Followup (NRFU) at addresses that do not self-respond electronically or by mail. Previously, when a NRFU enumerator was unable to contact residents at an address, he/she found a knowledgeable person, such as a neighbor or apartment manager, who could provide the census information for the residents. This was called a proxy response. The Census Bureau’s recent advances in merging federal and third-party databases raise the question: Are proxy responses for NRFU addresses more accurate than the administrative records available for the housing unit? Our study attempts to answer this question by comparing the quality of proxy responses and the administrative records for those housing units in the same timeframe using the results of 2010 Census Coverage Measurement (CCM) Program. The assessment of the quality of the proxy responses and the administrative records in the CCM sample of block clusters takes advantage of the extensive fieldwork, processing, and clerical matching conducted for the CCM.

Adjustment for nonresponse with variables from different sources: Bias correction and precision loss, with application to the Swiss European Social Survey 2012

ABSTRACTAdjustment for nonresponse should reduce the nonresponse bias without decreasing the precision of the estimates. Adjustment for nonresponses are commonly based on socio-demographic variables, although these variables may be poorly correlated with response propensities and with variables of interest. Such variables nevertheless have the advantage of being available for all sample units, whether or not they are participating in the survey. Alternatively, adjustment for nonresponse can be obtained from a follow-up survey aimed at sample units which did not participate in the survey and from which the variables are designed to be correlated with response propensities. However, information collected through these follow-up surveys is not available for people in the sample who participated neither in the survey nor in its nonresponse follow-up. These two sets of variables when used in a nonresponse model for the Swiss European Social Survey 2012 differ only slightly with regard to their effect on bias correction and on the precision of estimates. The variables from the follow-up are performing slightly better. In both cases, the adjustment for nonresponse performs poorly.

Development and implementation of a local government survey to measure community supports for healthy eating and active living.

The ability to make healthy choices is influenced by where one lives, works, shops, and plays. Locally enacted policies and standards can influence these surroundings but little is known about the prevalence of such policies and standards that support healthier behaviors. In this paper, we describe the development of a survey questionnaire designed to capture local level policy supports for healthy eating and active living and findings and lessons learned from a 2012 pilot in two states, Minnesota and California, including respondent burden, survey sampling and administration methods, and survey item feasibility issues. A 38-item, web-based, self-administered survey and sampling frame were developed to assess the prevalence of 22 types of healthy eating and active living policies in a representative sample of local governments in the two states. The majority of respondents indicated the survey required minimal effort to complete with half taking < 20 min to complete the survey. A non-response follow-up plan including emails and phone calls was required to achieve a 68% response rate (versus a 37% response rate for email only reminders). Local governments with larger residential populations reported having healthy eating and active living policies and standards more often than smaller governments. Policies that support active living were more common than those that support healthy eating and varied within the two states. The methods we developed are a feasible data collection tool for estimating the prevalence of municipal healthy eating and active living policies and standards at the state and national level.

Adaptive design research for the 2020 Census1

The U.S. Census Bureau is researching and testing new methods to reduce the cost of the 2020 Census while maintaining data quality. One of the most costly components of the 2010 Census was Nonresponse Followup. In this operation, enumerators conducted inperson interviews at housing units that did not return a census questionnaire by mail. For the 2010 Census, enumerators were instructed to visit each of these units up to three times until the case was resolved. Additionally, enumerators were to make up to three contact attempts by telephone. In this paper, we present an overview of current research on determining the number of contact attempts that should be made to nonresponding units with an emphasis on cost containment and improved overall productivity. Rather than the fixed contact strategy employed in the 2010 Census, we consider adaptive approaches that maintain the quality of the data. We present initial results of possible approaches using data from the 2010 Census and discuss the implications of the methods. We also discuss modeling contact probabilities for each hour of the day to support the case management system.

An approach for using administrative records to reduce contacts in the 2020 Decennial Census

For the 2020 Decennial Census, the U.S. Census Bureau is researching how to use administrative record information from government and other sources in place of field visits during the Nonresponse Followup (NRFU) operation. This paper describes an approach for identifying vacant and occupied housing units to be enumerated using administrative records, removing them from the NRFU workload. While the approach allows flexibility in balancing cost and quality, we evaluate one possible scenario via a retrospective study of the 2010 Census.

Effects of Incentive Amount and Type of Web Survey Response Rates

Providing meaningful incentives demonstrates to respondents that researchers understand the competing demands on their time and value their input. The effects of incentives, particularly when prepaid, are strongly established in survey research literature as effective tools for increasing response. However, effectiveness of incentives on web-administered surveys is less clear, and can be impacted by a number of factors, including incentive type and amount, timing, and mode of survey administration. This paper seeks to contribute to research on incentives for web-based surveys by examining the effects of the following: An additional post-incentive for early completion A pre-paid incentive in combination with a post-paid incentive Pre-incentives as a nonresponse follow-up strategy This paper uses data collected during the first wave of a nationally-representative survey of public school principals designed to take place across three waves of data collection. To determine a maximally-effective incentive strategy for subsequent years, we embedded an experiment into the study using electronic gift cards. All sample members were eligible for a standard $50 post-response incentive, but were also randomly assigned to one of four experimental conditions: (1) an additional $50 incentive for completing early in the field period, (2) a $25 incentive pre-paid with the initial survey mailing, (3) a $25 pre-paid incentive used during nonresponse follow-up, and (4) the standard $50 post response incentive, which serves as the control group for this experiment.

Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland

All social surveys suffer from different types of errors, of which one of the most studied is non-response bias. Non-response bias is a systematic error that occurs because individuals differ in their accessibility and propensity to participate in a survey according to their own characteristics as well as those from the survey itself. The extent of the problem heavily depends on the correlation between response mechanisms and key survey variables. However, non-response bias is difficult to measure or to correct for due to the lack of relevant data about the whole target population or sample. In this paper, non-response follow-up surveys are considered as a possible source of information about non-respondents. Non-response follow-ups, however, suffer from two methodological issues: they themselves operate through a response mechanism that can cause potential non-response bias, and they pose a problem of comparability of measure, mostly because the survey design differs between main survey and non-response follow-up. In order to detect possible bias, the survey variables included in non-response surveys have to be related to the mechanism of participation, but not be sensitive to measurement effects due to the different designs. Based on accumulated experience of four similar non-response follow-ups, we studied the survey variables that fulfill these conditions. We differentiated socio-demographic variables that are measurement-invariant but have a lower correlation with non-response and variables that measure attitudes, such as trust, social participation, or integration in the public sphere, which are more sensitive to measurement effects but potentially more appropriate to account for the non-response mechanism. Our results show that education level, work status, and living alone, as well as political interest, satisfaction with democracy, and trust in institutions are pertinent variables to include in non-response follow-ups of general social surveys.

Prioritizing Low Propensity Sample Members in a Survey: Implications for Nonresponse Bias

Many survey methodologists now agree that simply striving to increase the response rate is not an optimal approach for reducing nonresponse bias in the final survey estimates. Targeting sample cases that are underrepresented can help reduce nonresponse bias. However, the challenge lies in which cases to prioritize when resources are finite, and reducing the risk of nonresponse bias is the goal. We present an approach which identifies, prioritizes, and intervenes on low-propensity-to-respond cases during nonresponse follow-up. Targeted cases were assigned to in-person interviewing. Our results suggest that in-person interviewing can be an effective approach for gaining participation from lowpropensity cases. We also find that targeting low-propensity cases could improve representation and therefore should be considered by survey practitioners as a tool for nonresponse bias reduction.

An Experimental Examination of the Content of Persuasion Letters on Nonresponse Rates and Survey Estimates in a Nonresponse Follow-Up Study

Nonresponse follow-up studies are often conducted to understand whether respondents and nonrespondents di er on survey variables of interest in sample surveys. Methods used to recruit respondents often di er between nonresponse follow-up studies and main studies. One method is persuasion letters sent from study sta to nonrespondents that are tailored to the types of concerns raised by the respondent and recorded in paradata about the survey recruitment process. This study examined whether tailored persuasion letters yield higher response rates in nonresponse follow-up and whether respondents to a nonresponse follow-up di er depending on the content of the persuasion letter. Nonrespondents to the University of Michigan Dioxin Exposure Study, a survey of adults conducted in 2004 and 2005 in five counties in Michigan, were randomly assigned to either an appeal to help the community or tailored types of persuasion letters. No di erence in response rates to the letter types was found, but meaningful di erences in the survey variables appeared between nonrespondents responding to the follow-up and respondents to the main interview. These di erences also occurred between the two groups receiving di erent types of letters. The community appeal letter appeared to address unvoiced concerns and brought a di erent sample of nonrespondents compared to the tailored letters.

Exploring Mode Effects in a Panel Survey of New Businesses

Establishment surveys among the business population are often hampered by low response rates and high costs. In an effort to improve response rates and reduce costs, survey researchers have employed multi-mode data collection methods, often mixing a self-administered visual mode (such as a mail or web survey) with an aural mode such as telephone. This method provides both the benefits of the self-administered questionnaire (cost and convenience) and those of the more persuasive interviewer-administered mode for nonresponse follow-up. A better understanding of the potential effects of conducting multi-mode establishment surveys can provide information about the characteristics of businesses and respondents who choose the different modes and also guide future data collection efforts among this population. We used data from the baseline and three follow-up waves of the Kauffman Firm Survey (KFS), a longitudinal survey of new businesses offering a web survey to the entire sample followed by an opportunity for web nonresponders to complete their interviews by telephone, to shed light on the differences between businesses that complete the survey using the different modes and determine whether any mode effects can be detected. Specifically, we present a longitudinal analysis of three groups of KFS businesses: those that opted to complete all four rounds of the survey on the web; those that opted to complete all four rounds by telephone; and a group of “switchers” that completed two rounds by web and two rounds by telephone. For this descriptive analysis, we examine differences in these three groups across industry, size, and the complexity of the business, as well as differences in respondent demographics. We also analyze indicators of business growth, such as employment, assets, revenue, and profit or loss.

41: Applying an Expanded Set of Cognitive Design Principles to Formatting the NKF Kidney Early Evaluation Program Longitudinal Survey (KEEP-LS)

APPLYING AN EXPANDED SET OF COGNITIVE DESIGN PRINCIPLES TO FORMATTING THE NKF KIDNEY EARLY EVALUATION PROGRAM LONGITUDINAL SURVEY (KEEPLS) Jose Luis Calderon, 2 Erik Fleming, Monica Gannon, Shu-Cheng Chen, 4 Joseph A. Vassaloti, 3 Keith C. Norris,. College of Pharmacy, Health Professions Division, Nova Southeastern University, Fort Lauderdale, Florida, Charles R. Drew University of Medicine and Science, Los Angeles, California, National Kidney Foundation, New York, New York, KEEP Data coordinating Center, Minneapolis, Minnesota, David Geffen School of Medicine, University of California, Los Angeles (UCLA) The Kidney Early Evaluation Program (KEEP) is a targeted, community-based screening of populations at greatest risk for chronic kidney disease (CKD), those with high rates of diabetes, hypertension, and a high proportion of racial/ethnic minorities. The KEEP longitudinal study will adopt survey methods similar to those used in KEEP to gather follow-up (F/U) data to measure CKD-related heath status and gauge program effectiveness for repeated KEEP participants with evidence of moderate to severe kidney function deficiency. KEEP has defined objectives to enhance F/U survey response rates and target vulnerable populations who bear the highest CKD risk factor burdens. The KEEP F/U Form was assessed for adherence to six cognitive design principles (simplicity, consistency, organization, natural order, clarity, attractiveness) considered to summate the techniques guiding good survey development, and for the additional cognitive design principles readability and variation of readability across survey items. The form was found to include violations of each cognitive design principle and readability principle, possibly contributing to item nonresponse and low follow-up rates in KEEP. It was revised according to empirically substantiated formatting techniques guided by the principles, and found during qualitative assessment to be more userfriendly, simpler, better organized, more attractive, and easier to read. To ensure ease of use by populations with limited literacy skills, poor health literacy, and limited survey literacy, survey researchers must apply cognitive design principles to survey development to improve participation and response rates.

Following Up Nonrespondents to an Online Weight Management Intervention: Randomized Trial Comparing Mail versus Telephone

Background Attrition, or dropout, is a problem faced by many online health interventions, potentially threatening the inferential value of online randomized controlled trials.Objective In the context of a randomized controlled trial of an online weight management intervention, where 85% of the baseline participants were lost to follow-up at the 12-month measurement, the objective was to examine the effect of nonresponse on key outcomes and explore ways to reduce attrition in follow-up surveys.Methods A sample of 700 nonrespondents to the 12-month online follow-up survey was randomly assigned to a mail or telephone nonresponse follow-up survey. We examined response rates in the two groups, costs of follow-up, reasons for nonresponse, and mode effects. We ran several logistic regression models, predicting response or nonresponse to the 12-month online survey as well as predicting response or nonresponse to the follow-up survey.ResultsWe analyzed 210 follow-up respondents in the mail and 170 in the telephone group. Response rates of 59% and 55% were obtained for the telephone and mail nonresponse follow-up surveys, respectively. A total of 197 respondents (51.8%) gave reasons related to technical issues or email as a means of communication, with older people more likely to give technical reasons for noncompletion; 144 (37.9%) gave reasons related to the intervention or the survey itself. Mail follow-up was substantially cheaper: We estimate that the telephone survey cost about US $34 per sampled case, compared to US $15 for the mail survey. The telephone responses were subject to possible social desirability effects, with the telephone respondents reporting significantly greater weight loss than the mail respondents. The respondents to the nonresponse follow-up did not differ significantly from the 12-month online respondents on key outcome variables.ConclusionsMail is an effective way to reduce attrition to online surveys, while telephone follow-up might lead to overestimating the weight loss for both the treatment and control groups. Nonresponse bias does not appear to be a significant factor in the conclusions drawn from the randomized controlled trial.

Eliciting Survey Cooperation

Given the weaker ties to community as noted by scholars such as Robert Putnam, survey researchers should not be surprised by a decline in survey participation over the past 10 years. This research analyzes the use of incentives coupled with introductory themes emphasizing cooperation and helpfulness--cooperative norms in American society--to understand their effects on survey response. This article analyzes two separate experiments (one phone and one mail) that provide evidence that norms of cooperation matter in the decision to participate in a survey, suggesting that this is particularly true at the refusal conversion stage. These results indicate that survey researchers may use such themes to their advantage, especially when conducting a nonresponse follow-up in a mail survey.