Response Rates, Nonresponse Bias, and Data Quality

Abstract

Survey response rate is regarded as a key data-quality indicator, yet response rate is not necessarily predictive of nonresponse bias. Our study objective was to use a high-response-rate survey to assess nonresponse bias across successive waves. This survey of healthcare leaders utilized a web-based, self-report format with an initial invitation and four nonrespondent follow-ups. Across five waves, comparisons were made for demographic and facility characteristics, proportion of items completed, and distribution of three question types: factual reports of customized categorical responses; single-item evaluations using five-point Likert scales; and multi-item scales, across four- or five-point Likert scales. The overall response rate was 95 percent (118/124); waves did not differ by demographic and facility characteristics or missing data. Across waves, there were no significant differences between responses to two factual report questions or the single- or multi-item scale measures of attitudes. According to a “what-if” analysis of cumulative results by wave, the same conclusions would have been reached if data collection had been halted at earlier points in time. Precision and statistical power increased as number of respondents accumulated by wave. The high response rate facilitated studying the impact of nonresponse bias by wave. Although high response rates are desirable because of precision and power, as survey fatigue increases, absolute thresholds representing “adequate” response rates may be less realistic. Results from “low” response-rate surveys should be considered on their merits, as they may accurately represent attitudes of the population. Therefore, low response rates should not be cited as reasons to dismiss results as uninformative.