Do Coefficients of Variation of Response Propensities Approximate Non-Response Biases During Survey Data Collection?

Jamie C Moore,Gabriele B Durrant,Peter W F Smith

doi:10.1111/rssa.12624

Abstract

AbstractWe evaluate the utility of coefficients of variation of response propensities (CVs) as measures of risks of survey variable non-response biases when monitoring survey data collection. CVs quantify variation in sample response propensities estimated given a set of auxiliary attribute covariates observed for all subjects. If auxiliary covariates and survey variables are correlated, low levels of propensity variation imply low bias risk. CVs can also be decomposed to measure associations between auxiliary covariates and propensity variation, informing collection method modifications and post-collection adjustments to improve dataset quality. Practitioners are interested in such approaches to managing bias risks, but risk indicator performance has received little attention. We describe relationships between CVs and expected biases and how they inform quality improvements during and post-data collection, expanding on previous work. Next, given auxiliary information from the concurrent 2011 UK census and details of interview attempts, we use CVs to quantify the representativeness of the UK Labour Force Survey dataset during data collection. Following this, we use survey data to evaluate inference based on CVs concerning survey variables with analogues measuring the same quantities among the auxiliary covariate set. Given our findings, we then offer advice on using CVs to monitor survey data collection.

Highlights

Methodologists no longer advocate only maximising response rates to minimise risks of survey variable non-response biases (Kreuter, 2013; Olson, 2006)
This can inform method modifications to target under-represented subgroups and reduce the risks of bias and/or minimise costs. Practitioner interest in this more refined approach to managing survey dataset quality is increasing, but limited information on the performance of the proposed bias risk indicators restricts use. We address this issue using survey variable data to evaluate the performance of one set of indicators, coefficients of variation of response propensities (CVs), when monitoring data collection
We evaluate CV-based inference about survey variables with auxiliary covariate analogues by first computing logistic-regression-based estimates of census auxiliary covariate standardised ‘non-response biases’ for comparison

Summary

| INTRODUCTION

Methodologists no longer advocate only maximising response rates to minimise risks of survey variable non-response biases (Kreuter, 2013; Olson, 2006). Instead, assessing variation in response across groups defined by subject attributes that are correlated with the survey variables is advised, including monitoring during data collection if interview attempt details exist This can inform method modifications to target under-represented subgroups and reduce the risks of bias and/or minimise costs (adaptive strategies: Groves & Heeringa, 2006; Peytchev et al, 2010; Wagner, 2008). CVs and their counterparts, R indicators (together, representativeness indicators), are potentially valuable tools for assessing survey dataset quality (Schouten et al, 2012; see Section 2.1) Both indicators quantify variation in sample response propensities estimated given a set of auxiliary covariates observed for all subjects in the issued sample. We advise on how to use CVs to monitor data collection

| METHODS

Findings

| DISCUSSION