Beyond question wording: How survey design and administration shape estimates of disability

Abstract

BackgroundBetween 2008 and 2014, annual estimates of disability prevalence among U.S. adults varied somewhat across federal surveys that use a standardized measure of disability, but trends over-time were relatively stable and consistent. In 2014, however, estimates of disability from the Survey of Income and Program Participation (SIPP) increased markedly relative to previous years and were much higher than disability estimates from other federal surveys. ObjectiveTo examine why disability prevalence among adults aged 40 and older substantially increased in the first wave of the 2014 SIPP Panel. MethodsWe consider three factors that may have contributed to the rise in disability: data processing, context effects linked to question order, and sampling. To do so, we compare estimates with and without survey weights and imputed data, analyze supplemental disability-related data collected among SIPP participants, and employ decomposition analysis to assess what proportion of the increase in disability can be attributed to changes in sample composition. ResultsWe find evidence that differences in sample composition contributed to the observed rise in disability prevalence in SIPP between 2011 and 2014. There is less evidence that weighting and imputation or context effects played a role. ConclusionsPrevious studies emphasize differences in operationalization and conceptualization of disability as the major factor driving discrepancies in disability estimates. This study suggests that other factors related to survey design and administration may influence disability measurement. Such aspects of surveys, like question order and sampling, may be difficult to standardize, leading to meaningful cross-survey differences in disability estimates.