Unstable Inferences? An Examination of Complex Survey Sample Design Adjustments Using the Current Population Survey for Health Services Research

Abstract

Statistical analysis of the Current Population Survey's Annual Social and Economic Supplement is used widely in health services research. However, the statistical evidence cited from the Current Population Survey (CPS) is not always consistent because researchers use a variety of methods to produce standard errors that are fundamental to significance tests. This analysis examines the 2002 Annual Social and Economic Supplement's (ASEC) estimates of national and state average income, national and state poverty rates, and national and state health insurance coverage rates. Findings show that the standard error estimates derived from the public use CPS data perform poorly compared with the survey design-based estimates derived from restricted internal data, and that the generalized variance parameters currently used by the U.S. Census Bureau in its ASEC reports and funding formula inputs perform erratically. Because the majority of published research (both by academics and Census Bureau analysts) does not make use of the survey design-based information available only on the internal ASEC data file, we argue that the Census Bureau ought to use alternative methods for its official ASEC reports. We also argue that for public use data the Census Bureau should produce a set of replicate weights for the ASEC or release a set of sample design variables that incorporate statistical "noise" to maintain respondent confidentiality (e.g., pseudo-primary sampling units) as other federal government surveys do. This is essential to make appropriate inferences using the ASEC data regarding statistical significance and estimate variance for health policy analysis.