Sampling Design Effects: Do They Affect the Analyses of Data from the National Survey of Families and Households?

Abstract

Most large national surveys, such as the National Survey of Families and Households (NSFH), involve clustered and stratified samples. These complex sample designs have consequences for data analysis techniques. Standard errors calculated using procedures that do not adjust for design effects often are too small and lead to incorrect inferences. We discuss design effects and estimate them for a set of variables selected from the 1988 NSFH. Included are examples of descriptive estimates and regression results with household income and marital happiness as dependent variables. Statistical software that adjusts standard errors in complex designs is discussed, as are issues related to weighting and the analysis of subsamples. As family researchers increase their use of data collected in large, complex personal-interview surveys, such as the NSFH, there is a need for greater awareness of the ways that the sampling design affects the analysis of the data and statistical inferences. Statistical techniques and the standard software (e.g., SPSS, SAS) used by most family researchers make the assumption that the data were collected by simple random sampling. Most large-scale personal-interview surveys, for reasons of efficiency and economy, use probability sampling designs that are not simple random samples. Stratification, clustering, and differential case weighting are common in the design of these multistage probability samples, all of which have consequences for statistical inference. Sampling statisticians have convincingly demonstrated that complex sampling designs can have large effects on the estimates of standard errors (Lehtonen & Pahkinen, 1995). Special statistical routines and software have been developed to analyze data from large-scale complex surveys sponsored by United States government agencies. Indeed, many federal agencies require that researchers who plan to analyze data from complex samples indicate how they will account for design effects in their analysis. There is little evidence that family researchers are concerned about these issues, particularly with regard to the analysis of data from the NSFH sample. Over 250 papers that use these data are reported on the NSFH website (more than 65 of those papers appeared in Journal of Marriage and the Family), and we are aware of only a few that reported that the researchers attempted to account for the effect of the sampling design on the statistical inferences. Is this a serious error, resulting in a number of biased findings, or are the design effects in the NSFH sample sufficiently minor so that family researchers can routinely disregard them? In this article, we introduce the notion of a design effect and explain how it can affect research results. We discuss procedures and software for assessing whether the presence of design effects bias the research results. Finally, we present two examples using the 1988 NSFH survey data to demonstrate the consequences of design effects in regression models. WHAT ARE DESIGN EFFECTS? Complex interview surveys often employ sampling designs that include clustering, stratification, and differential weighting of cases (Deaton, 1997; Lehtonen & Pahkinen, 1995). These procedures are used in personal-interview studies to increase the cost efficiency of the data collection procedures. Although probability processes are used in these designs, they depart from the selfweighting and independent observation requirements of simple random sampling. The notion of a design effect was introduced (Kish, 1965) to measure the consequences of these departures from simple random sampling for statistical inferences. A design effect-Deg-is the sampling variance-var(ComplexDesign)-of an estimate in a complex probability sample divided by the sampling variance-var(SRS)-that would have been found if the sample of the same size had been selected using a simple random sample: The variance of the complex design-var(ComplexDesign)-can only be calculated using special formulas or procedures that take design features, such as clustering, into account. …