Chi-Squared Tests with Survey Data

Abstract

SUMMARY It is usually unrealistic to regard observations from a complex survey as independent, but practitioners often use standard chi-squared tests with survey data. We look at the effect of correlations among elements on the performance of the tests. Empirical results are given for two national surveys which suggest that the effect is severe for tests of goodness-of-fit or homogeneity but less severe for tests of independence. KISH AND FRANKEL (1974) have drawn attention to the problems that arise when standard statistical methods, based on the assumption that observations are independent, are applied to survey data. In most surveys, the assumption of independence is far from realistic. Any large- scale survey will involve stratified multi-stage sampling and correlations between units in the same cluster (or stratum) can have a substantial impact. In this paper we look at the effect of this structure on the behaviour of ordinary chi-squared tests for goodness-of-fit, homogeneity and independence. Examples of the use of standard chi-squared tests in survey data are plentiful. An investigator may wish to compare the achieved sample proportions for the categories of variables such as age, sex or marital status with known population proportions, for example, as a check on the quality of the sampling. Brackstone and Gosselin (1973) give such an example for the underenumeration in the 1971 Canadian Census. More generally, we might make comparisons between several different surveys from the same population or, between different regions of the country or between similar surveys from different countries as in the World Fertility Survey comparisons. An ordinary chi-squared test of homogeneity would be natural here except for the complexity of the sampled populations. Examples of use of the standard chi- squared test for independence in a two-way contingency table with survey data can be found in almost any issue of most methodological journals in the Social Sciences. Casperis and Vaz (1975) or Chase (1975) are typical examples. In all these situations, it is common practice to ignore the complexity of the survey and proceed as if the ordinary chi-squared tests will behave much the same as under multinomial sampling. In Section 2, we examine the behaviour of the goodness-of-fit test. A brief sketch of the large sample theory is given and we look at the effect in practice for variables from two large national surveys; the General Household Survey (GHS) of 1971 and the British Election Study (BES) of 1974. The problem of homogeneity is tackled in Section 3. The results are very similar in both cases; the chi-squared test can be very seriously affected by the lack of independence in the observations. However a simple multiplying factor, which can be calculated from information which should be available in many surveys, improves the performance dramatically. In Section 4 we look at the important problem of testing for independence in a two-way table. The theoretical results are much more complicated here and an appropriate modifying factor is more difficult to compute. In contrast to the results of homogeneity and goodness-of-fit the impact of dependence among the observations seems much less severe in practice, and the ordinary chi-squared test often needs very little modification. These conclusions can be regarded as a natural extension of established results for the difference of sub-class means, of