Reinterview Data Research Articles

Survey Reliability: Models, Methods, and Findings.

Although most survey researchers agree that reliability is a critical requirement for survey data, there have not been many efforts to assess the reliability of responses in national surveys. In addition, there are quite different approaches to studying the reliability of survey responses. In the first section of the Lecture, I contrast a psychological theory of over-time consistency with three statistical models that use reinterview data, multi-trait multi-method experiments, and three-wave panel data to estimate reliability. The more sophisticated statistical models reflect concerns about memory effects and the impact of method factors in reinterview studies. In the following section of the Lecture, I examine some of the major findings from the literature on reliability. Despite the differences across methods for exploring reliability, the findings mostly converge, identifying similar respondent and question characteristics as major determinants of reliability. The next section of the paper looks at the correlations among estimates of reliability derived from the different methods; it finds some support for the validity of the measures from traditional reinterview studies. The empirical claims motivating the more sophisticated methods for estimating reliability are not strongly supported in the literature. Reliability is, in my judgment, a neglected topic among survey researchers, and I hope the Lecture spurs further studies of the reliability of survey questions.

Comparing Methods for Assessing Reliability.

The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewedtwice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticatedmethods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.

Social criticism moderates the relationship between anxiety and depression 10 years later

BackgroundResearch has consistently documented anxiety and depression as bidirectional risk factors for one another. However, little research investigates the sequential comorbidity of anxiety and depression over lengthy durations, and the influence of contextual variables on this relationship have not been fully empirically investigated. MethodThe current study examined perceived social criticism as a moderator of the relationship between a history of anxiety and a past 12-month depressive episode at least 10 years later (and vice versa) utilizing the National Comorbidity Survey Baseline (N = 8,098) and Re-interview data (N = 5,001). History of anxiety and depressive diagnoses were assigned at Wave 1, past year diagnosis at Wave 2, and perceived social criticism was assessed at Wave 1. ResultsStructural equation modeling indicated that when controlling for a Wave 1 latent depression factor, a positive relationship between Wave 1 latent anxiety and a Wave 2 latent depression emerged for those endorsing higher perceived social criticism from friends and relatives, respectively. Unexpectedly, when controlling for Wave 1 latent anxiety, a negative relationship between Wave 1 latent depression and Wave 2 latent anxiety emerged for those endorsing higher perceived social criticism from friends, but no relationship when moderated by perceived social criticism from relatives. LimitationsPerceived social criticism was self-reported, which may introduce self-perception bias. ConclusionsResults identified perceived social criticism as an important moderator in the sequential comorbidity of anxiety and depression over a long period of time.

Population Assessment of Tobacco and Health (PATH) reliability and validity study: selected reliability and validity estimates

IntroductionThis paper reports a study done to estimate the reliability and validity of answers to the Youth and Adult questionnaires of the Population Assessment of Tobacco and Health (PATH) Study.Methods407...

Who Can You Count On? Understanding The Determinants of Reliability.

Using reinterview data from the PATH Reliability and Validity (PATH-RV) study, we examine the characteristics of questions and respondents that predict the reliability of the answers. In the PATH-RV study, 524 respondents completed an interview twice, five to twenty-four days apart. We coded a number of question characteristics and used them to predict the gross discrepancy rates (GDRs) and kappas for each question. We also investigated respondent characteristics associated with reliability. Finally, we fitted cross-classified models that simultaneously examined a range of respondent and question characteristics. Although the different models yielded somewhat different conclusions, in general factual questions (especially demographic questions), shorter questions, questions that did not use scales, those with fewer response options, and those that asked about a noncentral topic produced more reliable answers than attitudinal questions, longer questions, questions using ordinal scales, those with more response options, and those asking about a central topic. One surprising finding was that items raising potential social desirability concerns yielded more reliable answers than items that did not raise such concerns. The respondent-level models and cross-classified models indicated that five adult respondent characteristics were associated with giving the same answer in both interviews-education, the Big Five trait of conscientiousness, tobacco use, sex, and income. Hispanic youths and non-Hispanic black youths were less likely to give the same answer in both interviews. The cross-classified model also found that more words were associated with less reliable answers. The results are mostly consistent with earlier findings but are nonetheless important because they are much less model-dependent than the earlier work. In addition, this study is the first to incorporate such personality traits as needed for cognition and the Big Five personality factors and to examine the relationships among reliability, item nonresponse, and response latency.

Pornography, Religion, and Parent-Child Relationship Quality.

Frequent pornography use is often negatively associated with marital quality. Recent research has argued that this negative association is particularly strong for those who are embedded in religious communities, likely due to the greater stigma and shame associated with viewing pornography. In order to test and extend this theory, the current study examined how religious service attendance moderates the link between parents' pornography consumption and four measures of parent-child relationship quality. Analyses of 2006 Portraits of American Life Study data (N=2610) revealed that greater pornography viewing predicted negative outcomes on two out of four measures of parent-child relationship quality, while religious service attendance was associated with more positive parent-child relationship outcomes. Interaction effects, however, affirmed that the negative association between porn viewing frequency and three parent-child relationship outcomes was stronger for participants who attended religious services more often. Analyzing fathers (N=771) and mothers (N=904) separately revealed that the observed relationships held more consistently for fathers than mothers. Evidence for directionality was presented by incorporating re-interview data from 2012. While pornography use may be negatively associated with some aspects of parent-child relationship quality, this association was particularly strong for those embedded within religious communities, possibly owing to greater attendant guilt and shame.

How Real is Mobility Between Low Pay, High Pay and Non-Employment?

SummaryThe aim of the paper is to investigate the effect of measurement error on low pay transition probabilities. Our approach combines the virtues of panel regression and latent class models, though it does not require the use of validation or reinterview data. Using British, German and Dutch panel data, we show that the true estimated low pay transition probability is much lower that what previous research has found. This implies that almost half of the observed transitions can be attributed to measurement error. The highest low pay transition probabilities are found in Germany and the lowest in the Netherlands. When applying this correction for measurement error in a multivariate model of low pay transitions, the results indicate that measurement error attenuates considerably the effects of the main covariates, such as training, job change, change in the type of employment contract and shift from part-time to full-time employment.

Network stability in longitudinal data: A case study from rural Malawi

This paper investigates the stability of egocentric informal conversational networks in rural Malawi by analyzing reinterview data collected by a longitudinal survey. Contrary to the findings of test–retest studies carried out in developed countries, in rural Malawi geographically and socially close networks are found to exhibit a low level of stability even over short periods of time. This is mainly attributed to the specific structure of the networks considered, and it is therefore recommended that the importance of measurement bias should not be overemphasized. It is also found that, despite the significant changes in the named partners, network size and reported actions of network partners are quite stable, thus suggesting a negligible impact of network instability for further analyses of these data.

Reliability of sport fish consumption in the New York State Angler Cohort Study

This paper examines the reliability of sport fish consumption data from the New York State Angler Cohort Study (NYSACS). NYSACS is a prospective cohort study conducted among New York State registered sportsmen and fishermen. Sport fish consumption information for New York State waters including the Great Lakes between June 1990 and June 1991 were collected through self-administered questionnaires, Spouses of male anglers were asked to provide their husbands’ fish consumption during the same time period. A short telephone interview after the cohort was ascertained was also conducted among about 100 study participants for the purpose of quality control. Percentage agreement, kappa, and weighted kappa were calculated to evaluate the reliability of the interview using spousal data and reinterview data. Overall, for the total fish consumption between June 1990 and June 1991, percentage agreement, kappa, and weighted kappa between spousal data and primary response were 67.28%, 0.5087, and 0.6157, respectively. For reinterview data, weighted kappa ranged from 0.4510 to 0.5285 for season-specific analysis and kappa ranged from 0.4615 to 0.7006 for fish species-specific analysis. Spouses may be a good source of proxy data for fish consumption. The reliability of sport fish consumption data for the NYSACS study is acceptable, suggesting that the food frequency measuring methods employed are a viable approach to obtain retrospective sport fish consumption data from sportsmen and subsistence anglers.

Incidence of mental disorders in Edmonton: estimates of rates and methodological issues

An incidence study was conducted in Edmonton, Alberta, Canada, in which 3956 community residents were interviewed using the Diagnostic Interview Schedule (DIS) and a sample of 1964 subjects were reinterviewed with the DIS an average of 2.8 years later. Incidence rates, estimated for a range of DSM-III disorders, were surprisingly large, raising questions about the reliability of DIS data. We examined major depression in detail to uncover possible sources of unreliability. There were 138 ‘incident’ cases of major depression, giving an annual incidence rate for both sexes of 27.9 (per 1000). However, based on reinterview data, 106 (80%) of the incident cases reported an age of onset prior to the initial interview. These findings appear to be the result of a difference between the DIS definition of age of onset and the one used in our analysis of incidence. A syndrome called depressive spell is defined and used to demonstrate that the large number of incident cases is likely due, at least in part, to incomplete recall of lifetime depressive symptoms. Despite these potential sources of bias, the possibility remains that incidence rates and lifetime risks in Edmonton are larger than has generally been reported in the literature.

On the quality of reinterview data with application to the Current Population Survey.

"The [U.S.] Current Population Survey (CPS) reinterview sample consists of two subsamples: (a) a sample of CPS households is reinterviewed and the discrepancies between the reinterview responses and the original interview responses are reconciled for the purpose of obtaining more accurate responses..., and (b) a sample of CPS households, nonoverlapping with sample (a), is reinterviewed 'independently' of the original interview for the purpose of estimating simple response variance (SRV). In this article a model and estimation procedure are proposed for obtaining estimates of SRV from subsample (a) as well as the customary estimates of SRV from subsample (b).... Data from the CPS reinterview program for both subsamples (a) and (b) are analyzed both (1) to illustrate the methodology and (2) to check the validity of the CPS reinterview data. Our results indicate that data from subsample (a) are not consistent with the data from subsample (b) and provide convincing evidence that errors in subsample (a) are the source of the inconsistency."

On the Quality of Reinterview Data with Application to the Current Population Survey

Abstract The Current Population Survey (CPS) reinterview sample consists of two subsamples: (a) a sample of CPS households is reinterviewed and the discrepancies between the reinterview responses and the original interview responses are reconciled for the purpose of obtaining more accurate responses (i.e., response bias estimates), and (b) a sample of CPS households, nonoverlapping with sample (a), is reinterviewed “independently” of the original interview for the purpose of estimating simple response variance (SRV). In this article a model and estimation procedure are proposed for obtaining estimates of SRV from subsample (a) as well as the customary estimates of SRV from subsample (b). In this way, an improved estimator of SRV that combines data from both subsamples can be computed. Additionally, under conditions that are usually satisfied in practice, several inequalities involving statistics computed from both subsamples are derived. These inequalities can be used to check the validity of the reinterv...

Test-Retest Reliability in Population Research

The widespread use of the questionnaire and interview as measuring instruments in population research has been, for the large part, unaccompanied by evaluation of their reliability. More than 25,000 publications in the field of population were referenced in the Population Index during the 1960s, which gives an indication of the breadth of research in this growing field. Nevertheless, virtually none of these studies addressed the problem of the reliability of the data being used. This omission is a serious one. Without assurance that responses are reliable, a researcher can never be certain that he is measuring the variable he intends to measure. In other words, validity presupposes reliability. A recent study of fertility patterns in the United States by Ryder and Westoff (1971) includes reinterview data on 382 white, married, adult women. It thus represents one attempt to assess consistency, or stability, of knowledge, attitude, and practice responses over time. The present study adds to these much needed methodological data by presenting test-retest reliability results for teenagers in a metropolitan area in the United States who completed questionnaires focused on issues related to family formation and population. The goal of this study was to ascertain if and how reliability is differentially affected by characteristics of the questionnaire and of the respondent. Specifically, it asked whether reliability varies as a function of: (1) the content (personal history, family formation, population); (2) the psychological nature of the item (knowledge; attitudes, beliefs, and opinions; desires, expectations, and intentions); and (3) the characteristics of the respondent (sex; age [younger versus older teenagers]; race [white versus nonwhite]; religion [Catholic versus non-Catholic]).

Expenditures for Life Insurance among Working-Wife Families

Working-wife families with fully-employed husbands may constitute both an opportunity and a challenge to the life insurance industry because they spend substantially less on life insurance than do housewife families with similar incomes. This article investigates the extent of and reasons for the underconsumption of life insurance by these relatively affluent families whose number is growing rapidly. The relevance of several economics and sociological hypotheses to this phenomenon is investigated. The quantitative portion of the analysis is accomplished by tabular comparisons and multiple regression. Suggestions for further research and marketing action are offered. Approximately twenty per cent of today's United States labor force consists of married women with spouse present. Indications are that such families will increase in number absolutely and as a proportion of the labor force.' Workingwife families, with husband present and fully employed, may constitute a market segment of special interest to the life insurance industry for two reasons. First, by their very nature, they have above Jacob M. Duker, Ph.D., is Associate Professor, Marketing, in the School of Business Administration of the University of Connecticut. This article was submitted in July, 1968. The author expresses appreciation for suggestions received from Professors Irving Schweiger, Harry V. Roberts and Harry Davis of the University of Chicago and Harry M. Johnson of the University of Connecticut; for financial support from the University of Connecticut Research Foundation; for data received from the Survey Research Center of the University of Michigan (through Ford Foundation assistance) and the University of Pennsylvania (in concert with the U.S. Bureau of Statistics); and for computer time at the University of Connecticut Computer Center through the National Science Foundation. 1 See Sophia Cooper and Denis F. Johnston, Labor Force Projections for 1970-1980, Monthly Review, February, 1965, pp. 129-140; Table 1, p. 130. average incomes, and, second, they spend less on life insurance than do families of similar incomes, in which the wife is a housewife rather than a working-wife. This article presents the evidence of such expenditure differences in tabular analyses and by multiple regression analysis. It also examines the meaning of this evidence by application of some behavioral hypotheses as well as by the multiple regression analysis. Data and Methodology The major portion of the data is derived from the Survey Research Center of the University of Michigan Survey of Consumer Finances for 1959 and subsequent reinterview data. Data from the 1950 BLS-Wharton Survey of Consumer Expenditures are used to reduce the dependence on one-year cross section data.2 2 Study 688, Decks 1, 2, 3, 4, and 6, University of Michigan Survey Research Center, Ann Arbor Michigan for the 1959 data; and Study 707, Deck F and Study 715, Deck 5, same source for the reinterview data. (See footnote 23 infra); and Deck MP-2, covering U.S. Bureau of Statistics, Wharton School of Finance and Commerce Survey of Consumer Expenditures, 1950. Of the 2,972 families in Deck 688, 591 met the criteria and were used in this study.

Housing and Permanent Income: Tests Based on a Three-Year Reinterview Survey

U NTIL the mid-1950's, the traditional view on the consumption of housing had been that the elasticity of housing consumption with respect to current income was less than unity.' However, recent literature on the theory of consumption function [15] [4] has pointed out that, in relating income to consumption, the concept of income should be that of permanent income rather than current or measured income, and that the use of measured income imparts a downward bias in estimating the effect of permanent income on consumption. Since housing consumption in particular may also be affected by the long-run prospect of income rather than by a single-year measured income, the permanent income elasticity may be higher than the previous estimates of the measured income elasticity of housing. The change in the concept of income encouraged a number of economists to test the effect of permanent income on housing consumption. Maisel and Winnick [14] tested the permanent income hypothesis of housing consumption by utilizing the I950 Survey of Consumer Expenditures. Contrary to the permanent income hypothesis, however, they reported that housing consumption was no more responsive to changes in permanent income than to changes in measured current income. Subsequently Margaret Reid [19] conducted an extensive cross-sectional study by using housing and income data obtained primarily from the 1950 Housing Census. Differing from Maisel and Winnick, Reid found that the demand for housing was indeed more responsive to changes in permanent income. Moreover, Reid's estimates of the permanent income elasticity of housing were substantially greater than one, and in fact ranged from 1.5 to 2. On the other hand, the author's recent time-series analyses [10] [11] indicated that while housing demand was more responsive to changes in permanent income, the permanent income elasticity of housing was still less than unity. The purpose of this paper is to obtain the cross-sectional estimates of permanent income elasticity on the basis of the 1960-1961-1962 reinterview Surveys of Consumer Finances. This study has three distinct features. First, this study differs from other housing studies in that it uses the instrumental variable method along the lines suggested by Livitan [12]. Livitan has recently shown that, for any year of analysis, the use of a lagged or future measured income as an instrumental variable yields a powerful test of the permanent income hypothesis. Second, since this study utilizes threeyear reinterview data, as a result both one-year and two-year lagged or future incomes can be used as instrumental variables. Due to the lack of data, Livitan [12] could not experiment with a two-year lagged or future income as an instrumental variable in his analysis of total consumption. Both Livitan [13] and Friedman [5], however, agreed that it is highly desirable to use a two-year lagged or future income as an instrumental variable. Finally, this paper develops and uses an extended version of Livitan's instrumental variable method in order to take account of the case in which a lagged or future income may not be a perfect instrumental variable for permanent income. The cross-sectional estimates of permanent income elasticity so estimated are substantially * The author is Professor of Economics at the University of Wisconsin-Milwaukee and a member of the Social Systems Research Institute at the University of Wisconsin, Madison. He is grateful to Melvin Lurie and Keith Phillips for their helpful comments on the initial draft of this paper. The author, however, remains responsible for the views expressed in this paper. This study was originally supported by the National Science Foundation under grant GS-631 and later by the Graduate School of the University of Wisconsin-Milwaukee. The data used in the study originally came from the Surveys of Consumer Finances conducted by the Survey Research Center, University of Michigan, in cooperation with the Board of the Governors of the Federal Reserve System. The author is indebted to the Social Systems Research Institute for making available the data coded on magnetic tapes. 1 For a brief account of the early history of housing studies, see the author's paper [10, pp. 82-83].

Consumer Attitudes, Buying Plans, and Purchases of Durable Goods: A Principal Components, Time Series Approach

FOR more than ten years, the University of Michigan Survey Research Center has gathered data on consumer attitudes and buying plans in its periodic sample surveys. According to the SRC, Information on consumer sentiment and expectations is collected because they exert a great influence on demand for durable goods and housing and thereby on economic trends in general. 1 But tests of the relationship between attitudes, intentions, and consumer behavior have produced conflicting results and have caused doubts as to the usefulness of the attitude variables for purposes of forecasting.2 attitudes and buying plans data reflect various aspects of consumer sentiment. It is not clear to what extent these are separate attitudinal dimensions, and which of these, if any, have a useful role in forecasting. Analyses of the predictive power of attitudes and buying plans in cross section reinterview data have tended to discredit the relevance of attitudes, but not of buying plans. Klein and Lansing, Tobin, and others 3 have found that after predicting on the objective variables, the effects of the attitudinal variables individually (and combined into an index) are not statistically significant in the cross section. the other hand, intentions to purchase do add significantly to the explanation of realized purchases. Aggregate time series give the opposite result. While the earliest time series studies of aggregate data by the Consultant Committee on Survey Statistics, Okun, and Mueller 4 were inconclusive, more recently the Mueller studies and other calculations by the Survey Research Center5 have consistently shown that in aggregate time series, attitudes make a significant contribution to explaining fluctuations in aggregate durables purchases, while buying intentions do not make a statistically significant net contribution to the forecast. Proponents of the attitudes data have argued that the ultimate test of their usefulness is their performance as an aid to prediction in aggregate time series. case for or against the predictive value of consumer attitudes and intentions data is still open. This paper, part of a larger effort to reexamine the role of subjective data in predicting consumers' expenditures on durable goods, is focused on the behavior of the aggregate * author wishes to thank Professor Lawrence R. Klein for his encouragement and advice throughout this project. A number of willing assistants, M. J. Desai, M. W. S. Davenport, and J. J. Kim, helped with the computations. Survey Research Center kindly provided most of the data on which the study is based. work benefited from an institutional grant from the National Science Foundation. 1 Survey Research Center, Outlook for Demand (Ann Arbor, 1961), i. 2U.S. Congress, Joint Economic Committee, Consumer Survey in Reports of the Federal Reserve Consultant Committee on Economic Statistics (Washington, 1955); and Arthur M. Okun, The Value of Anticipations Data in Forecasting National Product, in National Bureau of Economic Research, Quality and Economic Significance of Anticipations Data (Princeton, 1960). 'L. R. Klein and J. B. Lansing, Decisions to Purchase Durable Goods, Journal of Marketing xx (Oct., 1955), 109-132. In this study, attitudinal responses indicating a feeling of well-being and indicating anticipation of price increase did show a significant relationship to purchases; James Tobin, On the Predictive Value of Intentions and Attitudes, Review of Economics and Statistics XLI (Feb., 1959), 1-11; Eva Mueller, Effects of Attitudes on Purchases, American Economic Review (Dec., 1957), 946-965; and Consumer Attitudes: Their Influence and Forecasting Value, in National Bureau of Economic Research, op. cit., 149-179. Also, F. Thomas Juster, Expectations, Plans, and Purchases: A Progress Report, Occasional Paper No. 70, National Bureau of Economic Research (New York, 1958); and with regard to automobile expenditures, Peter E. dejanosi, Factors Influencing the Demand for New Automobiles, Journal of Marketing (April, 1959), 412-418. 'U.S. Congress, Joint Economic Committee, op. cit.; Okun, op. cit.; and Mueller, op. cit. 'The most recent calculations are summarized in Survey Research Center, Fifteen Years of Experience with Measurement of Expectations (mimeo.) paper presented at the Meetings of the American Statistical Association (Sept., 1962, Minneapolis).

Some Evidence on Consumer Tastes and Habits

This paper presents some empirical results that may prove useful in formulating a general dynamic theory of demand for durables. For our calculation, we utilise the reinterview data of the 1957 and 1958 Surveys of Consumer Finances, and concentrate on the demand for automobiles, believing that the resulting observations can be generalised to other durables.

Consumer Durable Goods Expenditures, with Major Emphasis on the Role of Assets, Credit and Intentions

Abstract Previous use of credit appears to be a significant discriminator of subsequent durable goods purchasing behavior and still more important as a discriminator of the financing method used by durable good purchasers, according to an analysis of reinterview data from the 1957 and 1958 Surveys of Consumer Finances. The relation of liquid asset holdings to subsequent purchasing behavior seems to be related to the type of such asset holdings, to previous and to current credit use. These and other relationships with respect to intentions to purchase suggest that surveys more pointedly focussed on the specific questions related to consumer durable goods purchasing behavior could yield better results than those yet attained.