Measurement Errors In Surveys Research Articles

A Mixture Model Approach to Assessing Measurement Error in Surveys Using Reinterviews

Abstract Researchers are often unsure about the quality of the data collected by third-party actors, such as survey firms. This may be because of the inability to measure data quality effectively at scale and the difficulty with communicating which observations may be the source of measurement error. Researchers rely on survey firms to provide them with estimates of data quality and to identify observations that are problematic, potentially because they have been falsified or poorly collected. To address these issues, I propose the QualMix model, a mixture modeling approach to deriving estimates of survey data quality in situations in which two sets of responses exist for all or certain subsets of respondents. I apply this model to the context of survey reinterviews, a common form of data quality assessment used to detect falsification and data collection problems during enumeration. Through simulation based on real-world data, I demonstrate that the model successfully identifies incorrect observations and recovers latent enumerator and survey data quality. I further demonstrate the model’s utility by applying it to reinterview data from a large survey fielded in Malawi, using it to identify significant variation in data quality across observations generated by different enumerators.

Subjective Return Expectations, Perceptions, and Portfolio Choice

Exploiting a representative sample of the French population by age, wealth, and asset classes, we document novel facts about their expectations and perceptions of stock market returns. Both expectations and perceptions of returns are very dispersed, significantly lower than their data counterparts, and a substantial portion of the variation in the former is explained by dispersion in the latter. Consistent with portfolio choice models under incomplete information, a conditional risk-return trade-off explains the intensive margin, while at the extensive margin, only expected returns matter. Despite accounting for survey measurement error in subjective return expectations, ’muted sensitivities’ at both portfolio choice margins obtain, getting consistently (i) bigger when excluding informed non-participants, and (ii) smaller, for inertial and professionally delegated portfolios.

Dependent interviewing: a remedy or a curse for measurement error in surveys?

Longitudinal surveys often rely on dependent interviewing (DI) to lower the levels of random measurement error in survey data and reduce the incidence of spurious change. DI refers to a data collection technique that incorporates information from prior interview rounds into subsequent waves. While this method is considered an effective remedy for random measurement error, it can also introduce more systematic errors, in particular when respondents are first reminded of their previously provided answer and then asked about their current status. The aim of this paper is to assess the impact of DI on measurement error in employment mobility. We take advantage of a unique experimental situation that was created by the roll-out of dependent interviewing in the Dutch Labour Force Survey (LFS). We apply Hidden Markov Modeling (HMM) to linked LFS and Employment Register (ER) data that cover a period before and after dependent interviewing was abolished, which in turn enables the modeling of systematic errors in the LFS data. Our results indicate that DI lowered the probability of obtaining random measurement error but had no significant effect on the systematic component of the error. The lack of a significant effect might be partially due to the fact that the probability of repeating the same error was extremely high at baseline (i.e when using standard, independent interviewing); therefore the use of DI could not increase this probability any further.

Reconciling Reports: Modelling Employment Earnings and Measurement Errors Using Linked Survey and Administrative Data

We contribute new UK evidence about measurement errors and employment earnings to a field dominated by findings about the USA. We develop and apply new econometric models for linked survey and administrative data that generalize those of Kapteyn and Ypma (Journal of Labor Economics, 2007). Our models incorporate mean-reverting measurement error in administrative data in addition to linkage mismatch and mean-reverting survey measurement error and 'reference period' error, while also allowing error distributions to vary across individuals. Annualised survey earnings underestimate true annual earnings on average. Mean-reversion in survey measurement errors is absent. Both earnings sources underestimate true earnings inequality. The survey earning measure is more reliable than the administrative data earnings measure, but hybrid earnings predictors based on both sources are distinctly more reliable than either source-specific measure. The models with heterogeneous measurement error distributions indicate how data quality may be improved. For example, for survey quality, our results highlight how respondents showing payslips to interviewers have smaller survey error variances. For administrative data, our results suggest that greater error variances are associated with non-standard jobs, private sector jobs, and employers without good payroll systems.

Measurement Error Mechanisms Matter: Agricultural Intensification with Farmer Misperceptions and Misreporting

The mechanism(s) that generate measurement error matter for inference. Survey measurement error is typically thought to represent simple misreporting correctable through improved measurement. But errors might also or alternatively reflect respondent misperceptions that materially affect the respondent decisions under study. We show analytically that these alternate data generating processes imply different appropriate regression specifications and have distinct effects on the bias in parameter estimates. We introduce a simple empirical technique to generate unbiased estimates under more general conditions and to apportion measurement error between misreporting and misperceptions in measurement error when one has both self‐reported and objectively measured observations of the same explanatory variable. We then apply these techniques to the longstanding question of agricultural intensification: Do farmers increase input application rates per unit area as the size of the plots they cultivate decreases? Using nationally representative data from four sub‐Saharan African countries, we find evidence that measurement error in plot size reflects a mixture of farmer misreporting and misperceptions. The results matter for inference around the intensification hypothesis and call into question whether more objective, precise measures are always preferable when estimating behavioral parameters.

How accurately do mothers recall prenatal visits and gestational age? A validation of Uruguayan survey data

Background: Many household surveys collect mothers’ retrospective reports of reproductive, maternal, and child health. However, few empirical exercises assess survey measurement error in these data, based on comparisons with administrative records. Objective: We provide evidence on the accuracy of maternal recall regarding weeks of gestation, premature births, and the timing and number of prenatal visits. Methods: We compare the survey maternal recall and the vital statistics administrative records based on the 2013 Nutrition, Child Development and Health Survey (ENDIS) for Uruguay (2,963 children aged 0‒3). We estimate measurement error and its determinants by using a set of probit models. Results: Mothers tend to overestimate gestational weeks and the incidence of prematurity by 0.1 weeks and 2.4 percentage points, respectively. Differences are larger regarding the timeliness and sufficiency of prenatal visits (respectively, 17.0 and 14.4 pp). Discrepancies are associated with lower educational levels, the length of the recall period (child’s age) and birth order. Conclusions: In general, our findings validate the use of survey data, although the identification of premature births and prenatal care sufficiency presents differences that could lead to errors in the evaluation of compliance with, for example, the United Nations’ Sustainable Development Goals. Since recall accuracy is negatively associated with maternal schooling, discrepancies could be larger in relatively less developed countries. Contribution: The main contribution of this paper lies in the assessment of measurement error levels arising from maternal reports of gestational age and prenatal visits for a relatively short recall period in a Latin American country. Although previous studies estimate measurement errors using administrative records linked to maternal recall data, this is the only study that is based on a nationally representative survey.

The R Package hmi: A Convenient Tool for Hierarchical Multiple Imputation and Beyond

Applications of multiple imputation have long outgrown the traditional context of dealing with item nonresponse in cross-sectional data sets. Nowadays multiple imputation is also applied to impute missing values in hierarchical data sets, address confidentiality concerns, combine data from different sources, or correct measurement errors in surveys. However, software developments did not keep up with these recent extensions. Most imputation software can only deal with item nonresponse in cross-sectional settings and extensions for hierarchical data - if available at all - are typically limited in scope. Furthermore, to our knowledge no software is currently available for dealing with measurement error using multiple imputation approaches. The R package hmi tries to close some of these gaps. It offers multiple imputation routines in hierarchical settings for many variable types (for example, nominal, ordinal, or continuous variables). It also provides imputation routines for interval data and handles a common measurement error problem in survey data: biased inferences due to implicit rounding of the reported values. The user-friendly setup which only requires the data and optionally the specification of the analysis model of interest makes the package especially attractive for users less familiar with the peculiarities of multiple imputation. The compatibility with the popular mice package (Van Buuren and Groothuis-Oudshoorn 2011) ensures that the rich set of analysis and diagnostic tools and post-imputation functions available in mice can be used easily, once the data have been imputed.

What did you Really Earn Last Year?: Explaining Measurement Error in Survey Income Data

Summary The paper analyses the sources of income measurement error in surveys with a unique data set. We use the Austrian 2008–2011 waves of the European Union ‘Statistics on income and living conditions’ survey which provide individual information on wages, pensions and unemployment benefits from survey interviews and officially linked administrative records. Thus, we do not have to fall back on complex two-sample matching procedures like related studies. We empirically investigate four sources of measurement error, namely social desirability, sociodemographic characteristics of the respondent, the survey design and the presence of learning effects. We find strong evidence for a social desirability bias in income reporting, whereas the presence of learning effects is mixed and depends on the type of income under consideration. An Owen value decomposition reveals that social desirability is a major explanation of misreporting in wages and pensions, whereas sociodemographic characteristics are most relevant for mismatches in unemployment benefits.

Heterogeneity in Expectations, Risk Tolerance, and Household Stock Shares: The Attenuation Puzzle

This article jointly estimates the relationship between stock share and expectations and risk preferences. The survey allows individual-level, quantitative estimates of risk tolerance and of the perceived mean, and variance of stock returns. These estimates have economically and statistically significant association for the distribution of stock shares with relative magnitudes in proportion with the predictions of theories. Incorporating survey measurement error in the estimation model increases the estimated associations 2-fold, but they are still substantially attenuated being only about 5% of what benchmark finance theories predict. Because of the careful attention in the estimation to measurement error, the attenuation likely arises from economic behavior rather than errors in variables.

Assessing measurement error in surveys using latent class analysis: application to self-reported illicit drug use in data from the Iranian Mental Health Survey.

Latent class analysis (LCA) is a method of assessing and correcting measurement error in surveys. The local independence assumption in LCA assumes that indicators are independent from each other condition on the latent variable. Violation of this assumption leads to unreliable results. We explored this issue by using LCA to estimate the prevalence of illicit drug use in the Iranian Mental Health Survey. The following three indicators were included in the LCA models: five or more instances of using any illicit drug in the past 12 months (indicator A), any use of any illicit drug in the past 12 months (indicator B), and the self-perceived need of treatment services or having received treatment for a substance use disorder in the past 12 months (indicator C). Gender was also used in all LCA models as a grouping variable. One LCA model using indicators A and B, as well as 10 different LCA models using indicators A, B, and C, were fitted to the data. The three models that had the best fit to the data included the following correlations between indicators: (AC and AB), (AC), and (AC, BC, and AB). The estimated prevalence of illicit drug use based on these three models was 28.9%, 6.2% and 42.2%, respectively. None of these models completely controlled for violation of the local independence assumption. In order to perform unbiased estimations using the LCA approach, the factors violating the local independence assumption (behaviorally correlated error, bivocality, and latent heterogeneity) should be completely taken into account in all models using well-known methods.

Measuring the Measurement Error: A Method to Qualitatively Validate Sensitive Survey Data

Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.

Correction for Measurement Errors in Survey Research: Necessary and Possible

Survey research is the most frequently used data collection method in many disciplines. Nearly, everybody agrees that such data contain serious measurement errors. However, only few researchers try to correct for them. If the measurement errors in the variables vary, the comparison of the sizes of effects of these variables on each other will be wrong. If the sizes of the measurement errors are different across countries, cross-national comparisons of relationships between variables cannot be made. There is ample evidence for these differences in measurement errors across variables, methods and countries (Saris and Gallhofer in Design, evaluation and analysis of questionnaires for survey. Wiley, Hoboken, 2007; Oberski in Measurement errors in comparative surveys. PhD thesis, University of Tilburg, 2011). Therefore, correction for measurement errors is essential for the social sciences. The correction for measurement errors can be made in a simple way, but it requires that the sizes of the error variances are known for all observed variables. Many experiments are carried out to determine the quality of questions. The relationship between the quality and the characteristics of the questions has been studied. Because this relationship is rather strong, one can also predict the quality of new questions. A program SQP has been developed to predict the quality of questions. Using this program, the quality of the questions (complement of error variance) can be obtained for nearly all questions measuring subjective concepts. For objective variables, other research needs to be used (e.g., Alwin in Margins of error: a study of reliability in survey measurement. Wiley, Hoboken, 2007). Using these two sources of information, making correction for measurement error in survey research is possible. We illustrate here that correction for measurement errors can and should be performed.

Returns to Education in India

Summary In India both men and women with more education live in households with greater consumption per capita. Yet aggregating across age cohorts and states, an extra year of education brings male cohorts only 4% more consumption and provides no additional consumption for female cohorts. This result is robust to: (1) accounting for survey measurement error, (2) different measures of household consumption and composition, (3) allowing returns to differ by state and school quality, and (4) age misreporting. The only area with substantial returns is entering into regular wage work which still employs only a small fraction of the population.

Measurement Error in Income and Schooling, and the Bias of Linear Estimators

The characteristics of measurement error determine the bias of linear estimators. We propose a method for validating economic survey data allowing for measurement error in the validation source, and we apply this method by validating Survey of Health, Ageing and Retirement in Europe (SHARE) data with Danish administrative registers. We find that measurement error in surveys is classical for annual gross income but non-classical for years of schooling, causing a 21% amplification bias in IV estimators of returns to schooling. Using a 1958 Danish schooling reform, we contextualize our result with an estimate of the income returns to schooling.

CHILD MENTAL HEALTH AND EDUCATIONAL ATTAINMENT: MULTIPLE OBSERVERS AND THE MEASUREMENT ERROR PROBLEM

SUMMARY We examine the effect of survey measurement error on the empirical relationship between child mental health and personal and family characteristics, and between child mental health and educational progress. Our contribution is to use unique UK survey data that contain (potentially biased) assessments of each child's mental state from three observers (parent, teacher and child), together with expert (quasi-)diagnoses, using an assumption of optimal diagnostic behaviour to adjust for reporting bias. We use three alternative restrictions to identify the effect of mental disorders on educational progress. Maternal education and mental health, family income and major adverse life events are all significant in explaining child mental health, and child mental health is found to have a large influence on educational progress. Our preferred estimate is that a one-standard-deviation reduction in ‘true’ latent child mental health leads to a 2- to 5-month loss in educational progress. We also find a strong tendency for observers to understate the problems of older children and adolescents compared to expert diagnosis. © 2015 The Authors. Journal of Applied Econometrics published by John Wiley & Sons, Ltd.

The Use of Accuracy Indicators to Correct for Survey Measurement Error

SummaryAn accuracy indicator is an observed variable which is related to the size of measurement error. Basic and extended models are introduced to represent the properties of a binary accuracy indicator. Under specified assumptions, it is shown that an accuracy indicator can identify a measurement error model. An approach to estimating a distribution function is presented together with methodology for variance estimation. The approach is applied to data on earnings from the British Household Panel Survey, where the accuracy indicator is whether or not a payslip is observed. A validation study provides justification for the modelling assumptions.

PRM196 - How low can we go?--making meaningful inferences from small samples

To explore the impact of inferences from very samples on the outcome of management decisions. In many cases management has some prior belief about the states of nature. We explore the potential advantage of incorporating Bayesian inference to improve the confidence in managerial decisions based on small samples.. Traditionally, survey researchers reconcile differences between survey results and prior beliefs by citing the uncertainty reflected in the sampling error or looking for other explanatory factors (such as possible survey measurement error). The Bayesian approach integrates the different sources of information (i.e., prior belief and observed survey results) to arrive at the most probable estimate. In full realization, a Bayesian approach considers not just the probability that “truth” lies outside some range of values but seeks to estimate the probability of each of many possible hypotheses, given the data was that obtained. Using responses to a choice-based conjoint exercise that was embedded in an online survey of approximately 700 individuals, we created a series of samples of different sizes using different restrictions to reflect the ways in which both probability and convenience samples might be generated. We drew multiples of ten random samples of 25, 50, 75, 100, 150, 225 and 450 from our “population” of 897 respondents, resulting in 70 individual samples. We estimated HB models for each sample (using Sawtooth Software’s CBC-HB program). Simulated choice probabilities--a key output of discrete choice models--stabilize across samples starting with n=75. For smaller samples, decision confidence can be increased using Bayesian inference and bootstrapping methods. Meaningful inferences---and hence decisions--can be made with smaller sample sizes by utilizing Bayesian inference and methods such as bootstrapping to better estimate the degree of uncertainty in the data.

Latent class analysis of response inconsistencies across modes of data collection

Latent class analysis (LCA) has been hailed as a promising technique for studying measurement errors in surveys, because the models produce estimates of the error rates associated with a given question. Still, the issue arises as to how accurate these error estimates are and under what circumstances they can be relied on. Skeptics argue that latent class models can understate the true error rates and at least one paper (Kreuter et al., 2008) demonstrates such underestimation empirically. We applied latent class models to data from two waves of the National Survey of Family Growth (NSFG), focusing on a pair of similar items about abortion that are administered under different modes of data collection. The first item is administered by computer-assisted personal interviewing (CAPI); the second, by audio computer-assisted self-interviewing (ACASI). Evidence shows that abortions are underreported in the NSFG and the conventional wisdom is that ACASI item yields fewer false negatives than the CAPI item. To evaluate these items, we made assumptions about the error rates within various subgroups of the population; these assumptions were needed to achieve an identifiable LCA model. Because there are external data available on the actual prevalence of abortion (by subgroup), we were able to form subgroups for which the identifying restrictions were likely to be (approximately) met and other subgroups for which the assumptions were likely to be violated. We also ran more complex models that took potential heterogeneity within subgroups into account. Most of the models yielded implausibly low error rates, supporting the argument that, under specific conditions, LCA models underestimate the error rates.

Child Mental Health and Educational Attainment: Multiple Observers and the Measurement Error Problem

We examine the effect of survey measurement error on the empirical relationship between child mental health and personal and family characteristics, and between child mental health and educational progress. Our contribution is to use unique UK survey data that contains (potentially biased) assessments of each child's mental state from three observers (parent, teacher and child), together with expert (quasi-) diagnoses, using an assumption of optimal diagnostic behaviour to adjust for reporting bias. We use three alternative restrictions to identify the effect of mental disorders on educational progress. Maternal education and mental health, family income, and major adverse life events, are all significant in explaining child mental health, and child mental health is found to have a large influence on educational progress. Our preferred estimate is that a 1-standard deviation reduction in 'true' latent child mental health leads to a 2-5 months loss in educational progress. We also find a strong tendency for observers to understate the problems of older children and adolescents compared to expert diagnosis.

Modeling Sources of Self-report Bias in a Survey of Drug Use Epidemiology

Well-documented errors in the reporting of drug-related behaviors have been attributed to several sources. These include: 1) respondent difficulties in understanding survey questions, 2) problems in recalling the information necessary to accurately answer these questions, and 3) social pressures that discourage accurate reporting. We report covariance structure models designed to simultaneously evaluate each of these potential sources of error. Data examined are from a community survey of 627 Chicago adults which collected drug use self reports (via ACASI technology), multiple biological samples (including hair, urine, and saliva) that permit self-report validation, and a series of probes designed to collect systematic information regarding respondent comprehension and memory difficulties and social desirability concerns. These three sets of information were employed to construct latent variable covariance structure models that enabled an evaluation of the effects of each potential source of reporting error on the quality of drug use reporting. Social desirability concerns were predictive of discordant drug use reporting and drug use under-reporting. Memory difficulties were predictive of drug use over-reporting. Differences in the predictive power of these variables were found across race/ethnic groups. Both memory difficulties and social desirability concerns are independent sources of measurement error in surveys of drug use epidemiology.