Survey Error Research Articles

Meta-Analyses in Survey Methodology: A Systematic Review

Meta-analytic techniques have become the standard methods for aggregating the results from thematically related studies in the behavioral, health, and economic sciences. To analyze the state of the art of using meta-analyses in survey methodology, previous meta-analyses are systematically identified and classified according to the thematic areas the analyses address. This is followed by identifying gaps in research (i.e., areas where there are few or no existing meta-analyses) and investigating potential avenues for future meta-analyses in this field. The findings are based on a systematic search of two bibliographic harvesters (together covering 265 bibliographic databases), which yielded 54 eligible manuscripts reporting 60 meta-analytic studies and 91 effect sizes. To identify the thematic areas, the effect sizes are structured according to seven categories of the total survey error (TSE). Characteristics of primary studies that potentially influence the variability of meta-analytic findings also are considered, such as the survey mode, questionnaire design, and sample characteristics. The results show that the thematic areas of these meta-analyses cover only two of the seven TSE categories: measurement and nonresponse error. Thematic areas in the remaining TSE categories are not covered, and the key practical implication is that gaps exist in current research. Regarding the methodology of existing meta-analyses in the survey methodology field, there are fundamental differences in reporting styles in terms of transparency and replicability, calling for a concerted effort to develop meta-analytic reporting standards for survey methodology.

Improving the Quality of Survey Data Documentation: A Total Survey Error Perspective

Surveys are a common method in the social and behavioral sciences to collect data on attitudes, personality and social behavior. Methodological reports should provide researchers with a complete and comprehensive overview of the design, collection and statistical processing of the survey data that are to be analyzed. As an important aspect of open science practices, they should enable secondary users to assess the quality and the analytical potential of the data. In the present article, we propose guidelines for the documentation of survey data that are based on the total survey error approach. Considering these guidelines, we conclude that both scientists and data-holding institutions should become more sensitive to the quality of survey data documentation.

Creating Improved Survey Data Products Using Linked Administrative-Survey Data

AbstractRecent research linking administrative to survey data has laid the groundwork for improvements in survey data products. However, the opportunities have not been fully realized yet. In this article, our main objective is to use administrative-survey linked microdata to demonstrate the potential of data linkage to reduce survey error through model-based blended imputation methods. We use parametric models based on the linked data to create imputed values of Medicaid enrollment and food stamp (SNAP) receipt. This approach to blending data from surveys and administrative data through models is less likely to compromise confidentiality or violate the terms of the data sharing agreements among the agencies than releasing the linked microdata, and we demonstrate that it can yield substantial improvements of estimate accuracy. Using the blended imputation approach reduces root mean squared error (RMSE) of estimates by 81 percent for state-level Medicaid enrollment and by 93 percent for substate area SNAP receipt compared with estimates based on the survey data alone. Given the high level of measurement error associated with these important programs in the United States, data producers should consider blended imputation methods like the ones we describe in this article to create improved estimates for policy research.

Low survey response! Can I still use the data?

ABSTRACT Natural resource agencies often use mail surveys to collect stakeholder information. A major concern of mail surveys, however, has long been relatively low response rates compared to telephone or face-to-face interviews. Survey research has largely focused on achieving high response rates; however, in some situations even well-designed surveys can have low response rates. We present an example of a three-page (25 questions) survey measuring opinions and attitudes about native fish management in the South Dakota Black Hills region that received a relatively low response rate (21%) using a mailing, postcard reminder, and second mailing of the questionnaire. We compared response rate and data quality of a third mailing of the full questionnaire with a one-page (five questions) questionnaire measuring key variables to evaluate possible nonresponse bias. Within the total survey error (TSE) paradigm we provide evidence that reliable and useful information was collected by this survey.

Моделирование трассы железнодорожного пути по результатам мобильного лазерного сканирования

Objective: To develop a mathematical model of the railroad track based on the initial progressive-type data (laser scanning) in railroad design. Methods: Regression analysis (least-square method), as well as coordinate methods of calculating point position in space were applied. Results: The mathematical model, which could describe the position of the railroad track in three-dimensional space by means of mathematical relations, was obtained. Applicability of approximating models was established. The models make it possible to provide smoothing of laser survey data. Regularization and globalization algorithms of initial data were developed. Practical importance: The introduced model is universal when describing the position of the track at all stages of life cycle of the railway line. It is reasonable to apply the presented model in design engineering in order to balance survey errors, maintain the track in coordinates, as well as to calculate design and profile parameters.

Monitoring topographic changes through 4D-structure-from-motion photogrammetry: application to a debris-flow channel

The study of fast geomorphic changes in mountain channels and hillslopes, driven by intense geomorphic processes, requires frequent and detailed topographic surveys. In the last two decades, high-resolution topography (HRT) has provided new opportunities in the Earth Sciences. These have benefited from important developments in surveying techniques, methods, sensors, and platforms. Between these, the application of structure-from-motion (SfM) photogrammetry has become a widely used method to acquire HRT and high-resolution orthomosaics at multiple temporal and spatial scales. SfM photogrammetry has revolutionized the possibility to collect multi-temporal HRT in rugged or inaccessible environments like that observed in debris-flow catchments. However, appropriate workflows incorporating survey planning, data acquisition, post-processing, and error and uncertainty assessment are required, especially when multi-temporal surveys are compared to study topographic changes through time. In this paper, we present a workflow to acquire and process HRT. The workflow was applied in a debris-flow channel of the Moscardo Torrent (Eastern Italian Alps). Due to the topographic complexity of the study area, the SfM surveys were carried out integrating photos obtained from an unmanned aerial vehicle and from the ground. This integration guarantees high data density and avoids shadows. Eight photogrammetric surveys were collected between December 2015 and August 2017. In this time interval, five debris flows occurred. The surveying and data processing procedure described in the workflow permitted to summarize and integrate all the analysis steps and helped to identify and minimize potential sources of error in the multi-temporal SfM data (what we consider here 4D). Our case study demonstrates how the developed workflow presented here allows studying the geomorphic effects of debris flows and check dams functionality in mountain environments.

Accounting for Surveyor Effort in Large-Scale Monitoring Programs

Abstract Accounting for errors in wildlife surveys is necessary for reliable status assessments and quantification of uncertainty in estimates of population size. We apply a hierarchical log-linear Poisson regression model that accounts for multiple sources of variability in count data collected for the Integrated Waterbird Management and Monitoring Program during 2010–2014. In some large-scale monitoring programs (e.g., Christmas Bird Count) there are diminishing returns in numbers counted as survey effort increases; therefore, we also explore the need to account for variable survey duration as a proxy for effort. In general, we found a high degree of concordance between counts and effort-adjusted estimates of relative abundance from the Integrated Waterbird Management and Monitoring Program (x̄difference = 0.02%; 0.25% SD). We suggest that the model-based adjustments were small because there is only a weak asymptotic relationship with effort and count. Whereas effort adjustments are reasonable and effective when applied to count data from plots of standardized area, such adjustments may not be necessary when the area of sample units is not standardized and surveyor effort increases with number of birds present. That is, large units require more effort only when there are many birds present. The general framework we implemented to evaluate effects of varying survey effort applies to a wide variety of wildlife monitoring efforts.

Translation of country-specific programs and survey error: Measuring the education level of immigrants

Translation of country-specific programs and survey error: Measuring the education level of immigrants

Complex-conductivity imaging for the understanding of landslide architecture

Geophysical methods are widely used for landslide investigation to delineate depth and geometry of the sliding plane. In particular, electrical resistivity tomography (ERT) is often used because both porosity and water saturation control the electrical resistivity of the subsurface materials and are critical for slope stability. Moreover, ERT can be employed to monitor changes in pore-fluid pressure which is an important factor triggering landslides. However, the interpretation of ERT results in clay-rich landslides can be challenging considering that high electrical conductivity values may not only be related to an increase in saturation but also to the surface conduction mechanism, which becomes dominant in the presence of clays. Recently, environmental investigations have demonstrated an improved subsurface characterization through induced polarization (IP) imaging, an extension of the ERT method, which permits to gain information about electrical conductive and capacitive (i.e., polarization effect) properties of the subsurface. As the polarization effect is mainly controlled by surface charge, which is large in clays, IP images are expected to improve the lithological interpretation and overcome the limitations of the ERT method. Additionally, measurements collected over a broad frequency bandwidth, the so-called spectral IP (SIP), have been successfully used in laboratory experiments to quantify textural and hydrogeological parameters. However, the application of SIP field measurements for the delineation of hydrogeological structures in landslides has not been addressed to date. To fill this gap, in this study we present SIP imaging results for data collected at the La Valette landslide (South East French Alps), where an existing geotechnical model of the landslide is available for evaluation. Moreover, our study provides a detailed revision on the collection and processing of SIP datasets, as well as a description of the diverse sources of error in IP surveys, to stress the importance of data-error quantification for a quantitative application of the SIP method. Our results demonstrate that adequate data processing allows obtaining consistent results at different frequencies and independently of the measuring protocol. Furthermore, the frequency dependence of the complex conductivity obtained in the field-scale SIP survey is consistent with earlier laboratory experiments. In conclusion, our study shows the potential of the SIP method to improve our understanding of subsurface properties, and an improved delineation of the contact between the mobilized material and the bedrock as well as variations in the clay content within the landslide and the bedrock.

The Use of Response Propensity Modeling (RPM) for Allocating Differential Survey Recruitment Strategies: Purpose, Rationale, and Implementation

Response Propensity Modeling (RPM) is an empirical process that identifies a multivariate statistical model to predict the likelihood (propensity) that a given element in an initial sample will cooperate with a forthcoming survey request. This predicted probability ranges from 0 to 1 and reflects an element’s unique combination of characteristics that are expected to affect the relative likelihood of obtaining a response from the element. RPM is consistent with Leverage-Salience Theory in that it is used to tailor different recruitment strategies to different sampled elements, rather than using a one-size-fits-all (OSFA) approach whereby all sampled elements receive the same recruitment strategies. The appeal of using an RPM approach to allocating differential recruitment strategies is that, in theory, it should perform better than an OSFA approach, in terms of gaining a higher overall response rate, gaining a more representative unweighted final sample, and being more cost-effective. As described here, RPM fits into a Tailored Design Method of trying to most efficaciously utilize total survey costs to reduce total survey error. We explain what we mean by RPM, compare it to past work on allocating differential recruitment strategies within an initial sample, and explain the three-stage process used to identify, implement/test, and refine the response propensity model.

Detection errors in wildlife abundance estimates from Unmanned Aerial Systems (UAS) surveys: Synthesis, solutions, and challenges

AbstractUnmanned aerial systems (UAS) are emerging as an accessible and versatile tool for ecologists, promising to revolutionize the way abundance and distribution data are obtained in wildlife studies. Establishment ofUASas an efficient and reliable tool demands understanding how detection errors influenceUAS‐derived counts and possible solutions to address them.We describe two types of false‐negative errors (availability and perception errors) and two types of false‐positive errors (misidentification and double count) that may bias abundance estimates fromUASsurveys. Then, we discuss available methods to address detection errors inUASsurveys and point out challenges for future developments. We present hierarchical models as an integrative framework to account for multiple detection errors and datasets inUASabundance modelling.Methods to address detection errors inUASsurveys depend on how data are collected (flight plan, images processing, and reviewing procedure). Conventional aerial surveys literature offers a set of solutions, especially to deal with false‐negative errors. Available auxiliary information (such as ground counts and telemetry data) facilitates estimating detection errors, although the versatility ofUASpermits exploring novel approaches. Solutions involve planning separated strip transects, temporally replicating flights, carrying out counts in orthomosaics, and multiple observer protocol. When automatic image review is used, subsample manual reviewing, trial experiments, and semiautomated procedures might deal with algorithm errors.UASsurveys need to be consciously planned, thinking on what kind of errors can significantly affect counts and the use of raw counts and indices should be avoided. Approaches that formally account for false positives are needed, particularly for double counts. Hierarchical modelling (especiallyN‐mixture models) offers a fruitful framework to explore and combine solutions, integrating multiple datasets and accommodating different detection errors.

Enhancing the collection and analysis of survey and administrative data by linking

BackgroundLinking survey and administrative data can enhance their collection and analysis. By linking the issued sample to administrative data, the representativeness of a survey can be assessed during and at the end of data collection. An understanding of the (lack of) representativeness can inform adaptive and responsive survey designs to improve representativeness and/or reduce survey costs. It can also aid the assessment of potential survey non-response bias and the implementation of strategies to adjust for such bias during analysis. Comparison of responses to the same question in two linked datasets can help identify problematic questions. This information can then be used to improve question wording to reduce measurement error in future surveys and to account for such error during analysis.
 ObjectivesTo assess how the representativeness of a survey changes after each attempt to contact the interviewees, and to assess the nature and extent of differential reporting between a face-to-face survey and the self-completion 2011 UK Census.
 Methods (including data)We present our recent research developing representativeness indicators based on the coefficient of variation of the response propensities calculated after each attempt to contact the interviewees and illustrate their use using Labour Force Survey (LFS) data linked to contemporaneous data from the 2011 Census. We also describe our work on differential reporting between the LFS and the Census, focussing on the reporting of highest qualification and ethnicity.
 Findings and ConclusionsOur results suggest that considerable savings can be made by reducing the number of attempts to obtain a response without a detrimental effect on representativeness and that face-to-face interviews obtain more accurate information than self-completion questionnaires.

Should statin guidelines consider patient preferences? Eliciting preferences of benefit and harm outcomes of statins for primary prevention of cardiovascular disease in the sub-Saharan African and European contexts

BackgroundPatient preferences are key parameters to evaluate benefit-harm balance of statins for primary prevention but they are not readily available to guideline developers and decision makers. Our study aimed to elicit patient preferences for benefit and harm outcomes related to use of statins for primary cardiovascular disease prevention and to examine how the preferences differ across economically and socio–culturally different environments.MethodsWe conducted preference-eliciting surveys using best-worst scaling designed with a balanced incomplete-block design (BIBD) on 13 statins-related outcomes on 220 people in Ethiopia and Switzerland. The participants made tradeoff decisions and selected the most and least worrisome outcomes concurrently from each scenario generated using the BIBD. The design yielded 34,320 implied paired-comparisons and 2860 paired-responses as unit of analysis for eliciting the preferences that were analyzed using a conditional-logit model on a relative scale and surface under the cumulative ranking curve from multivariate random-effects meta-analysis model on a scale of 0 to 1.ResultsThere was high internal consistency of responses and minimal amount of measurement error in both surveys. Severe stroke was the most worrisome outcome with a ceiling preference of 1 (on 0 to 1 scale) followed by severe myocardial infarction, 0.913 (95% CI, 0.889–0.943), and cancer, 0.846 (0.829–0.855); while treatment discontinuation, 0.090 (0.023–0.123), and nausea/headache, 0.060 (0.034–0.094) were the least worrisome outcomes. Preferences were similar between Ethiopia and Switzerland with overlapping uncertainty intervals and concordance correlation of 0.97 (0.90–0.99).ConclusionsOur study provides much needed empirical evidence on preferences that help clinical guidelines consider for weighing the benefit and harm outcomes when recommending for or against statins for primary prevention of cardiovascular disease. The preferences are consistent across the disparate settings; however, we recommend inclusion of more countries in future studies to ensure the generalizability of the preferences to all environments.

Disentangling Bias and Variance in Election Polls

ABSTRACTIt is well known among researchers and practitioners that election polls suffer from a variety of sampling and nonsampling errors, often collectively referred to as total survey error. Reported margins of error typically only capture sampling variability, and in particular, generally ignore nonsampling errors in defining the target population (e.g., errors due to uncertainty in who will vote). Here, we empirically analyze 4221 polls for 608 state-level presidential, senatorial, and gubernatorial elections between 1998 and 2014, all of which were conducted during the final three weeks of the campaigns. Comparing to the actual election outcomes, we find that average survey error as measured by root mean square error is approximately 3.5 percentage points, about twice as large as that implied by most reported margins of error. We decompose survey error into election-level bias and variance terms. We find that average absolute election-level bias is about 2 percentage points, indicating that polls for a given election often share a common component of error. This shared error may stem from the fact that polling organizations often face similar difficulties in reaching various subgroups of the population, and that they rely on similar screening rules when estimating who will vote. We also find that average election-level variance is higher than implied by simple random sampling, in part because polling organizations often use complex sampling designs and adjustment procedures. We conclude by discussing how these results help explain polling failures in the 2016 U.S. presidential election, and offer recommendations to improve polling practice.

Applying the total survey error framework to PIAAC

PurposeSurveys in multinational, multiregional and multicultural contexts (or “3MC” surveys) are becoming increasingly important to global and regional decision-making and theory building. To serve this purpose, the surveys need to be well managed, with an awareness of key sources of survey error and how to minimize them, mechanisms in place to control the implementation process and an ability to intervene in that process when necessary in a spirit of continuous improvement (Pennell et al., 2017). One key approach for managing and assessing the quality of 3MC surveys is the total survey error (TSE) framework and associated survey process quality. This paper aims to examine the application of the TSE framework and survey process quality to the Programme for the International Assessment of Adult Competencies (PIAAC).Design/methodology/approachThe authors begin with a background on TSE and discuss recent adaptations of TSE and survey process quality for 3MC surveys. They then presents a TSE framework tailored with examples of potential contributions to error for PIAAC and ways to address those through effective quality assurance (QA) and quality control (QC) approaches.FindingsOverall, the authors find that the design and implementation of the first cycle of PIAAC largely reflect the current best practice for 3MC surveys. However, the authors identify several potential contributions to error that may threaten comparability in PIAAC and ways these could be addressed in the upcoming cycle.Originality/valueWith a view toward continuous improvement, the final section draws on the survey process quality approach adapted from Hansen et al.’s study (2016) to summarize the recommendations in terms of additional QA elements (inputs and activities) and associated QC elements (measures and reports) for PIAAC’s consideration in the next cycle.

Detecting and treating errors in tests and surveys

PurposeSurveys that include skill measures may suffer from additional sources of error compared to those containing questionnaires alone. Examples are distractions such as noise or interruptions of testing sessions, as well as fatigue or lack of motivation to succeed. This paper aims to provide a review of statistical tools based on latent variable modeling approaches extended by explanatory variables that allow detection of survey errors in skill surveys.Design/methodology/approachThis paper reviews psychometric methods for detecting sources of error in cognitive assessments and questionnaires. Aside from traditional item responses, new sources of data in computer-based assessment are available – timing data from the Programme for the International Assessment of Adult Competencies (PIAAC) and data from questionnaires – to help detect survey errors.FindingsSome unexpected results are reported. Respondents who tend to use response sets have lower expected values on PIAAC literacy scales, even after controlling for scores on the skill-use scale that was used to derive the response tendency.Originality/valueThe use of new sources of data, such as timing and log-file or process data information, provides new avenues to detect response errors. It demonstrates that large data collections need to better utilize available information and that integration of assessment, modeling and substantive theory needs to be taken more seriously.

Response Errors in Surveys of Defensive Gun Use: A National Internet Survey Experiment

A national Internet survey of a probability sample of 5,550 U.S. adults was used to study possible sources of error in surveys of defensive gun use (DGU). Respondents (Rs) were randomly exposed to variant question wordings, question sequences, and combinations of questions. Rs were 70% more likely to report a victimization when they were instructed to report incidents involving offenders known to them, and 43% more likely to report a victimization if they were instructed to include incidents that resulted in no injury or property loss. Rs were 125% more likely to report DGUs if they were directly asked about DGU than if they were first asked about victimization experiences, then asked about DGU in connection with those experiences.

Personal Resilience as a Predictor of Professional Development Engagement and Career Satisfaction of Agriscience Teachers

The profession of teaching is inherently stressful. Resilience, or the ability to adapt and develop competence despite exposure to disruptive change, has been shown to be beneficial for teachers. The purpose of this study was to determine if personal resilience is a predictor of professional development engagement and career satisfaction of agriscience teachers. A census of agriscience teachers in four purposefully selected states was taken. Data were collected using the TailoredDesign Method, using multiple points of contact with multiple modes to minimize survey error. The overall response rate was 72.5% (n = 892). The linear combination of independent variables in a stepwise backwards regression model explained 13.7% of the variance of professional development engagement and 21.4% of the variance of career satisfaction. These findings suggested that increasing the resilience of agriscience teachers, specifically in the areas of positive: world and focused, could lead to increased engagement in professional development and career satisfaction, which has been shown to be a factor that increases teacher efficacy. Additional research is needed to explore how to increase the resilience of agriscience teachers and the relationship between teacher resilience and student outcome variables.

Simultaneous Estimation of Multiple Sources of Error in a Smartphone-Based Survey

Simultaneous Estimation of Multiple Sources of Error in a Smartphone-Based Survey

Factor Structural Time Series Models for Official Statistics with an Application to Hours Worked in Germany

Abstract We introduce a high-dimensional structural time series model, where co-movement between the components is due to common factors. A two-step estimation strategy is presented, which is based on principal components in differences in a first step and state space methods in a second step. The methods add to the toolbox of official statisticians, constructing timely regular statistics from different data sources. In this context, we discuss typical measurement features such as survey errors, statistical breaks, different sampling frequencies and irregular observation patterns, and describe their statistical treatment. The methods are applied to the estimation of paid and unpaid overtime work as well as flows on working-time accounts in Germany, which enter the statistics on hours worked in the national accounts.