Survey Error Research Articles

Attention! The meanings of attention to politics in surveys

This research note explores the role of reported attention to politics in survey overclaiming about politics. Using recognition of real and fictitious political parties in the context of the UK's 2019 European Parliament elections, we find that people who report higher attention to politics are more likely to over-report recognition of fictitious parties, and are also more likely to recognize new real political parties - those that emerged around the issue of ‘Brexit’ in the months before the election. To resolve these patterns, we show that political attention makes little difference to the accuracy of responses for people who have high political knowledge, or if it does so, it increases accuracy. However, for those with lower political knowledge, high reported political attention is a source of potential survey error and bias. These findings are consistent with higher survey satisficing and norm compliance among those who report having greater attention to politics, particularly among those who have lower knowledge. The implications are important for understanding the meaning and consequences of ‘attention to politics’ in surveys.

A restricted multiple generator approach to enumerate personal support networks: An alternative to global important matters and satisficing in web surveys

Overlooked issues in the selection and presentation of name generators may lead to extreme measurement errors in Web surveys. This paper compares networks elicited from the standard “important matters” (IM) name generator, which records alter names on one or five boxes per page, with a two-generator design and an alternative approach, the R5D. The R5D consists of five name generators focused on a range of topics commonly discussed as important matters. Each generator is restricted to recording one alter. Generally accepted practices in egocentric surveys result in inaccurate measures of network size and composition. They include the use of the stand-alone IM generator and advice to present a single name box to record one alter name per page. The use of a single name box encourages satisficing. The R5D is a parsimonious alternative with stronger construct validity. It has less measurement error, provides measures of social support, and enumerates alters with less slippage in importance than other approaches.

A mixed-methods study to inform the clarity and accuracy of cannabis-use and cannabis-tobacco co-use survey measures

BackgroundLegalization of medical and recreational cannabis in US states has been accompanied by increases in availability, acceptability, and diversity in methods of cannabis use, as well as an increase in devices and methods for cannabis-tobacco co-use. Updated and specific survey measures of cannabis and cannabis-tobacco co-use are needed. MethodsWe employed a mixed-methods approach to identify sources of specification and measurement error in cannabis and cannabis-tobacco co-use measures. We surveyed and interviewed 36 young adult (age 18–29) cannabis and tobacco co-users in California (2017-2018), triangulated with document analysis of online cannabis websites and forums. We investigated how survey reports of cannabis use and cannabis-tobacco co-use compared to narrative descriptions provided during in-depth interviews. We identify key strategies for researchers collecting self-reported survey data to enhance accuracy and comprehensiveness of measures. FindingsPotential sources of survey error included: broad variation in cannabinoid content and concentration, inconsistent interpretation of questions between participants and researchers (e.g. blunts were not considered co-use), and substantial variation in dosage within and between products. No evidence of survey recall bias or response editing was detected. ConclusionTo enhance survey accuracy, we recommend surveys specify which cannabis delivery methods and forms are included and excluded in each measure, differentiate between cannabis products, and explicitly include or exclude CBD product use.

Analysis of Draught Survey Errors by Extended Fuzzy Analytic Hierarchy Process

Analysis of Draught Survey Errors by Extended Fuzzy Analytic Hierarchy Process

Qualitative and quantitative methods show stability in patterns of Cepaea nemoralis shell polymorphism in the Pyrenees over five decades.

Over the past century, the study of animal color has been critical in establishing some of the founding principles of biology, especially in genetics and evolution. In this regard, one of the emerging strengths of working with the land snail genus Cepaea is that historical collections can be compared against modern‐day samples, for instance, to understand the impact of changing climate and habitat upon shell morph frequencies. However, one potential limitation is that prior studies scored shell ground color by eye into three discrete colours yellow, pink, or brown. This incurs both potential error and bias in comparative surveys. In this study, we therefore aimed to use a quantitative method to score shell color and evaluated it by comparing patterns of C. nemoralis shell color polymorphism in the Pyrenees, using both methods on present‐day samples, and against historical data gathered in the 1960s using the traditional method. The main finding was that while quantitative measures of shell color reduced the possibility of error and standardized the procedure, the same altitudinal trends were recovered, irrespective of the method. The results also showed that there was a general stability in the local shell patterns over five decades, including altitudinal clines, with just some exceptions. Therefore, although subject to potential error human scoring of snail color data remains valuable, especially if persons have appropriate training. In comparison, while there are benefits in taking quantitative measures of color in the laboratory, there are also several practical disadvantages, mainly in terms of throughput and accessibility. In the future, we anticipate that genomic methods may be used to understand the potential role of selection in maintaining shell morph clines. In addition, photographs generated by citizen scientists conducting field surveys may be used with deep learning‐based methods to survey color patterns.

Nonsampling error in vegetation surveys: understanding error types and recommendations for reducing their occurrence

Observer error is ubiquitous in vegetation sampling. Observer error, along with other types of related nonsampling error, may result in species richness being artificially underestimated (i.e., false-negative errors) or artificially overestimated (i.e., false-positive errors). Because of the manner in which observer error is usually quantified, there exists a strong bias against the discovery of false positives. At least seven different types of nonsampling errors can occur when surveying vegetation species composition: overlooking, misidentification, cautious, mythical, anecdotal, transcription, and relocation. Six of these error types can result in false negatives and five can result in false positives. Another type of observer error that can occur in plant surveys is estimation error, which occurs when abundances are not accurately estimated. There are many potential underlying causes of nonsampling error. Humans observers, even when highly trained and experienced, are prone to commit errors through slips, lapses, and mistakes. A number of contributing factors of observer error have been identified, including characteristics associated with the vegetation, the environment, and the observers themselves; design-based flaws may also occur. Although it may not be possible to eliminate all sources of nonsampling error, most can be reduced through understanding the mechanisms underlying the various types of error, followed by training exercises and the consistent use of appropriate operating procedures.

Questionnaire surveys - sources of error and implications for design, reporting and appraisal.

Questionnaire surveys of dentists are a commonly used research method which can yield important results. Nevertheless, in order to give reliable information, surveys must be carefully designed to avoid bias. The challenge of planning, designing and implementing a questionnaire study is often underestimated. While headline response rate is commonly used as an indication of survey quality, there are four potential areas of survey error. These are coverage, sampling, measurement and response. These four areas of error are discussed, with examples from the current literature, to assist readers in critically appraising questionnaire studies. We make 12 key recommendations to researchers to help avoid bias in their research; approaches to response enhancement are also considered. Reporting guidelines for questionnaire studies are suggested which improve transparency and assist in understanding of research methodologies and results. We have termed these guidelines: Reporting Items for Surveys and Questionnaires (RISQ). In conclusion, there is no single solution that can guarantee a successful study. Conversely, failure to pay careful attention to any one of many crucial aspects of survey design can be the downfall of an otherwise robust study.

Estimation Under Mode Effects and Proxy Surveys, Accounting for Non-ignorable Nonresponse

We propose a new, model-based methodology to address two major problems in survey sampling: The first problem is known as mode effects, under which responses of sampled units possibly depend on the mode of response, whether by internet, telephone, personal interview, etc. The second problem is of proxy surveys, whereby sampled units respond not only about themselves but also for other sampled. For example, in many familiar household surveys, one member of the household provides information for all other members, possibly with measurement errors. Ignoring the existence of mode effects and/or possible measurement errors in proxy surveys could result in possible bias in point estimators and subsequent inference. Our approach accounts also for nonignorable nonresponse. We illustrate the proposed methodology by use of simulation experiments and real sample data, with known true population values.

SISTEM TES BERBASIS KOMPUTER UNTUK SELEKSI MITRA BADAN PUSAT STATISTIK

Dengan misi menyediakan data statistik berkualitas, data yang disediakan oleh Badan Pusat Statistik harus mempunyai total survey error yang kecil. Total survey error mencakup sampling error dan nonsampling error, dengan nonsampling error yang berperan lebih besar. Dalam survei atau sensus yang dilaksanakan oleh BPS, nonsampling error ini sangat dipengaruhi oleh kualitas petugas lapangan yang direkrut oleh BPS dan biasa disebut dengan istilah “mitra statistik”. Untuk mendapatkan calon mitra statistik yang berkualitas baik perlu melalui proses uji penilaian atau uji kelayakan menjadi seorang calon mitra. Proses uji penilaian saat ini masih menggunakan sistem ujian Paper and Pencil Test (PPT). Tes dengan metode PPT ini memiliki beberapa permasalahan diantaranya dari sisi keamanan soal dan transparansi nilai peserta. Penelitian ini bertujuan untuk mengembangkan sebuah sistem tes berbasis komputer untuk mengatasi permasalahan pada sistem ujian saat ini. Metode pengembangan sistem menggunakan metode modified waterfall. Hasil dari penelitian adalah sebuah sistem tes berbasis komputer bernama SICATMiS yang dikembangkan dari OSS TCExam. Sistem memiliki 3 aktor dan 5 menu utama yang dapat berjalan di luar jaringan (online) maupun di dalam jaringan local. Sistem yang dikembangkan dievaluasi dengan dengan uji scenario, uji penerimaan pengguna, dan system usability scale. Hasil uji skenario menunjukkan dari 47 skenario yang dilakukan, semua fitur berhasil dijalankan dengan sempurna. Dari 14 pernyataan dalam uji penerimaan pengguna, 68% responden sangat setuju bahwa SICATMiS telah memenuhi kebutuhan pengguna. Evaluasi SUS menunjukan SICATMiS dapat diterima dengan sangat baik dan siap digunakan.

Errors in Reporting and Imputation of Government Benefits and Their Implications

We document the extent, nature, and consequences of survey errors for receipt of cash welfare and SNAP in three major U.S. household surveys linked to administrative program records. Our results confirm high rates of misreporting of program receipt, particularly failure to report receipt. The surveys inaccurately capture patterns of participation in multiple programs, even though there is little evidence of program confusion. Error rates are higher among imputed observations, which also account for a large share of false positive errors. Many household characteristics have significant effects on errors in reporting receipt, both false positives and false negatives. We find large differences in survey errors by race, ethnicity, income and other household characteristics. We provide evidence on the consequences of these errors for models of program receipt. Estimated effects of income and race are noticeably biased. We then examine error due to item non-response and imputation, as well as whether imputation improves estimates. Item non-respondents have higher receipt rates than the population, even conditional on many covariates. The assumptions for consistent estimates in multivariate models fail both when excluding item non-respondents and when using the imputed values. In binary choice models of program receipt, estimates from the linked data favor excluding item non-respondents rather than using their imputed values. The biases in each case are well predicted by the error patterns we document, so such analyses can help researchers make more informed decisions on the use of imputed values.

Sampling nesting sea turtles: optimizing survey design to minimize error

Many sea turtle studies globally use counts of nesting activities as a proxy for population abundance estimates and as an indicator of trends within the population. Often these populations are sampled temporally and spatially, but few previous studies have examined the impact of different sampling techniques on the accuracy of these estimates. We investigated temporal sampling errors using a multi-species approach, examining 10 populations comprising green, loggerhead and leatherback sea turtles. Sampling errors were investigated from random, regular and continuous sampling regimes spanning 5-80% coverage. A count approach was used rather than an individual-based capture-mark-recapture approach to broaden the scope and application of the research. Modelling showed that even low survey coverage of 5% gave reasonably accurate estimates of annual nesting activity, with estimated errors of ca. 20% (mean + 1 SD equalling 84.1% of surveys). Survey error is low relative to changes in abundance from the inter-annual variations in nesting activity that occur in sea turtle populations. Thus, annual studies are important to estimate sea turtle abundance, even if these studies have low survey coverage. An increase in survey effort may be more cost effective if spent combining estimates of total nesting activity with sampling turtles as part of a capture-mark-recapture study. This approach will provide a second estimate of annual abundance as well as an estimate of demographic parameters including clutch frequencies, remigration intervals, survivorship, immigration and emigration. This estimation of sampling errors may be useful in the design of monitoring programmes and can be used to guide management and policy decisions.

Errors in Reporting and Imputation of Government Benefits and Their Implications

We document the extent, nature, and consequences of survey errors for receipt of cash welfare and SNAP in three major U.S. household surveys linked to administrative program records. Our results confirm high rates of misreporting of program receipt, particularly failure to report receipt. The surveys inaccurately capture patterns of participation in multiple programs, even though there is little evidence of program confusion. Error rates are higher among imputed observations, which also account for a large share of false positive errors. Many household characteristics have significant effects on errors in reporting receipt, both false positives and false negatives. We find large differences in survey errors by race, ethnicity, income and other household characteristics. We provide evidence on the consequences of these errors for models of program receipt. Estimated effects of income and race are noticeably biased. We then examine error due to item non-response and imputation, as well as whether imputation improves estimates. Item non-respondents have higher receipt rates than the population, even conditional on many covariates. The assumptions for consistent estimates in multivariate models fail both when excluding item non-respondents and when using the imputed values. In binary choice models of program receipt, estimates from the linked data favor excluding item non-respondents rather than using their imputed values. The biases in each case are well predicted by the error patterns we document, so such analyses can help researchers make more informed decisions on the use of imputed values.

Dealing with covariate measurement error in a clustered cross-sectional survey

Many surveys are often complex cross-sectional studies that involve clustered data. Such surveys can have the additional complexity of the measurement error problem. Ignoring the measurement error ...

Reconciling Reports: Modelling Employment Earnings and Measurement Errors Using Linked Survey and Administrative Data

We contribute new UK evidence about measurement errors and employment earnings to a field dominated by findings about the USA. We develop and apply new econometric models for linked survey and administrative data that generalize those of Kapteyn and Ypma (Journal of Labor Economics, 2007). Our models incorporate mean-reverting measurement error in administrative data in addition to linkage mismatch and mean-reverting survey measurement error and 'reference period' error, while also allowing error distributions to vary across individuals. Annualised survey earnings underestimate true annual earnings on average. Mean-reversion in survey measurement errors is absent. Both earnings sources underestimate true earnings inequality. The survey earning measure is more reliable than the administrative data earnings measure, but hybrid earnings predictors based on both sources are distinctly more reliable than either source-specific measure. The models with heterogeneous measurement error distributions indicate how data quality may be improved. For example, for survey quality, our results highlight how respondents showing payslips to interviewers have smaller survey error variances. For administrative data, our results suggest that greater error variances are associated with non-standard jobs, private sector jobs, and employers without good payroll systems.

A Sectional Control Method to Decrease the Accumulated Survey Error of Tunnel Installation Control Network

The location reference for the precision installation of components of particle accelerator is provided by tunnel installation control network. The long and narrow control network has big accumulated error with increasing distance. In order to decrease the accumulated survey error of tunnel installation control network of particle accelerator, a sectional control method is proposed. Firstly, the accumulation rule of positional error with the length of control network is obtained by simulation calculation according to the shape of tunnel installation control network. Then, the RMS of horizontal positional precision of tunnel backbone control network is taken as the threshold. When the accumulated error is bigger than the threshold, tunnel installation control network should be divided into subsections reasonably. On each segment, the middle survey station is taken as the datum for independent adjustment calculation. Finally, by taking the backbone control points as faint datums, the weighted partial parameters adjustment is performed with the adjustment results of each segment and the coordinates of backbone control points. The subsections are jointed and unified into the global coordinate system in the adjustment process. An installation control network of linac with a length of 1.6 km is simulated. The RMS of positional deviation of the proposed method is 2.583 mm, and the RMS of the difference of positional deviation between adjacent points reaches 0.035 mm. Experimental results show that the proposed sectional control method can not only effectively decrease the accumulated survey error, but also guarantee the relative positional precision of installation control network. So it can be applied in the data processing of tunnel installation control network, especially for large particle accelerators.

Interviewing the Interviewers: Perceptions of Interviewer–Respondent Familiarity on Survey Process and Error in Burkina Faso

A growing body of literature in low- and middle-income countries is challenging the long-held assumption that the respondent and interviewer should be strangers. We conducted a qualitative study in Burkina Faso comprised of in-depth interviews and focus group discussions to explore interviewers’ experiences of collecting data on sexual and reproductive health indicators in their communities, and their perceptions of how familiarity with the respondents influences measurement error and nonresponse. We found that interviewers perceive familiarity to affect the process of collecting data but has minimal impact on responses obtained. Rather, interviewers perceived the personality of the respondent and her willingness to disclose personal information to be of greater influence. This study illustrates the complexities of interviewer–respondent exchange, provides context for the hiring practices of future surveys, and suggests specific variables to check when assessing data quality.

An empirical total survey error decomposition using data combination

Survey error is known to be pervasive and to bias even simple, but important, estimates of means, rates, and totals, such as the poverty and the unemployment rate. In order to summarize and analyze the extent, sources, and consequences of survey error, we define empirical counterparts of key components of the Total Survey Error Framework that can be estimated using data combination. Specifically, we estimate total survey error and decompose it into three high level sources of error: generalized coverage error, item non-response error and measurement error. We further decompose these sources into lower level sources such as failure to report a positive amount and errors in amounts conditional on reporting a positive value. For errors in dollars paid by two large government transfer programs, we use administrative records on the universe of program payments in New York State linked to three major household surveys to estimate the error components previously defined. We find that total survey error is large and varies in its size and composition, but measurement error is always by far the largest source of error. Our application shows that data combination makes it possible to routinely measure total survey error and its components. Our results allow survey producers to assess error reduction strategies and survey users to mitigate the consequences of survey errors or gauge the reliability of their conclusions.

Effect of Indigenous Language Interference on the Yoruba People Proficiency in English: A Syntactic Approach

A common phenomenon in a multilingual nation, especially a nation where a second language exists among different native languages, is the ‘linguistic interference’ which impedes effective communication among the people that use such a second language. This issue could be attributed to the fact that the second language users do not have all-encompassing knowledge of the rules which guide the use of the language. This research takes a run at finding out the effect of interference of indigenous language on proficiency in English (in both oral and written communications) among Yoruba People in Nigeria. This study focuses on the linguistic interference which usually occurs at the syntactic level of the indigenous language (Yoruba) and the Nigerian ‘lingua franca’ (English). The research adopts a descriptive survey method and error analysis approach for data analysis. The findings of this study show that the second language users of English bring the knowledge of the rules and features of their native languages into existence in the use of English in communication. This results in ungrammatical expressions in their everyday communication.

Photo-z outlier self-calibration in weak lensing surveys

Calibrating photometric redshift errors in weak lensing surveys with external data is extremely challenging. We show that both Gaussian and outlier photo-z parameters can be self-calibrated from the data alone. This comes at no cost for the neutrino masses, curvature and dark energy equation of state w0, but with a 65% degradation when both w0 and wa are varied. We perform a realistic forecast for the Vera Rubin Observatory (VRO) Legacy Survey of Space and Time (LSST) 3× 2 analysis, combining cosmic shear, projected galaxy clustering and galaxy—galaxy lensing. We confirm the importance of marginalizing over photo-z outliers. We examine a subset of internal cross-correlations, dubbed “null correlations”, which are usually ignored in 3× 2 analyses. Despite contributing only ∼ 10% of the total signal-to-noise, these null correlations improve the constraints on photo-z parameters by up to an order of magnitude. Using the same galaxy sample as sources and lenses dramatically improves the photo-z uncertainties too. Together, these methods add robustness to any claim of detected new Physics, and reduce the statistical errors on cosmology by 15% and 10% respectively. Finally, including CMB lensing from an experiment like Simons Observatory or CMB-S4 improves the cosmological and photo-z posterior constraints by about 10%, and further improves the robustness to systematics. To give intuition on the Fisher forecasts, we examine in detail several toy models that explain the origin of the photo-z self-calibration. Our Fisher code LaSSI (Large-Scale Structure Information), which includes the effect of Gaussian and outlier photo-z, shear multiplicative bias, linear galaxy bias, and extensions to LCDM, is publicly available at \\href{https://github.com/EmmanuelSchaan/LaSSI}{https://github.com/EmmanuelSchaan/LaSSI}.

Recall length and measurement error in agricultural surveys

This paper assesses the relationship between the length of recall and nonrandom error in agricultural survey data. Using data from LSMS-ISA surveys in Malawi and Tanzania, we show that key input and output variables are systematically related to the length of the recall period, indicating the presence of nonrandom measurement error. With longer recall periods, farmers report higher quantities of harvest, labor and fertilizer inputs. Farmers list fewer plots as the recall period increases. We argue that it is plausible that farmers over-estimate plot-level outcomes – harvest, labor and fertilizer inputs – while it is also plausible that they forget some of their more marginal plots as their memory decays due to longer recall periods. We find evidence of measurement error related to the length of recall also in common measures of agricultural productivity. The size of the recall effect typically varies between 2 and 5 percent per additional month of recall length, making its impact on the reliability of key agricultural indicators, including SDG indicator 2.3, economically significant. With data reliability affecting policy effectiveness, improving agricultural survey data quality remains an important concern, especially in light of the ambitious 2030 agenda reflected in the Sustainable Development Goals. Mainstreaming objective measures where possible and reducing the risk of recall error through shorter recall periods appear to be promising avenues to improve the quality of key variables in agricultural surveys.