Survey Error Research Articles

Environmental Attitudes, Concerns, and Behaviors Across Survey Modes. Assessing Selection and Measurement Biases in ISSP 2020 ‘Push-to-Web’ Surveys

How much does survey mode matter for predicting environmental attitudes and behaviors? It is essential to consider the influence of survey mode on the measurement of variables, particularly given the increasing popularity of web-based data collection following the Covid crisis. In 2020, several ISSP countries adopted a mixed-mode design, combining web and paper surveys for the Environment IV module. While mixed-mode data collection can reduce mode-specific errors, its success depends on accurately estimating and adjusting for selection and measurement differences between the modes. Failing to do so can increase the Total Survey Error, compromising the accuracy and reliability of the data. In this article, we examine selection and measurement differences in the web and paper modes in the Environment IV data from Switzerland and Finland. Register data are used as a benchmark to estimate selection biases in key socio-demographic variables, and we use these estimates to disentangle mode measurement effects from other sources of error. While we find that age, education, and labor market status account for selection effects across modes, we do not find significant differences in the measurement of environmental attitudes and behaviors across modes once the selection effect is controlled for. Mixing online and paper modes should thus not compromise data quality by introducing measurement biases for environmental variables.

Efficient detection of data entry errors in large-scale public health surveys: an unsupervised machine learning approach

Data entry errors in large-scale public health surveys can undermine the effectiveness of data-driven interventions. Therefore, identifying these data entry errors is crucial for public health experts. In large-scale public health surveys, manually verifying the accuracy of every data point by domain experts is nearly impossible. This study evaluates unsupervised machine learning algorithms for detecting these errors, focusing on the 'weight' parameter in the Annual Health Survey (AHS) dataset. The AHS, conducted by the Ministry of Health and Family Welfare, Government of India, in collaboration with the Registrar General of India, is a large-scale, stratified, household-level survey targeting maternal and child health across nine states in India. The dataset is freely available on the Open Government Data (OGD) Platform of India for public health research. In this study, five algorithms—DBSCAN, K-Means, Gaussian Mixture Model (GMM), Isolation Forest (IF), and One-Class SVM (1C-SVM) were applied to detect erroneous data entries. The evaluation process involved comprehensive preprocessing and feature engineering to optimize detection capabilities. Performance metrics such as precision, recall, accuracy, false anomaly, and missed anomaly rates were used to assess each algorithm. Among these, DBSCAN demonstrated superior performance, achieving a recall of 94.7% and a precision of 81.9%, making it highly effective for this task. The findings underscore the potential of unsupervised machine learning in automating the detection of data entry errors, thereby improving the integrity of public health data. This research contributes to the advancement of precision public health, supporting more accurate and reliable evidence-based decision-making and policy formulation.

Проявление агентности студенчества в проектном обучении: кейс федерального университета

According to experts, there will be a growing demand for hybrid and multidisciplinary specialists with advanced cognitive skills, emotional intelligence, and technological and social competence by 2030. These professionals will be able to make independent and innovative decisions in a rapidly changing labor market. The system of higher professional education faces the challenge of addressing the quantitative and qualitative shortage of young professionals. In this context, the topic of modernizing personnel training includes the introduction and active promotion of project-based learning. Project-based training should promote the agency of students as young professionals by transferring them within the framework of training from passive behavior to active independent actions, effective interaction with the professional community, disclosure of leadership potential, development of initiative and readiness to design new interdisciplinary solutions. The manifestations of student agency in project-based learning is an urgent issue on the research agenda and management practice of universities. The aim of the study was to investigate the key features of UrFU students' agency, as manifested in their project-based learning experiences. The article presents the findings from a survey conducted among students (N=1,070) using a quota sampling method in 2024 (including sex, training level and major subject). The survey error does not exceed 3 percent. According to the survey results, the relevance of key features of agency and project-based learning was established. It was empirically proved that project-based learning encourages independent, proactive behavior, promotes the development of leadership qualities, and enhances students' innovative potential.

A framework for comparing large-scale survey assessments: contrasting India’s NAS, United States’ NAEP, and OECD’s PISA

Large-scale survey assessments (LSAs) are important tools for measuring educational outcomes and shaping policy decisions. We present a framework for comparing LSAs to facilitate studying the impact of design choice on the precision of results, contrasting India’s National Achievement Survey (NAS), the United States’ National Assessment of Educational Progress (NAEP), and the OECD’s Programme for International Student Assessment (PISA). Our framework focuses on four key elements: sampling design, assessment design, analysis methodology, and reporting. The notion of total survey error, which is the accumulation of errors across the four key elements, can be used for both designing and evaluating LSAs. As example, we compare statistics that are commonly (but not always) reported from NAS, NAEP, and PISA to summarize outcomes related to sampling, measurement, and reporting. Our examination reveals several key similarities and differences among the three assessments, thereby highlighting the nuanced ways in which each LSA is tailored to meet the specific needs of their purpose and the challenges they face.

A spatiotemporal comparison of interobserver error in vegetation sampling

AbstractQuestionsWe asked how interobserver error in sampling vegetation (excluding trees) varied over time, space and habitat type; determined whether there were any obvious correlates of observer error; and evaluated evidence of bias among observers.LocationNine national park units in the Midwestern USA.MethodsWe quantified observer error in the context of a long‐term monitoring program employing three observers, evaluating interobserver error across 11 locations in the Midwestern USA over five years. The vegetation (excluding trees) was sampled independently by two teams of observers at prairie and woodland locations (n = 94 plots total).ResultsTotal pseudoturnover ranged between 20.2% and 22.1% at prairie locations, and between 16.8% and 28.6% at woodland locations. The overlooking component of pseudoturnover accounted for 75% or more of total pseudoturnover, with misidentification and cautious components each contributing 19% or less of the total, depending on location. The percentage of comparisons in which both observers recorded the same cover class ranged from 71.3% to 78.5% at the prairie locations and 56.9% to 85.6% at woodland locations. When observers did not agree on cover class, they were off by more than one class less than 6% of the time. Overlooking error was more likely to occur for species with less cover, while estimation error was more likely to occur for species with greater cover. A bias existed among observers, as the least experienced observer recorded 6.2%–11.8% more species than the other two observers. Interobserver bias also existed for rates of estimation error, as one observer consistently recorded higher cover classes.ConclusionsObserver error is a pervasive aspect of vegetation sampling. Continued training and experience yielded limited increases in precision. Elements of the sampling design probably reduced observer error to a certain degree, although some level of interobserver error in vegetation surveys is unavoidable.

Evaluating the impact of an invasive pathogen on tree population decline: An evidence based modelling approach

An upsurge of invasive forest pathogens (IFPs) has been causing widespread damage to forest ecosystems worldwide. Modelling future forest loss caused by IFPs is challenging, as it requires a sophisticated understanding of the pathogen-hosts-surrounding interactions. We developed a complexity-appropriate model using an evidence-based approach to predict the decline of the European ash (Fraxinus excelsior L.) population caused by ash dieback (Hymenoscyphus fraxineus) in a British deciduous woodland. Our model predicts that (1) the ash population will decline by ∼26 % in the next 10 year; (2) an ±10 % relative error in mortality survey would cause a ∼8 % bias in 10-year population decline; and (3) a 5 % increase in resistant trees would save ∼3 % population over 10 years. Our research demonstrates the merit of systematic reviews in balancing model complexity against generalisation. By scaling up the methodology to other IFPs, it is possible to forecast forest health with various management scenarios.

Recent Innovations and Advances in Mixed-Mode Surveys

Abstract The use of mixed-mode surveys has grown rapidly in recent years, due to both technological advances and the COVID-19 pandemic. The increased mixing of modes (and the adoption of newer digital modes like web and text messaging) necessitates an evaluation of the impact of these newer designs on survey errors and costs, as well as new techniques for disaggregating and adjusting for nonresponse and measurement errors. This special issue highlights recent innovations, applications, and evaluations of mixed-mode survey designs and identifies areas where additional research is required.

The return of non-probability sample: the electoral polls at the time of internet and social media

For the past 80 years survey researchers have used a probability sampling framework. Probability samples have a well-defined set of quality criteria that have been organized around the concept of Total Survey Error (TSE). Non-probability samples do not fit within this framework very well and some possible alternatives to TSE are explored. In recent years, electoral polls have undergone changes as a result of the dispersion of public opinion due mostly, but not only, to the development of the web and social media. From a methodological point of view, the main changes concerned sampling and data collection techniques. The aim of the article is to provide a critical contribution to the methodological debate on electoral polls with particular attention to the samples used which appear to be more similar to non-probability samples than to the traditional probability samples used for many decades in electoral polls. We will explore several new approaches that attempt to make inference possible even when a survey sample does not match the classic probability sample. We will also discuss a set of post hoc adjustments that have been suggested as ways to reduce the bias in estimates from non-probability samples; these adjustments use auxiliary data in an effort to deal with selection and other biases. Propensity score adjustment is the most well know of these techniques. The empirical section of the article analyzes a database of 1793 electoral polls conducted in Italy from January 2017 to July 2023.

What leads to measurement errors? Evidence from reports of program participation in three surveys

Measurement errors are often a large source of bias in survey data. Lack of knowledge of the determinants of such errors makes it difficult to reduce the extent of errors when collecting data and to assess the validity of analyses using the data. We study the determinants of reporting error using high quality administrative data on government transfers linked to three major U.S. surveys. Our results support several theories of misreporting: Errors are related to event recall, forward and backward telescoping, salience of receipt, the stigma of reporting participation in welfare programs and respondent's degree of cooperation with the survey overall. We provide evidence on how survey design choices affect reporting errors. Our findings help survey users to gauge the reliability of their data and to devise estimation strategies that can correct for systematic errors, such as instrumental variable approaches. Understanding survey errors allows researchers collecting survey data to reduce them by improving survey design. Our results indicate that survey design should take into account that higher response rates as well as collecting more detailed information may have negative effects on survey accuracy.

Responsive and Adaptive Designs in Repeated Cross-National Surveys: A Simulation Study

AbstractCross-national surveys run the risk of differential survey errors, where data collected vary in quality from country to country. Responsive and adaptive survey designs (RASDs) have been proposed as a way to reduce survey errors, by leveraging auxiliary variables to inform fieldwork efforts, but have rarely been considered in the context of cross-national surveys. Using data from the European Social Survey, we simulate fieldwork in a repeated cross-national survey using RASD where fieldwork efforts are ended early for selected units in the final stage of data collection. Demographic variables, paradata (interviewer observations), and contact data are used to inform fieldwork efforts. Eight combinations of response propensity models and selection mechanisms are evaluated in terms of sample composition (as measured by the coefficient of variation of response propensities), response rates, number of contact attempts saved, and effects on estimates of target variables in the survey. We find that sample balance can be improved in many country-round combinations. Response rates can be increased marginally and targeting high propensity respondents could lead to significant cost savings associated with making fewer contact attempts. Estimates of target variables are not changed by the case prioritizations used in the simulations, indicating that they do not impact nonresponse bias. We conclude that RASDs should be considered in cross-national surveys, but that more work is needed to identify suitable covariates to inform fieldwork efforts.

A Mixture Model Approach to Assessing Measurement Error in Surveys Using Reinterviews

Abstract Researchers are often unsure about the quality of the data collected by third-party actors, such as survey firms. This may be because of the inability to measure data quality effectively at scale and the difficulty with communicating which observations may be the source of measurement error. Researchers rely on survey firms to provide them with estimates of data quality and to identify observations that are problematic, potentially because they have been falsified or poorly collected. To address these issues, I propose the QualMix model, a mixture modeling approach to deriving estimates of survey data quality in situations in which two sets of responses exist for all or certain subsets of respondents. I apply this model to the context of survey reinterviews, a common form of data quality assessment used to detect falsification and data collection problems during enumeration. Through simulation based on real-world data, I demonstrate that the model successfully identifies incorrect observations and recovers latent enumerator and survey data quality. I further demonstrate the model’s utility by applying it to reinterview data from a large survey fielded in Malawi, using it to identify significant variation in data quality across observations generated by different enumerators.

Appraising Auditor Error in Medicaid Audits

Medicaid conducts statistical audits to determine whether providers appropriately billed for Medicaid services. The basis for any claimed liability is an audit based on statistical extrapolation derived from a sample drawn from the totality of claims filed by the provider. Auditor error may be present in nearly every Medicaid audit, yet it is rarely explicitly taken into account in favor of a focus on survey error. It is demonstrated that for commonplace, plausible, audit conditions the presence of audit-error may result in coverage percentages that fall short of those derived via the classical Central Limit Theorem-based method. This data artifact may impeach the Medicaid auditor’s examination of the claimant. The methodology here is not necessarily specific to Medicaid audits; rather, it may generalize to every statistical sampling-based health-care forensic audit.

Effectiveness of vertical error budget model for portable multi-beam echo-sounder in shallow water bathymetric survey

With the increasing availability of portable survey equipment and platforms, hydrographic data acquisition now offers a diverse range of device options and installation methods. Tailoring the mounting and installation of hydrographic survey devices to fit the boat or platform dimensions has become common practice. The use of portable equipment is essential, given the wide selection of combined options available in the market. Consequently, sensor offsets and device performance differ for each survey conducted, necessitating the constant verification of accuracy performance.This paper examines the capabilities of portable multibeam echosounder (MBES) deployments on various types of survey boats. The investigation focuses on error budget modeling and field tests. A model is developed to estimate the propagated uncertainty resulting from different sources of measurement uncertainties. The model primarily considers vertical sources of error in MBES surveys, as they have the most significant impact on hydrographic data quality. To validate the model, field tests are conducted at two different locations using different boat and device configurations. The results demonstrate that the modeled uncertainties align with the measured standard deviations, particularly in the Pramuka area, where the standard deviation of data acquired at the widest beam angle of ±60° is 0.25m, close to the predicted value of 0.28m. However, in the Patimban area, the model overestimates the uncertainty, predicting 0.27m compared to the measured standard deviation of 0.17m.

Total Bias in Income Surveys when Nonresponse and Measurement Errors are Correlated

Abstract Household surveys on income might suffer from quality limitations mainly due to the difficulty of enrolling households (unit nonresponse) and retrieving correct information during the interview (measurement error [ME]). These errors are likely to be correlated because of latent factors, such as the threat of disclosing personal information, the perceived sensitivity of the topic, or social desirability. For survey organizations, assessing the interplay of these errors and their impact on the accuracy and precision of inferences derived from their data is crucial. In this article, we propose to use a standard sample selection model within a total survey error framework to deal with the case of correlated nonresponse error (NR) and ME in estimating average household income. We use it to study the correlation between the two errors, quantify the ME component due to this correlation, and evaluate ME among nonrespondents. Using the Italian Survey on Income and Wealth linked with administrative income data from tax returns, we find a positive correlation between the two errors and that households at the extremes of the income distribution mainly cause this association. Our results show that ME contributes more to the total error than unit nonresponse and that it would be larger in absence of the correlation between the two errors. Finally, efforts to reduce nonresponse rates are worthwhile only for nonrespondents in the lowest estimated response propensity group. If these households participate, the bias decreases because of the reduction in NR that offsets the increase in ME.

Reducing vertical bias and error in tidal marsh digital elevation models with machine learning and LiDAR derivatives

The vertical bias and uncertainty of LiDAR-derived elevation data in tidal marshes is a major challenge for researchers in these critical ecosystems. Small positive bias errors in land surface elevation can lead to large underestimates of coastal inundation. Previous studies have shown that the bias and uncertainty of elevation data can be reduced using a variety of statistical techniques. However, many studies cover a relatively small extent, cover a large extent at coarse resolution, or rely on local data products that may not be widely available or are of unknown quality. This study aimed to develop and compare bias reduction approaches for digital elevation models (DEMs) using LiDAR derivatives in tidal marshes along Delaware Bay, Delaware, USA, a region that experiences high rates of relative sea-level rise and frequent coastal flooding. In this study, several statistical and machine-learning approaches were evaluated based on their ability to reduce vertical DEM error using field GPS survey data and LiDAR and terrain derivatives as inputs. The evaluated approaches for reducing DEM error included ensembles of multiple linear regressions, kernel-K Nearest Neighbors, gradient boosted regression trees, random forests, and deep neural networks (DNNs), which were compared based on statistical performance metrics. An ensemble of deep neural networks performed best at removing vertical DEM bias and reducing DEM uncertainty, though other approaches also performed well. GPS survey data indicated a mean absolute error, mean bias, and root mean square error in GPS surveys of 11.9 cm, 10.9, and 14.8 cm, respectively, in the original DEM. These error metrics were reduced to 3.5 cm, −0.2 cm, and 5.1 cm in the DNN ensemble-corrected DEM. The DNN ensemble was then applied to all tidal marshes throughout the study area. This corrected DEM clearly showed how interpretations of marsh platform inundation based on uncorrected DEM surfaces could be misguided. The approach used in this study only requires LiDAR-derived products and field surveys, so it may be applied readily in other regions.

중소형빌딩 리모델링의 리스크 요인 유형화 연구

In Seoul, old small and medium-sized buildings that are over 25 years old account for more than 60% of the total. So these old buildings need to be remodeled to restore building functions and improve aesthetics. Nevertheless, due to the absence of previous studies on risk factors occurring in remodeling projects, risk management is difficult during remodeling, therefore remodeling is sluggish. This study aims to contribute to revitalizing remodeling by developing risk factors for remodeling small and medium-sized buildings and analyzing and presenting the importance of each factor using the AHP technique. As a result of the analysis, it was found that the 'preparation stage' was overwhelmingly important among the four stages of the major classification. Among the 10 attributes of the medium classification, “market analysis” and “asset analysis” ranked first and second. Among the 25 factors of the minor classification, it was analyzed that it is important in the order of 'market survey error', 'MD composition failure', and 'current status map and real mismatch'. As an implication, the most carefully considered risk in the remodeling project is not only a thorough investigation of the sale price and rent information of similar buildings, but also a floor-by-floor rental composition(MD) plan optimized for the location of the target building. Because ‘MD composition’ is a plan that can maximize the rent that the target building can generate according to its location. If this plan succeeds, it will lead to an increase in building value as well as an increase in rental income, which is the most important factor both for the building owner and the developer.

Comparing the Photometric Calibration of DESI Imaging and Gaia Synthetic Photometry

The relative photometric calibration errors in the DESI Legacy Imaging Surveys (LS), which are used for DESI target selection, can leave imprints on the DESI target densities and bias the resulting cosmological measurements. We characterize the LS calibration systematics by comparing the LS stellar photometry with Gaia DR3 synthetic photometry. We find the stellar photometry of LS DR9 and Gaia has an rms difference of 4.7, 3.7, 4.4 mmag in DECam grz bands, respectively, when averaged over an angular scale of . There are distinct spatial patterns in the photometric offset resembling the Gaia scan patterns (most notably in the synthesized g-band) which indicate systematics in the Gaia spectrophotometry, as well as honeycomb patterns due to LS calibration systematics. We also find large and smoothly varying photometric offsets at decl. < −29.°25 in LS DR9 which are fixed in DR10.

Measuring ground surface elevation changes in a slow-moving colluvial landslide using combinations of regional airborne lidar, UAV lidar and UAV photogrammetric surveys

Slow-moving, chronically destructive landslides are projected to grow in number globally in response to precipitation increases from climate change, and land disturbances from wildfire, mining and construction. In the Cincinnati and northern Kentucky metropolitan area, USA, landslides develop in colluvium that covers the steep slopes along the Ohio River and its tributaries. Here we quantify elevation changes in a slow-moving colluvial landslide over 14 years using county-wide lidar, uncrewed aerial vehicle (UAV) structure-from-motion (SfM) surveys and a UAV lidar survey. Because the technology and quality differ between surveys, the challenge was to calculate a threshold of detectable change for each survey combination. We introduce two methods; the first uses propagated elevation difference errors and the second back-calculates the individual survey errors. Thresholds of detection range from ±0.05 to ±0.20 m. Record rainfall in 2011 produced the largest vertical changes. Since then, the landslide toe has continued to deform, and the landslide has doubled its width by extending into a previously undisturbed slope. Although this study presents a technique to utilize older datasets in combination with modern surveys to monitor slow-moving landslides, it is broadly applicable to other studies where topographic data of differing quality are available.

Using UAVs and Photogrammetry in Bathymetric Surveys in Shallow Waters

The use of UAV (unmanned aerial vehicle) platforms and photogrammetry in bathymetric surveys has been established as a technological advancement that allows these activities to be conducted safely, more affordably, and at higher accuracy levels. This study evaluates the error levels obtained in photogrammetric UAV flights, with measurements obtained in surveys carried out in a controlled water body (pool) at different depths. We assessed the relationship between turbidity and luminosity factors and how this might affect the calculation of bathymetric survey errors using photogrammetry at different shallow-water depths. The results revealed that the highest luminosity generated the lowest error up to a depth of 0.97 m. Furthermore, after assessing the variations in turbidity, the following two situations were observed: (1) at shallower depths (not exceeding 0.49 m), increased turbidity levels positively contributed error reduction; and (2) at greater depths (exceeding 0.49 m), increased turbidity resulted in increased errors. In conclusion, UAV-based photogrammetry can be applied, within a known margin of error, in bathymetric surveys on underwater surfaces in shallow waters not exceeding a depth of 1 m.

Analysis of Interview Breakoff in the Behavioral Risk Factor Surveillance System, 2018 and 2019

Survey breakoff is an important source of total survey error. Most studies of breakoff have been of web surveys-less is known about telephone surveys. In the past decade, the breakoff rate has increased in the Behavioral Risk Factor Surveillance System, the world's largest annual telephone survey. Analysis of breakoff in Behavioral Risk Factor Surveillance System can improve the quality of Behavioral Risk Factor Surveillance System. It will also provide evidence in research of total survey error on telephone surveys. We used data recorded as breakoff in the 2018 and 2019 Behavioral Risk Factor Surveillance System. We converted questions and modules to a time variable and applied Kaplan-Meier method and a proportional hazard model to estimate the conditional and cumulative probabilities of breakoff and study the potential risk factors associated with breakoff. Cumulative probability of breakoffs up to the end of the core questionnaire was 7.03% in 2018 and 9.56% in 2019. The highest conditional probability of breakoffs in the core was 2.85% for the physical activity section. Cumulative probability of breakoffs up to the end of the core was higher among those states that inserted their own questions or optional modules than among those that did not in both years. The median risk ratio of breakoff among all states was 5.70 in 2018 and 3.01 in 2019. Survey breakoff was associated with the length of the questionnaire, the extent of expected recollection, and the location of questions. Breakoff is not an ignorable component of total survey error and should be considered in Behavioral Risk Factor Surveillance System data analyses when variables have higher breakoff rates.