Bias In Surveys Research Articles

Evaluation of branch sampling, ocular assessments, and aerial surveys for estimating spruce budworm defoliation

We compared three methods for estimating current-year spruce budworm ( Choristoneura fumiferana (Clem.)) defoliation from 2014 to 2021 using a network of 99 permanent sample plots in central Gaspé Peninsula, Québec. Percent current-year defoliation was measured by assessing shoots from mid-crown branches, ocular ratings of all individual trees using binoculars, and provincial government aerial surveys. Ocular survey defoliation differed from branch sample defoliation in 5–6 out of 7 years, consistently underestimating defoliation, across the full range of defoliation severity observed. Nested mixed-effect models for fir–spruce combined, balsam fir, white spruce, and black spruce ocular survey defoliation bias resulted in marginal R2 of 0.40, 0.47, 0.82, and 0.86, respectively. Current defoliation severity and its interaction with previous year defoliation and weather conditions significantly affected ocular survey bias. Correspondence of aerial survey estimates and mean plot defoliation occurred in only 43% of all plot-years and ranged from 14%–58% in individual years. Differences between aerial survey defoliation and plot values mainly resulted from assigning an adjacent class (e.g., light <30% assigned as moderate 31%–70% defoliation) or misplaced defoliation polygon boundaries, suggesting that assignment of aerial survey defoliation to plots or specific ground areas needs ground truth sampling.

Calibration and XGBoost reweighting to reduce coverage and non-response biases in overlapping panel surveys: application to the Healthcare and Social Survey

BackgroundSurveys have been used worldwide to provide information on the COVID-19 pandemic impact so as to prepare and deliver an effective Public Health response. Overlapping panel surveys allow longitudinal estimates and more accurate cross-sectional estimates to be obtained thanks to the larger sample size. However, the problem of non-response is particularly aggravated in the case of panel surveys due to population fatigue with repeated surveys.ObjectiveTo develop a new reweighting method for overlapping panel surveys affected by non-response.MethodsWe chose the Healthcare and Social Survey which has an overlapping panel survey design with measurements throughout 2020 and 2021, and random samplings stratified by province and degree of urbanization. Each measurement comprises two samples: a longitudinal sample taken from previous measurements and a new sample taken at each measurement.ResultsOur reweighting methodological approach is the result of a two-step process: the original sampling design weights are corrected by modelling non-response with respect to the longitudinal sample obtained in a previous measurement using machine learning techniques, followed by calibration using the auxiliary information available at the population level. It is applied to the estimation of totals, proportions, ratios, and differences between measurements, and to gender gaps in the variable of self-perceived general health.ConclusionThe proposed method produces suitable estimators for both cross-sectional and longitudinal samples. For addressing future health crises such as COVID-19, it is therefore necessary to reduce potential coverage and non-response biases in surveys by means of utilizing reweighting techniques as proposed in this study.

Pedagogical innovation and curricular adaptation in enhancing digital literacy: A local wisdom approach for sustainable development in Indonesia context

This research investigates the relationship between Pedagogical Innovation (PED), Curricular Adaptation (CUR), Local Wisdom Approach (LWA), and Digital Literacy (DIGLI) in Early Childhood Education, aligns with Sustainable Development Goals (SDGs), promoting quality education (SDG 4), fostering cultural inclusion, and advocating for sustainable partnerships to address limitations and enhance global understanding in Banten, Indonesia. Employing a quantitative method, a sample of 220 participants, including teachers, parents, and education stakeholders, was surveyed from July to August 2023. The main findings reveal significant impacts of PED and CUR on both the LWA and DIGLI. Moreover, the LWA mediates the relationships between PED and DIGLI, as well as CUR and DIGLI. Implications of this research highlight the pivotal role of integrating cultural values for optimizing the effects of innovative teaching methods and adaptive curricula on the development of digital literacy skills among young learners. Limitations include the study's geographical specificity to Banten, potential response bias in the self-report survey, and the cross-sectional design limiting temporal insights. Future prospects involve expanding the study's geographical scope, incorporating qualitative methodologies, and conducting longitudinal analyses to provide a more comprehensive understanding of the complex dynamics in Early Childhood Education.

Methodological influences on ecosystem valuation: a systematic review of contingent valuation studies

ABSTRACT The contingent valuation method (CVM) has long been criticised for its capability in predicting the value of public goods. Split-sample studies are conducted to assess the effects of research methodologies on economic value estimation. However, there is a lack of literature that comprehensively reviews the previous works. This study addresses this knowledge gap by identifying the effect of questionnaire design and survey methods on CVM outcomes, focusing on ecosystem valuation. The preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework is adopted, and 74 previous studies are included, gathering 100 observations. The findings reveal that questionnaire design elements such as providing more detailed information, using hypothetical payments, incorporating additional questions, applying a more extensive scope of the conservation programme, and utilising dichotomous choice questions positively influence the mean willingness to pay (WTP). Regarding survey methods, results show that active users of the ecosystem exhibit higher WTP than non-users, and direct interaction with the respondents yields higher WTP outcomes. This study signifies how research methodology choices influence economic values derived from CVM, highlighting the importance of considering survey bias and aiding the development of a more effective survey instrument.

Evaluating Non-response Rates and Bias in the American Time Use Survey

This paper investigates the ongoing issue of non-response in the American Time Use Survey (ATUS) and its impact on the survey's reliability and representativeness. Despite targeted efforts, ATUS has experienced a consistent decrease in response rates, which was exacerbated during the COVID-19 pandemic. This decreasing response rate may link to potential non-response bias, which could distort the survey's depiction of time use patterns across the U.S. population. The study analyzes the complexities associated with non-response rates, emphasizing biases that might emerge from groups with weaker community ties. Challenging traditional assumptions, recent research indicates that high response rates do not automatically mitigate bias. This paper highlights the necessity for innovative survey methods and the adoption of new technologies, such as web-administered diaries and smartphone apps, to address these issues. The analysis underscores the importance of adapting survey strategies to contemporary societal and technological landscapes, aiming to enhance the accuracy, efficiency, and representativeness of ATUS data.

Gender gaps in financial literacy: a multi-arm RCT to break the response bias in surveys

Gender gaps in financial literacy: a multi-arm RCT to break the response bias in surveys

Revisiting Sample Bias in the UK's Annual Survey of Hours and Earnings, with Implications for Estimates of Low Pay and the Bite of the National Living Wage

Revisiting Sample Bias in the UK's Annual Survey of Hours and Earnings, with Implications for Estimates of Low Pay and the Bite of the National Living Wage

Genuine or impression management? Uncovering biases in how people respond to the Black Lives Matter movement in sport

PurposeNational surveys reveal that sports fans exhibit greater support for athletes, sports teams and leagues endorsing social justice initiatives compared to the general population, highlighting the potential of sports for positive social impact. This study investigates whether such responses are influenced by systematic biases.Design/methodology/approachReplicating a Nielsen national survey, two experiments explore whether biases affect support for athletes' participation in the Black Lives Matter (BLM) movement. The study also examines partisan motivated reasoning as a factor driving sports fans' support for BLM.FindingsWhile avid fans display stronger endorsement of BLM compared to causal/non-sports fans, evidence suggests that systematic biases distort these responses. When sport identity becomes salient, reported support for the BLM movement becomes inflated.Research limitations/implicationsResearchers often employ self-report surveys to gauge audience perceptions of athlete activism or cause-related initiatives, particularly when assessing their impact. This study's findings indicate that this context is susceptible to SDB.Originality/valueThe study underscores the role of systematic biases in self-report surveys, particularly in socially desirable contexts. People tend to over-report “positive behavior,” leading survey participants to respond more favorably to questions that are socially desirable. Therefore, interpreting survey results with caution becomes essential when the research context is deemed socially (un)desirable. It is crucial for researchers to apply appropriate measures to identify and mitigate systematic response biases. The authors recommend that researchers adopt both procedural and statistical remedies to detect and reduce social desirability biases.

Measuring participation in undeclared work in Europe using survey data: A method for resolving social desirability bias

Measuring participation in undeclared work using surveys has been criticized for under-estimating the level of engagement due to social desirability bias that leads to an under-reporting of “bad” behavior. Until now, few studies have sought to quantify the amplitude of this bias in surveys of undeclared work. The aim of this paper is to fill this gap by using the most appropriate methodologies for estimating the probability of misleading responses in such surveys. Reporting data from special Eurobarometer survey no. 498 conducted in 2019 and involving 27,565 respondents in EU-27 countries and the UK, only 3.5% openly admitted to participating in undeclared work. The results of a Probit model with correction for misclassified cases (i.e., those undertaking undeclared work but declaring that they do not) reveals that nearly a quarter (23.3%) of the respondents undertaking undeclared work refused to openly admit this during the survey, due to the social desirability bias. The estimated overall proportion of undeclared workers is 17.3%. We obtained this value by correcting for both misclassification and the additional source of negative bias due to the large imbalance in the data (i.e., observations in one class are much lower than the other). The outcome of this new advanced approach in analysing undeclared work is that survey estimates can now report its size and determinants in a more accurate manner than has been previously the case.

Reweighting in panel surveys: machine learning techniques for the Health Care and Social Survey

Abstract Background Healthcare statistical services have used probability surveys to respond to such information needs. The Health Care and Social Survey (ESSOC) research project arises from the need to provide data on the evolution of the COVID-19 impact. This survey has an overlapping panel survey design with 4 measurements throughout 1 year with random samples. Objective To develop a new reweighting method for overlapping panel surveys affected by non-response. Methods Each ESSOC measurement is composed of two samples: a longitudinal sample from previous measurements and a new sample at each measurement. In each measurement, missing units are substituted by new surveyed units, allowing the obtention of cross-sectional and longitudinal estimates. The advantage of this design is that, in addition to being able to obtain longitudinal estimates, cross-sectional estimates are more accurate because of the larger sample size. However, the problem of non-response is particularly aggravated in the case of panel surveys, due to the fatigue of the population to be repeatedly surveyed. Results Considering the design, timing and objectives of this survey, our reweighting method produces suitable estimators for both cross-sectional and longitudinal samples. The weights are the result of a two-step process: the original sampling design weights are corrected during a 1st phase by modelling the non-response with respect to the longitudinal sample obtained in a previous measurement using machine learning techniques. Then, during a 2nd phase, they are calibrated using the auxiliary information available at the population level. The proposed method is applied to the estimation of totals, proportions, ratios, and differences between measurements as well as gender gaps. Conclusions For addressing future health crises like COVID-19, it is therefore necessary to reduce potential coverage and non-response biases in surveys by means of utilising reweighting techniques as proposed in this study. Key messages • It is therefore necessary to reduce potential coverage and non-response biases in surveys. • This study proposes the utilisation of reweighting techniques based on calibration and XGBoost.

Respondent biases in agricultural household surveys

Two sources of respondent bias introduce measurement error into household statistics: asymmetric information between the proxy respondent and the individual on whom they report; and aggregation bias when a proxy respondent reports on a household-level outcome across multiple individuals. We estimate the effects of respondent biases in a survey experiment in Burkina Faso by varying who reports on the agricultural production of household members. We find respondent biases are not solely attributable to asymmetric information. Choosing a household head proxy lowers aggregation biases, but results in both over and under-estimates of agricultural variables relative to random proxies. Random proxies systematically under report agricultural statistics. Self-reporting protocols increase enumerator work days by only 5% indicating a high bias-cost tradeoff in choosing proxy response over self-reports. Survey designers should weight whether proxy bias magnitude or direction of bias are more significant threats to parameter estimation when determining their proxy response protocol.

Hybrid mark–resight approaches account for heterogeneous detection bias in aerial surveys

Abstract Population abundance estimation is a critical yet challenging task for wildlife management decision‐making. For aerial surveys of terrestrial mammals in highly heterogenous landscapes, estimation models that address multiple sources of detection bias are needed. A typical hybrid mark–resight model includes an additive parameter that accounts for detection probabilities derived from marked animals. While this additive hybrid method improves abundance estimates, there remains a need to account for heterogeneity due to the conditional dependence in detection probability inherent in aerial surveys with multiple observers. We propose a modified approach, referred to as a conditional hybrid model, to address this heterogeneity directly. The conditional hybrid model applies a mark–resight approach typically used to account for tag loss in mark–recapture studies to aerial surveys with a subset of the population marked with GPS collars. We conduct a structured comparison between the additive and conditional hybrid estimation models and fit these two hybrid models to data from a 2020 aerial survey of mule deer in southern California. Both the additive and conditional hybrid models had general agreement in overall model structure and variable importance, however, the conditional model outperformed the additive model in capturing sources of detection heterogeneity and overall abundance estimates differed by 9%. Our top‐ranked models suggest that deer movement, larger group sizes, and rougher topography increased detection probability, whereas high cloud cover, obscuring vegetation and high observer fatigue and helicopter height decreased detection probability. Our findings illustrate the need to assess multiple correction methods when estimating abundance from aerial survey data to ensure accurate abundance estimates are being used to inform wildlife management strategies, with the caveat that specific survey conditions may dictate which method is the most appropriate for achieving robust estimates. These approaches represent the cutting‐edge science on wildlife abundance estimation from aerial survey data, a critical ingredient for effective wildlife management. By testing innovative hybrid methods, we present the first formal comparison of the strengths and weaknesses of each method within a highly heterogeneous landscape, providing practitioners with necessary information for future application.

Ensuring cross-cultural data comparability by means of anchoring vignettes in heterogeneous refugee samples

BackgroundConfigural, metric, and scalar measurement invariance have been indicators of bias-free statistical cross-group comparisons, although they are difficult to verify in the data. Low comparability of translated questionnaires or the different understanding of response formats by respondents might lead to rejection of measurement invariance and point to comparability bias in multi-language surveys. Anchoring vignettes have been proposed as a method to control for the different understanding of response categories by respondents (the latter is referred to as differential item functioning related to response categories or rating scales: RC-DIF). We evaluate the question whether the cross-cultural comparability of data can be assured by means of anchoring vignettes or by considering socio-demographic heterogeneity as an alternative approach.MethodsWe used the Health System Responsiveness (HSR) questionnaire and collected survey data in English (n = 183) and Arabic (n = 121) in a random sample of refugees in the third largest German federal state. We conducted multiple-group Confirmatory Factor Analyses (MGCFA) to analyse measurement invariance and compared the results when 1) using rescaled data on the basis of anchoring vignettes (non-parametric approach), 2) including information on RC-DIF from the analyses with anchoring vignettes as covariates (parametric approach) and 3) including socio-demographic covariates.ResultsFor the HSR, every level of measurement invariance between the Arabic and English languages was rejected. Implementing rescaling or modelling on the basis of anchoring vignettes provided superior results over the initial MGCFA analysis, since configural, metric and – for ordered categorical analyses—scalar invariance could not be rejected. A consideration of socio-demographic variables did not show such an improvement.ConclusionsSurveys may consider anchoring vignettes as a method to assess cross-cultural comparability of data, whereas socio-demographic variables cannot be used to improve data comparability as a standalone method. More research on the efficient implementation of anchoring vignettes and further development of methods to incorporate them when modelling measurement invariance is needed.

Precise self-calibration of interloper bias in spectroscopic surveys

ABSTRACT Interloper contamination due to line misidentification is an important issue in the future low-resolution spectroscopic surveys. We realize that the algorithm previously used for photometric redshift self-calibration, with minor modifications, can be particularly applicable to calibrate the interloper bias. In order to explore the robustness of the modified self-calibration algorithm, we construct the mock catalogues based on China Space Station Telescope (CSST), taking two main target emission lines, Hα and [O iii]. The self-calibration algorithm is tested in cases with different interloper fractions at 1 per cent, 5 per cent, and 10 per cent. We find that the interloper fraction and mean redshift in each redshift bin can be successfully reconstructed at the level of ∼ 0.002 and ∼ 0.001(1 + z), respectively. We also find the impact of the cosmic magnification can be significant, which is usually ignored in previous works, and therefore propose a convenient and efficient method to eliminate it. Using the elimination method, we show that the calibration accuracy can be effectively recovered with slightly larger uncertainty.

Lobbying in social media as a new source of survey bias

Questionnaires are widely-used tools in empirical social science, including tourism research. The key premise of each quantitative survey is the representativeness of the investigated sample. Yet, the rapid development of information and communication technology forces social scientists to take new sources of biases into account. This paper describes the problem of lobbying in social media, initiated by survey participants themselves. We report a collective action of online communities which resulted in over-representation of the collective opinions of certain interest groups. The bias has been observed when analyzing data collected during a visitor monitoring study in a popular forest recreation destination: the Tricity Landscape Park in Poland. Two different methods: on-site survey (n = 994) and online survey (n = 1023) were used to collect data on local recreational preferences. A significant effect of lobbying in social media on online survey responses has been identified in the case of questions related to the forest management. Management implicationsOur study confirms significant differences between results obtained via on-site and online surveys.Cost-efficient distribution of online survey link via social media may cause biased results.Especially, distribution channels of online surveys related to problematic management issues need to be carefully chosen.In such cases, social media activity (link sharing, post assessment (e.g. likes or negative reactions) accompanying the discussion) need to be additionally monitored in order to detect uncontrolled lobbying of certain users’ opinions.

The yea-paradox: Cognitive bias in technology acceptance surveys

Technology acceptance is widely regarded as one of the most established, mature, and continuously relevant streams of information systems research. Its fundamental assumption is that technology acceptance scores accurately reflect users’ intentions, needs, and emotions toward the technology of interest. However, many studies (including this one) present evidence that challenges this assumption, suggesting that the link between reported scores and actual perception is more complex than often assumed. We aim to find out how cognitive biases influence self-reported technology acceptance scores. We draw on dialogical iterations between an exploratory case study at Cloud Corp, a multinational Software-as-a-Service provider and related literature on technology acceptance and cognitive bias. Our study reveals a paradoxical relationship between reported acceptance scores and actual perceptions: Despite providing high acceptance scores, users did not actually use the technology. Conversely, users who complained a lot by filing many support tickets were more engaged with the technology When yea-saying users meet yea-hearing designers, technology acceptance scores may be spuriously inflated, providing an exaggerated picture of users’ attitudes toward a technology. We call this the ‘yea-paradox’ because of its roots in people's tendency to agree half-heartedly (‘yea’), even when they privately disagree, to avoid more complex and potentially difficult conversations.

Using increasing hypothetical chronic wasting disease prevalence rates to explore hunter behavioral intention and hypothetical bias in surveys

ABSTRACT Deer hunter surveys commonly employ hypothetical chronic wasting disease (CWD) prevalence scenarios to measure behavioral intentions. These surveys may suffer from hypothetical bias (i.e. respondents describe unrealistic behaviors in hypothetical situations) and it is unclear how hypothetical bias affects results. We used an online-based survey of Arkansas deer hunters to (1) investigate how behavioral intentions change as CWD prevalence hypothetically varies, and (2) explore the presence of hypothetical bias in behavioral intention by comparing responses across hypothetical prevalence rates from those whose primary hunting counties had CWD detected versus not. We found undesired behaviors (e.g. hunting less often, hunting cessation) were unlikely to change substantially as prevalence increased, and assistance with some CWD monitoring and management behaviors may increase. We found some evidence of hypothetical bias in anticipated behavior change. Agencies should be aware that survey respondents may overstate behavioral intentions, and should consider this when making CWD management decisions.

Fish avoidance of ships during acoustic surveys tested with quiet uncrewed surface vessels

Acoustic surveys to estimate fish biomass and abundance are a major component of many fisheries monitoring programs. An important bias in acoustic surveys is that fish may avoid the survey vessel. Here, we utilized quiet uncrewed surface vessels (USVs) equipped with 120 kHz split beam echosounders to evaluate fish responses to motorized survey vessels. Two of these USVs were deployed in Lakes Michigan and Huron in summer 2021 and compared against three motorized vessels used in conventional fisheries acoustic surveys. Paired comparisons employed vessel-drone passes that provided the opportunity to observe fish response as vessels approached and then overtook their quiet USV counterparts. Sound originating from the vessels was primarily in the 10–1000 Hz range. Overall received sound pressure levels for ships at the closest pass were 100 dB (re 1 µPa @ 1 m) for one and 133–134 dB for the other two vessels. In contrast, sound originating from the USV was not detectable over ambient noise. We examined acoustic data from the USVs for potential changes in total acoustic backscatter, average target depth, and average in situ target strength as vessels approached. We observed weak evidence of an avoidance response by fish to the vessel with the loudest noise profile and highest survey speed but not for the other two vessels. We also compared acoustic data from 33 2-km transects surveyed by both vessels and the USVs, finding few differences between vessel and USV data for water depths between 5 and 80 m. Results from this work suggest that acoustics estimates of fishes in Lakes Michigan and Huron (primarily alewife, rainbow smelt, and bloater) are largely consistent among the vessels used in these two lakes for standard acoustic surveys and that fish avoidance is minimal in water depths > 5 meters.

Nonresponse trends in establishment panel surveys: findings from the 2001–2017 IAB establishment panel

Many household panel surveys have experienced decreasing response rates and increasing risk of nonresponse bias in recent decades, but trends in response rates and nonresponse bias in business or establishment panel surveys are largely understudied. This article examines both panel response rates and nonresponse bias in one of the largest and longest-running establishment panels, the IAB Establishment Panel. Response rate trends are reported over a 17-year period for each annual cohort and rich administrative data are used to evaluate changes in nonresponse bias and test hypotheses regarding short-term and long-term panel participation. The findings show that while cumulative panel response rates have declined over time, wave-to-wave reinterview rates have remained largely stable. Reinterview nonresponse bias has also remained stable, while cumulative nonresponse bias has consistently increased within all cohorts. Larger establishments and those that experienced an interviewer change or did not answer all survey questions (item nonresponse) in a previous wave were less likely to continue participating in the panel. These findings and their practical implications are discussed in conclusion.

Surveyor and analyst biases in forest density estimation from United States Public Land Surveys

AbstractAccurate forest density estimates based on United States Public Land Surveys have long been questioned because of doubts about randomness of both the surveyors' selection of witness trees and the underlying tree dispersion. This study analyzes the surveyor sampling of witness trees in six Midwestern states in the mid‐1800s. It develops universal methods for identification, quantification, and correction of bias, and then calculation of unbiased density. Applying these techniques produces unbiased site‐specific densities before Euro‐American settlement, which are the essential baseline for determining historic changes in forest structure. Previous analysts used untested assumptions, inaccurate estimators, unknown or unrealistic sampling designs, and omitted or poorly corrected surveyor bias, resulting in hundreds of unreliable density estimates. The surveyors' recording of the empirical distance, bearing, and diameter of witness trees documented the exact sampling design. The intended design and deviation from it are investigated with a combination of descriptive statistics, probability theory, computer simulations, analogue geometric models, and modern stand conditions. Herein, analyst bias is eliminated using the robust Morisita II density estimator matching the predominant sampling design of two trees in opposing semicircles. Six widespread surveyor biases deviating from the nearest tree to the corner are evaluated. Quadrant bias and diameter bias for medium‐sized trees are subsumed under newly framed design and small tree biases. Two novel surveyor positional biases (pair angle and near‐post) are introduced here. Previously recognized azimuthal and species biases are analyzed with new techniques. Widely postulated surveyor bias for certain species was found to be minimal. Bias correction and density estimation are applied in detail over 68 townships in northern Wisconsin. The estimated historical forest density in northern Wisconsin, corrected for bias and small tree truncation, averaged 323 trees/ha ≥20 cm. Over 80 Midwestern subregions, surveyors bypassed an estimated 17% of the nearest trees due to their position, resulting in an average bias correction of +24% over the base density. If censored trees below a 20‐cm “veil‐line” are considered, the surveyors bypassed 48% of the nearest trees >12.7 cm in diameter. This study resolves a 70‐year‐old conundrum of surveyor and analyst biases in historical density estimation.