Survey Mode Research Articles

Bots in web survey interviews: A showcase

Cost- and time-efficient web surveys have progressively replaced other survey modes. These efficiencies can potentially cover the increasing demand for survey data. However, since web surveys suffer from low response rates, researchers and practitioners start considering social media platforms as new sources for respondent recruitment. Although these platforms provide advertisement and targeting systems, the data quality and integrity of web surveys recruited through social media might be threatened by bots. Bots have the potential to shift survey outcomes and thus political and social decisions. This is alarming since there is ample literature on bots and how they infiltrate social media platforms, distribute fake news, and possibly skew public opinion. In this study, we therefore investigate bot behavior in web surveys to provide new evidence on common wisdom about the capabilities of bots. We programmed four bots – two rule-based and two AI-based bots – and ran each bot N = 100 times through a web survey on equal gender partnerships. We tested several bot prevention and detection measures, such as CAPTCHAs, invisible honey pot questions, and completion times. The results indicate that both rule- and AI-based bots come with impressive completion rates (up to 100%). In addition, we can prove conventional wisdom about bots in web surveys wrong: CAPTCHAs and honey pot questions pose no challenges. However, there are clear differences between rule- and AI-based bots when it comes to web survey completion.

Instructed Response Item Attention Checks in Mixed-mode Questionnaires: Evidence from a Probability-based Mail and Online Panel

Mostly in web surveys, attention checks have been proposed to identify inattentive respondents in self-administered surveys as previous research has argued that low-quality answers may introduce severe biases in data analyses. The increasing popularity of mixing survey modes for conducting probability-based surveys amplifies the need for systematic research on the implementation of attention checks in these survey designs. Our study investigates whether an instructed response item (IRI) can be an adequate predictor of response quality in a self-administered mixed-mode panel survey with a probability-based German general population sample. By doing so, we compare the results of the attention check with those of other response quality indicators. However, while the results for the online respondents were somewhat promising, the mail respondents result suggests that this response quality indicator seems reliable, so that we cannot recommend using IRIs in general population mixed-mode surveys, which include a mail option.

On the Line and Online: Higher Non‐Response to Web‐Based Surveys Over‐Represents Avid Recreational Fishers Compared With Telephone Surveys

ABSTRACTRecreational fishing surveys have an important role in providing data to inform fisheries management. The selection of a contact method is an important and often challenging consideration that influences the potential for non‐sampling errors that can result in unrepresentative data and biased estimates. Telephones are used for many off‐site recreational fishing surveys, but widespread use of digital technology suggests that online surveys could be a viable alternative. We compared phone surveys with alternative online surveys using probability sampling from a licence database. Response rates were lower for online surveys (13%–26%) than phone surveys (97%–98%). The proportion of respondents who reported marine fishing in the previous 12 months did not differ between survey methods (98% for both survey modes), but the proportion of avid fishers was higher for online surveys (41%–54%) than phone surveys (37%–40%). These differences reflect higher respondent burden for online surveys where data are self‐administered and respondents are more likely to opt‐out. Higher non‐response and over‐representation of avid fishers from online surveys may compromise data reliability. We recommend comparative studies to quantify and correct for biases when evaluating alternative survey modes for new or ongoing surveys.

Monetary Incentives in Clinician Surveys: An Analysis and Systematic Review With a Focus on Establishing Best Practices.

Surveys involving health care providers continue to be characterized by low and declining response rates (RRs), and researchers have utilized various strategies to increase survey participation. An important approach is to employ monetary incentives to improve survey response. Using a systematic review and analyses of 100 randomized comparisons (published in 48 papers) between monetary incentives and a non-incentive condition, this paper seeks to advance the understanding of best practices for using monetary incentives in clinician surveys. These analyses show even small incentives (≤$2) to be effective in improving clinician response relative to non-incentive subgroups, with diminished returns associated with serial incremental increases above that amount up to amounts greater than $25, at which point there is an appreciable improvement, supporting the use of higher incentives in this population. Cash and direct cash equivalents (e.g., cash cards and checks) produced greater odds of survey participation compared to vouchers, lotteries and charitable contributions, with lotteries and charities being the least effective forms of monetary incentive. Survey mode, timing and ethical considerations are also addressed. Noting the challenges associated with surveying clinicians, researchers must make every effort to improve access to this difficult-to-reach population by implementing appropriate incentive-based strategies designed to improve participation rates.

Environmental Attitudes, Concerns, and Behaviors Across Survey Modes. Assessing Selection and Measurement Biases in ISSP 2020 ‘Push-to-Web’ Surveys

How much does survey mode matter for predicting environmental attitudes and behaviors? It is essential to consider the influence of survey mode on the measurement of variables, particularly given the increasing popularity of web-based data collection following the Covid crisis. In 2020, several ISSP countries adopted a mixed-mode design, combining web and paper surveys for the Environment IV module. While mixed-mode data collection can reduce mode-specific errors, its success depends on accurately estimating and adjusting for selection and measurement differences between the modes. Failing to do so can increase the Total Survey Error, compromising the accuracy and reliability of the data. In this article, we examine selection and measurement differences in the web and paper modes in the Environment IV data from Switzerland and Finland. Register data are used as a benchmark to estimate selection biases in key socio-demographic variables, and we use these estimates to disentangle mode measurement effects from other sources of error. While we find that age, education, and labor market status account for selection effects across modes, we do not find significant differences in the measurement of environmental attitudes and behaviors across modes once the selection effect is controlled for. Mixing online and paper modes should thus not compromise data quality by introducing measurement biases for environmental variables.

Survey response rates in European comparative surveys: a 20-year decline irrespective of sampling frames or survey modes

AbstractThis paper assesses trends in three survey outcome rates within four prominent crossnationalcomparative surveys conducted in European countries in the 21st century: theEuropean Quality of Life Survey, the European Social Survey, the European ValuesStudy, and the International Social Survey Programme. These projects are recognisedfor their high-quality sampling and fieldwork procedures, extensive track records, andcommitment to rigorous methodological standards. The analysis is based on 753national surveys conducted on probability samples of the general population in 36European countries from 1999 to 2018. We investigated whether two essential surveycharacteristics, namely sampling frames and data collection modes, moderated thedecrease of survey outcome rates over time. To analyse these relationships, thesurvey year was included as the explanatory variable, and we applied multi-level linearregressions with surveys nested within countries. Additionally, the project name wasincorporated as a fixed factor, and the sampling frame and mode of data collectionwere control variables for the effect of time. Our study provides valuable insights intothe challenges of conducting high-quality Pan-European cross-national comparativesurveys over nearly two decades. We observed a consistent decline in survey outcomerates, irrespective of country or project. Neither the sampling frame nor the datacollection mode moderated this decline. Hence, even though personal register samplesand Face-to-Face interviews are often regarded as enhancements to overall surveyquality, their application does not effectively counter the factors causing a decline insurvey outcome rates.

Physical and Mental Health of Informal Carers from Culturally and Linguistically Diverse (CALD) and Non-CALD Groups in Australia.

Empirical evidence shows that many family carers, especially immigrants, experience considerable health disadvantages and poorer quality of life. Australia has a rapidly increasing multicultural population, officially referred to as Culturally and Linguistically Diverse (CALD) people. This paper explores similarities and differences in the carer profile and physical and mental health of CALD and non-CALD family carers. A cross-sectional anonymous survey was conducted of self-reported family carers aged 18 years and older. Identical paper and online survey modes were provided to enable choice. Key variables included demographic and carer profile, diagnosed chronic physical health conditions, and validated scales such as CESD-12 and MOS-SF12, including derivative composite Physical and Mental Component Summary (PCS and MCS, respectively) scores. The sample comprised 649 participants (CALD = 347, non-CALD = 302). The analyses included univariate, bivariate, and multivariable linear regression analyses for three outcome variables: PCS, MCS, and CESD-12. CALD carers were comparatively younger and married, and 54% had university-level education (29% in the gfvnon-CALD group). Women were primary carers in both groups (67.4% versus 72.2%). The weekly care hours were higher for non-CALD carers. Both groups had below population-referenced scores for mean PCS and MCS values. For CESD-12, non-CALD respondents had higher scores (17.5 vs. 11.2, p < 0.022). Regression analyses showed significant differences for demographic, carer, and physical health variables across the three outcome variables. Women have a higher domestic workload, which, when combined with high care hours, adversely impacts physical and mental health. The need for improved and culturally aligned care support systems is required.

Examining Bhutanese Science Teachers’ Epistemic Views of Scientific Inquiry

AbstractScientific inquiry is regarded as the bedrock of science education in Bhutan. Bhutanese science teachers, for example, are increasingly required to possess accurate and deep epistemic views of scientific inquiry. Hence, this cross-sectional study was carried out to examine Bhutanese science teachers’ epistemic views of scientific inquiry. The study recruited 301 science teachers using convenience and snowball sampling procedures. Data was collected using Views About Scientific Inquiry (VASI) questionnaire administered through an online survey mode and analysed using descriptive statistics and inferential statistical methods. Findings from this study revealed Bhutanese science teachers as being ignorant of epistemic aspects related to questions and hypotheses, procedures of investigations, results, and procedures of investigations, scientific data and scientific evidence, and scientific explanations and scientific theories. The independent sample t-test revealed no significant difference between Bhutanese male and female science teachers’ epistemic views of scientific inquiry (p &gt; .05). The one-way ANOVA revealed significant differences amongst Bhutanese science teachers’ epistemic views of scientific inquiry based on academic qualification (p &lt; .05). The Tukey HSD post hoc test, however, showed the differences existing only between science teachers with master’s degree and certificate qualification in favour of the former (p &lt; .05). The three-way ANOVA revealed Bhutanese science teachers’ epistemic views of scientific inquiry as being independent of individual and interaction effects of school type, teaching subject, and teaching experience (p &gt; .05).

The extent and diversity of harm from the drinking of unknown others in Australia: An analysis of data from during the COVID-19 pandemic.

This paper aims to produce a comprehensive estimate of alcohol's harm to others (AHTO) from strangers among Australia adults in 2021. This survey was undertaken during COVID-19 and aims to compare AHTO results with those from 12 years before and identify differences across socio-demographics. Cross-sectional data of 2574 Australian adults were collected in November 2021 via two survey modes: random digit dialling and the Life in Australia™ panel. Questions pertained to harms from the drinking of known and unknown others ('strangers') in the previous 12 months. Multivariable logistic regression is used to analyse differences in the experience of AHTO from strangers across age, gender and other socio-demographics. Under half of Australian adults reported experiencing AHTO from a stranger (42.2%) during the period of COVID-19. Women were at significantly higher odds of reporting harm than men. Significantly higher percentages of participants aged 18-49 reported harm than those 65 and over. Rates of experience of harm from strangers' drinking varied between different Australian regions. Adults engaging in less frequent risky drinking (less than 4 days a month) reported significantly greater experiences of harm than those not engaging in risky drinking in the past year. The current study provides a national estimate of AHTO from strangers in Australia during the COVID-19 pandemic. Finding greater risk of experiencing AHTO from strangers among women than men differs from previous AHTO literature, and further research is required to determine if this is an emerging trend.

Estimating public opinion from surveys: the impact of including a “don't know” response option in policy preference questions

Abstract What are the consequences of including a “don't know” (DK) response option to attitudinal survey questions? Existing research, based on traditional survey modes, argues that it reduces the effective sample size without improving the quality of responses. We contend that it can have important effects not only on estimates of aggregate public opinion, but also on estimates of opinion differences between subgroups of the population who have different levels of political information. Through a pre-registered online survey experiment conducted in the United States, we find that the DK response option has consequences for opinion estimates in the present day, where most organizations rely on online panels, but mainly for respondents with low levels of political information and on low salience issues. These findings imply that the exclusion of a DK option can matter, with implications for assessments of preference differences and our understanding of their impacts on politics and policy.

Recruiting people with selected citizenships for the health interview survey GEDA Fokus throughout Germany: evaluation of recruitment efforts and recommendations for future research

BackgroundGermany is the second most common country of immigration after the US. However, people with own or familial history of migration are not represented proportionately to the population within public health monitoring and reporting. To bridge this data gap and enable differentiated analyses on migration and health, we conducted the health interview survey GEDA Fokus among adults with Croatian, Italian, Polish, Syrian, or Turkish citizenship living throughout Germany. The aim of this paper is to evaluate the effects of recruitment efforts regarding participation and sample composition.MethodsData collection for this cross-sectional and multilingual survey took place between 11/2021 and 5/2022 utilizing a sequential mixed-mode design, including self-administered web- and paper-based questionnaires as well as face-to-face and telephone interviews. The gross sample (n = 33436; age range 18–79 years) was randomly drawn from the residents’ registers in 120 primary sampling units based on citizenship. Outcome rates according to the American Association for Public Opinion Research, the sample composition throughout the multistage recruitment process, utilization of survey modes, and questionnaire languages are presented.ResultsOverall, 6038 persons participated, which corresponded to a response rate of 18.4% (range: 13.8% for Turkish citizenship to 23.9% for Syrian citizenship). Home visits accounted for the largest single increase in response. During recruitment, more female, older, as well as participants with lower levels of education and income took part in the survey. People with physical health problems and less favourable health behaviour more often took part in the survey at a later stage, while participants with symptoms of depression or anxiety more often participated early. Utilization of survey modes and questionnaire languages differed by sociodemographic and migration-related characteristics, e.g. participants aged 50 years and above more often used paper- than web-based questionnaires and those with a shorter duration of residence more often used a translated questionnaire.ConclusionMultiple contact attempts, including home visits and different survey languages, as well as offering different modes of survey administration, increased response rates and most likely reduced non-response bias. In order to adequately represent and include the diversifying population in public health monitoring, national public health institutes should tailor survey designs to meet the needs of different population groups considered hard to survey to enable their survey participation.

Differences in fisher demographics and fishing behaviour from concurrent phone‐recall and smartphone app surveys of recreational angling in south‐western Australia

AbstractTraditional methods for collecting robust samples of recreational effort and catch data from offsite mail or telephone surveys are becoming increasingly difficult to obtain. Smartphone applications (apps) that allow recreational fishers to report their effort and catch are a potentially viable data collection tool, but may be biased. We evaluated demographics, effort, and catch of freshwater anglers in south‐west Western Australia by comparing independent data from concurrent phone‐recall and app‐diary surveys for a 12‐month period in 2017–2018. Survey modes differed, with higher proportions of app respondents angling in freshwater and being avid anglers. For both survey modes, unweighted mean effort (days fished) was higher in dams than rivers, but was higher from the phone‐recall survey for dams and rivers. The unweighted distribution of kept and released catches (number of fish per angler) of rainbow trout (Oncorhynchus mykiss) and redfin (Perca fluviatilis) were similar between survey modes, with higher retention of redfin and higher release of rainbow trout. Kept and released rates of freshwater cobbler (Tandanus bostocki) and brown trout (Salmo trutta) were less common in both survey modes. Lower response rates and higher survey participation by avid anglers (≥20 days per year) from the app‐diary survey may bias survey data. Such bias needs to be addressed when using digital data collection methods, along with improving recruitment and retention of app participants.

A Second Chance at Response: A Multimode Reinterview Analysis Using Data from the National Survey on Drug Use and Health

Traditionally administered in-person, restrictions imposed by the COVID-19 pandemic prompted the National Survey on Drug Use and Health to transition to a multimode survey that offers a self-administered web option prior to in-person data collection. A relatively small portion of respondents who began the survey on the web but broke off prior to completing were later readministered the survey (from the beginning) in-person. We refer to these as “second chance” respondents. The aim of this in-brief note is to report the individual-level mode discrepancies these second chance respondents exhibited with respect to lifetime use of cigarettes, alcohol, and marijuana. In addition to higher-than-anticipated discrepancy rates overall, we found directional differences that run contrary to what would be expected from the literature on survey mode effects. Second chance respondents are more than 15 percentage points more likely to report ever smoking cigarettes or using marijuana than web-only respondents.

Youth use of electronic nicotine delivery systems (ENDS) in the Population Assessment of Tobacco and Health (PATH) Wave 6: Impact of survey mode

BackgroundThe Population Assessment of Tobacco and Health (PATH) provides annual prevalence data on youth use of electronic nicotine delivery systems (ENDS). However, trends may be complicated by COVID-related changes in survey mode (self-completed vs. telephone interview) across 2019–2021. MethodsTrends in past 30-day (P30D) ENDS use over PATH waves 5 (2019; 100 % self-completed), 5.5 (2020; 100 % telephone interview), and 6 (2021; 33.5 % self-completed, 66.5 % telephone interview) were examined among continuing youth, overall and within survey mode. Further analyses examined the nature of these changes by examining 1) potential response biases in social contexts of nicotine use, and 2) sources of ENDS over time. ResultsTelephone interviewees were less likely to report ENDS use (4.6 % vs. 8.6 % for self-completers), and more likely to report social disapproval of nicotine use, suggesting a reporting bias in telephone interviews. Survey-mode-naïve analyses suggested a large decline in P30D ENDS use prevalence between 2019–20 (10.2–4.6 %) followed by an apparent uptick in 2021 (5.9 %); however, comparing like-to-like survey modes showed a more modest decline (10.2 % in 2019; 8.6 % in 2021; self-completed) with no change between 2020 and 2021 (4.6 % in both; telephone interviews). Analyses suggested that the mode effects were partly, but not wholly, related to social desirability effects. DiscussionChanges in PATH survey mode introduced artifacts into ENDS use prevalence, possibly due in part to social desirability bias suppressing reporting in telephone interviews, rather than a true uptick. It is essential to account for survey mode in PATH surveys.

Getting the Race Wrong: A Case Study of Sampling Bias and Black Voters in Online, Opt-In Polls

Abstract Researchers are increasingly reliant on online, opt-in surveys. But prior benchmarking exercises employ national samples, making it unclear whether such surveys can effectively represent Black respondents and other minorities nationwide. This paper presents the results of uncompensated online and in-person surveys administered chiefly in one racially diverse American city—Philadelphia—during its 2023 mayoral primary. The participation rate for online surveys promoted via Facebook and Instagram was .4%, with White residents and those with college degrees more likely to respond. Such biases help explain why neither our surveys nor public polls correctly identified the Democratic primary’s winner, an establishment-backed Black Democrat. Even weighted, geographically stratified online surveys typically underestimate the winner’s support, although an in-person exit poll does not. We identify some similar patterns in Chicago. These results indicate important gaps in the populations represented in contemporary opt-in surveys and suggest that alternative survey modes help reduce them.

Nonresponse in name generators across countries and survey modes

Past research indicates interviewer effects lead to an underestimation of network size and higher nonresponse to the “important matters” name generator. Self-administered surveys offer a potential solution, but evidence is mixed and context-specific. We employ a logistic multilevel regression, estimated using a Bayesian Markov Chain Monte Carlo approach, to analyze nonresponse to this name generator from 33 post-electoral surveys across 21 countries in the Comparative National Election Project. We find higher nonresponse in interviewer-administered surveys compared to self-administered surveys, particularly among specific demographic groups. Finally, we discuss the trade-offs in selecting survey modes for collecting ego-network data using this instrument.

The effect of prepaid incentives on panelists’ response across survey modes in a sequential mixed-mode design

AbstractWhile the use of prepaid incentives and data collection in a sequential mixed-mode survey design is standard in cross-sectional and longitudinal studies, little is known about the interaction of these strategies and how this affects target persons’ survey participation. Therefore, based on a panel study, this study investigates whether such unconditional prepayments—monetary or non-monetary incentives—increase the efficiency of a sequential mixed-mode survey design in regard to boosted response rates and speed of return after receipt of an invitation to participate in a survey. While the survey design is always identical, the different incentives vary across the waves considered. Different prepayments, such as cash or in-kind incentives, have different effects on invitees’ participation in different survey modes. In particular, non-monetary incentives seem to work only in a sequential mixed-mode design, while cash, as a universal medium, always works in the expected way, across different survey modes. In sum, the study finds that when a single sequential mixed-mode design is used across panel waves the overall response rates are rather similar, even when different incentives are provided.

Comparing responses from a paper-based survey with a web-based survey in environmental criminology

This article aims to compare the pattern of responses obtained by a web-based and a paper-based survey used to investigate the transit safety of travellers in railway stations in Sweden. This aim is achieved by evaluating whether the response and the completion rates change as the surveys progress, assessing the effect of the survey mode on respondents’ answers (after controlling for the surveys’ internal consistency and differences in the samples), and the potential impact of the order of alternatives in multiple-choice questions on the responses. To carry out the study, a sample of 500 responses was taken from each population and later compared using a series of statistical tests. Findings indicate that despite the surveys’ high internal consistency, the prevalence of victimisation, fear of crime, and precautions detected in the web survey was higher than those found in the paper survey. The web survey shows a major drop just after the initial questions, while the paper survey shows a more stable pattern of responses, but was also affected by a single compulsory question that pushed the completion rate down. Finally, the order of alternatives in multiple-choice questions (fixed or random) did not affect the answers given by the respondents, providing a solid base for safety interventions in transit environments, regardless of survey mode. The article concludes by making suggestions for both research and practice.

Feasibility of replacing face-to-face with telephone interviews for the World Mental Health Qatar survey during the COVID-19 pandemic.

We investigated the feasibility of replacing face-to-face with telephone interviews conducted as part of the World Mental Health Qatar (WMHQ) survey and discuss the main methodological changes across the two pilots that were subsequently implemented in the full-scale WMHQ telephone survey. We assessed the net mode effect by comparing the lifetime prevalence estimates of the main mental disorder classes (mood and anxiety disorders) and a number of disorders across the two survey pilots conducted prior to and post-pandemic. The main differences in terms of methodology for both pilots stemmed from differences in the survey mode, including questionnaire length, study recruitment method, and fielding team size and structure. These factors influenced response rates and costs. However, the lifetime prevalence estimates and other key indicators of survey results did not differ across modes. Our findings confirm the comparability of data collected via telephone and face-to-face modes, supporting the adoption of telephone surveys for future mental health studies, particularly in the context of pandemics. They also confirm the feasibility of changing or mixing modes depending on field conditions in future psychiatric epidemiological research.

Participation in a 2-year weekly phone surveillance system: results from Lesotho

Interest in cell phones as a valid survey mode in sub–Saharan Africa increased precipitously in the past decade. However, the mode has mainly been used for cross-sectional studies. This paper presents enrollment and participation rates for a two-year (July 2020 – June 2022), weekly population-based computer-assisted telephone interview (CATI) surveillance system in Lesotho. We found high participation and share lessons learned.