Survey Response Bias Research Articles

The Effect of Communication Emails on Web Survey Response Rate, Representativeness, and Response Bias: Results from a Factorial Randomized Control Trial in a College Student Population

The Effect of Communication Emails on Web Survey Response Rate, Representativeness, and Response Bias: Results from a Factorial Randomized Control Trial in a College Student Population

Understanding, Detecting, and Adjusting for Response Bias in Surveys

Understanding, Detecting, and Adjusting for Response Bias in Surveys

Gender gaps in financial literacy: a multi-arm RCT to break the response bias in surveys

Gender gaps in financial literacy: a multi-arm RCT to break the response bias in surveys

Choir of believers? Experimental and longitudinal evidence on survey participation, response bias, and public service motivation

Collecting data through surveys are a vital method in public administration research, but an overseen challenge for the validity of survey research is response bias, which is when response behavior correlates with study outcomes. Response bias can be particularly threatening in motivation research because motivation is an important determinant for behavior and most likely also for survey participation. Specifically, public service motivation (PSM) can increase the willingness to devote time and effort to respond to surveys for the benefit of others, whereas extrinsically motivated individuals may be less inclined to participate in surveys without prospects of economic compensation. This article examines motivation and response bias by use of a preregistered field experiment and two panel datasets. The experimental study shows that a monetary incentive increases the response rate, whereas a PSM-oriented appeal does not. Furthermore, the increased response rate of the incentive is achieved without any detectable response bias on PSM. Likewise, the panel studies offer little and mixed support for response bias related with PSM. This evidence suggests that (1) PSM is not a (strong) driver of response bias, and that (2) extrinsic motivational tools applied to PSM surveys are not necessarily associated with response bias.

Gender Gaps in Financial Literacy: A Multi-Arm Rct to Break the Response Bias in Surveys

Gender Gaps in Financial Literacy: A Multi-Arm Rct to Break the Response Bias in Surveys

Clinical adult outcome 11-30 years after pediatric epilepsy surgery: Complications and other surgical adverse events, seizure control, and cure of epilepsy.

Pediatric epilepsy surgery promises seizure freedom or even cure of epilepsy. We evaluated the long-term (≥10 years) adult clinical outcome including surgery-related adverse events and complications, which are generally underreported. A monocentric, single-arm, questionnaire study in now adult patients who underwent epilepsy surgery during childhood. A novel ad hoc parental/patient questionnaire, which addressed diverse outcome domains was applied. From a total of 353 eligible patients, 203 could be contacted (3 patients died of causes unknown) and 101 (50%) returned appropriately filled-in surveys. No evidence for a survey-response bias was found. The rate of surgical complications according to the patient records was 9%. As regards the survey, half of the parents/patients reported surgical adverse events (expected and unexpected issues) and one-third reported permanent aversive sequels. Two-thirds of the patients were seizure-free during the last year before follow-up; 63% were Engel class 1A; favorable seizure outcomes (including auras only) were obtained in 73%; and 54% were seizure-free and off antiseizure medicine (ASM), that is, cured of epilepsy. In non-seizure-free patients, seizure relapse occurred at any time during the follow-up interval but 87% of those with a seizure-free first postoperative year were seizure-free at follow-up. One patient experienced a seizure relapse during the ASM withdrawal trial but became seizure-free again with ASMs. Eleven patients reported an increased number of ASMs as compared to the time before surgery. Earlier focal surgery did not affect the long-term clinical outcome. Pediatric epilepsy surgery was capable of curing epilepsy in about one-half of the children and to significantly control seizures in about three-fourths. Long-term success of focal surgery did not depend on age at surgery or duration of epilepsy. Surgical adverse events including complications may be underreported and must be assessed more thoroughly.

Characteristic Distinctions Between Pre-/Post-COVID-19 Teledermatology Adoptees: A Cross-Sectional United States-based Analysis and the Implications for Dermatologic Healthcare Equity.

Studies suggest potential heterogeneity in telemedicine adoption with potential to exacerbate healthcare access inequity. A pre-validated survey was electronically sent to a proprietary listserv of practicing US-based dermatologists. Results were stratified by when teledermatology was adopted. Chi-square and odds ratios (OR) with 95% confidence intervals (95%CI) were used to analyze categorical data while single-factor ANOVA with posthoc Tukey-Kramer was used for continuous data. 338 practicing US-based dermatologists completed the questionnaire. Academic/Government dermatologists were 4-times more likely (OR 4.08, 95%CI 2.37-7.03) to adopt teledermatology pre-COVID than private-practice dermatologists. Dermatologists with ≤10 years of experience were 1.8-times (OR 1.8, 95%CI 1.01-3.18) and 2.82-times more likely (OR 2.82, 95%CI 0.78-10.25) to adopt teledermatology pre-COVID-19 or at all, respectively, compared to dermatologists with ≥20 years of experience. Teledermatology adopters practiced more medical-dermatology (P<.0001) than non-adopters, who reported practicing more dermatologic surgery (P=.003; Tukey-Kramer α<.05) and dermatopathology (P<.0001; Tukey-Kramer α<.05). Pre-COVID-19 adopters were 4-times more likely (OR 4.69, 95%CI 1.46-15.07) to switch/incorporate live-interactive-only teledermatology (LI) post-COVID-19. Post-COVID-19 adopters were 6-times more likely (OR 6.09, 95%CI 3.36-11.06) to utilize LI than Pre-COVID-19 adopters. Pre-COVID-19 adopters use teledermatology for a larger proportion of patient visits than Post-COVID-19 adopters (19.6% v 10.4%, P<.0001), but also are 3.43-times more likely (OR 3.43, 95%CI 1.82-6.46) to report future decreases in usage. Cross-sectional retrospective survey and potential response bias. Current teledermatology usage may be a suitable tool for medical-dermatology-focused practices. Material hurdles still exist for procedurally-oriented practices and future studies should investigate these barriers to maximize equitable access to dermatological care. J Drugs Dermatol. 2023;21(1):101-104. doi:10.36849/JDD.7169.

The effect of sampling mode on response rate and bias in elite surveys

The literature frequently recommends purposive sampling of elites based on the assumptions that random sampling negatively affects the response rate and that it induces bias. I test these assumptions drawing on metadata from 282 samples of political, economic, and social elites, and on microdata from 2,658 elites. First I use permutations to calculate confidence intervals for the expected response rate following each sampling method. Second, I estimate the effect of random sampling on the final response rate using a range of regression models. Finally, I compare the distributions of the estimators for the average age, the share of male elites, and elites’ ideology by simulating repeated random and purposive samples. Results indicate that both random and purposive sampling of elites generate sufficiently large samples, as well as consistent and unbiased estimators of population parameters. Contradicting methodological guidelines in the field, the conclusion is that random sampling of elites is efficient.

S3292 Differences in Gastroenterologists’ Self-Reported Race Affect Fear Levels During Endoscopic Procedures: RESULTS From a US National Survey

Introduction: Racial disparities continue to exist during the COVID-19 pandemic. A systematic review found that Black and Hispanic patients experience disproportionate COVID-19 hospitalizations and higher morbidity and mortality. Low level evidence in that review suggested that Asians' outcomes are similar to those of non-Hispanic whites. As most practicing gastroenterologists are White, and as endoscopies are aerosolizing procedures risking viral transmission, this study evaluates differences in fear levels among gastroenterologists from different racial backgrounds. Methods: This IRB-approved multi-center cross-sectional study used a snowball sampling approach to disseminate a 42-question survey, pilot-tested for reliability and validity, to gastroenterologists across different geographic locations in the US. Fear levels during endoscopic procedures on suspected/confirmed COVID-19 patients were assessed using a 1-10 Likert scale, with 10 being the greatest fear. Fear was assessed at three points for upper and lower endoscopies: first procedure, subsequent before the COVID-19 vaccine, and subsequent after the vaccine. Data was analyzed using Pearson's chi-squared, Mann-Whitney U, and Wilcoxon rank tests and significance was determined at p<0.05 Results: We analyzed 69 responses from gastroenterologists at 30 sites. Of the respondents, 39 selfidentified as white, 3 as Black, 6 as Latinx, 4 as Middle Eastern, and 17 as Asian. Due to the small sample size, non-White racial groups were combined to evaluate the impact of race on reported fear level reported by gastroenterologists who performed endoscopies during the COVID-19 pandemic. Fear levels at the six assessed points were not significantly different when comparing Asians to Whites, and when comparing Blacks and Latinx to Whites. However, Asians, Blacks, and Latinx had a higher fear level post vaccine when performing lower endoscopies compared to Whites (mean (m) 53.59, standard deviation (SD)=2.83 vs m=1.94, SD=1.083, respectively;P=0.045). Conclusion: While results can be due to chance due to the small sample size and survey response bias, our study suggests that after receiving the COVID-19 vaccine, minority endoscopists experienced greater fear than White endoscopists while performing lower endoscopies. It is possible that the disproportionate COVID-19 disease burden in Black and Hispanic communities negates the decrease in fear levels that was experienced by White endoscopists even after receiving the COVID-19 vaccine..

Willingness to pay to standardize patient medication information

ABSTRACT Pharmacists dispense prescription drugs with leaflets containing information about the specific medication a patient receives. These leaflets vary in format, length, content, and readability for the same prescription drug at different pharmacies; they also vary for different prescription drugs at the same pharmacy. Previous research has found that most patients prefer standardization of these leaflets; however, standardization is costly, and there are no published studies that estimate the value of such standardization to consumers. We use a contingent valuation survey to investigate consumer willingness to pay for standardized informational leaflets in the retail pharmacy setting. We present results of analyses based on contingent valuation responses of 510 federal government employees. The survey design was a double-bounded advisory referendum elicitation format where respondents were presented with examples of standardized and non-standardized prescription drug information formats. The study found the willingness to pay for standardized informational leaflets was approximately $1.37 per household per month. The estimated willingness to pay is sensitive to alternative econometric specifications, evidence of possible survey response bias; however, across all models, the estimates are statistically and economically significant.

Survey response bias and the ‘privacy paradox’: evidence from a discrete choice experiment

ABSTRACT The discrepancy between the stated and revealed value of personal information is frequently referred to as the ‘privacy paradox.’ We test for evidence that survey response bias contributes to the privacy paradox through a discrete choice survey experiment based on hypothetical ride-hailing services, some of which track the rider’s location and share it with third parties. The value that respondents place on their personal location data is measured by the willingness to pay higher prices to avoid rides with location sharing. Our survey experiment compares a within-subjects design in which location sharing was presented as an attribute of the hypothetical rides within each choice scenario to a between-subjects design in which location sharing was not presented as an attribute – instead, the location-sharing practices of the hypothetical firms were presented to a treatment group before the respondents made their choices. On average, survey respondents placed a positive value on their personal location data under both survey designs. The difference in value between the survey designs was statistically insignificant, indicating there is no evidence that the within-subjects design led to the type of survey response bias that might contribute to the privacy paradox.

What’s the Benefit of a Video? The Effect of Nonmaterial Incentives on Response Rate and Bias in Web Surveys

Traditional survey research faces declining response rates due to changing cultural habits and technological developments. Researchers have developed novel approaches to increase respondents’ likelihood of participating in web surveys. However, we lack information about whether these methods indeed increase response rates and, if so, whether they bias the resulting data. This article focuses on the use of nonmaterial incentives in the form of a video that provides the invitees with information tailored to their life situation. Analysis of our experimental data shows that instead of increasing respondents’ probability of starting the survey, the video treatments actually decrease it. We provide evidence that the lower salience of the intrinsic benefits of survey participation in the invitation email significantly contributes to this reduction. Additionally, the effect of the nonmaterial incentive differs across subgroups, affecting nonresponse biases in line with employment status, gender, and migration background. We therefore conclude that using additional information in the form of a video as a nonmaterial survey incentive is only suitable under specific conditions.

Association of Resilience and Social Networks with Pain Outcomes Among Older Adults.

Depression, stress, and poor sleep have been associated with increased pain among older adults; positive resources, such as resilience and social networks, may help to buffer the impacts of these negative attributes on pain outcomes. The primary objective was to determine the relative effects of positive resources and negative attributes on pain outcomes among older adults with diagnosed back pain, osteoarthritis, and rheumatoid arthritis. The stratified study sample was identified from older adults ages ≥65 years. Members received a survey assessing positive resources (resilience, social networks), negative attributes (depression, stress, poor sleep), and pain outcomes (severity, interference). Opioid and other medication use was determined from pharmaceutical claims. After weighting to representative distributions of pain conditions and adjusting for survey response bias, multinomial logistic regressions were used to determine the relative associations of positive and negative attributes on pain outcomes. Among survey respondents (N = 4161), prevalence of self-reported pain severity and interference for no/mild, moderate, and severe categories was 61%, 21%, and 18%, and 67%, 16%, and 17%, respectively. In bivariate models, negative attributes of depression, stress, and poor sleep had stronger associations with pain severity and interference than the moderating effects of positive resources of high resilience and diverse social networks. In fully adjusted multivariate models, the strongest associations with moderate and severe pain severity and interference remained depression, stress, and poor sleep. Based on these results, multidimensional pain management strategies should include management of negative attributes along with enhancement of positive resources for effective management of chronic pain.

Advancing regulatory science and assessment of FDA REMS programs: A mixed-methods evaluation examining physician survey response.

Food and Drug Administration's (FDA) Draft Guidance for Industry on pharmaceutical REMS (Risk Evaluation and Mitigation Strategies) assessment and survey methodology highlights physician knowledge-attitudes-behaviors (KAB) surveys as regulatory science tools. This mixed-methods evaluation advances regulatory science and the assessment of FDA REMS programs when using physician surveys. We: (1) reviewed published physician survey response rates; and (2) assessed response bias in a simulation study of secondary survey data using different accrual cut-off strategies. A systematic literature review was conducted of US physician surveys (2000-2014) on pharmaceutical use (n = 75). Kruskal-Wallis tests were used to examine the relationships between response rates and survey design characteristics. The simulation was conducted using secondary data from a population-based physician KAB survey on diabetes risk management with antipsychotic use in Missouri Medicaid (n = 973 accrued over 30 weeks). Survey item responses were compared using Pearson's chi-square tests for two faster completion simulations: Fixed Sample (n = 300) and Fixed Time (8 weeks). Survey response rates ranged from 7% to 100% (median = 48%, IQR = 34%-68%). Surveys of targeted populations and surveys using member lists were associated with higher response rates (p = 0.02). In the simulation, 9 of 20 (45%) KAB items, including diabetes screening advocacy, differed significantly using the smaller Fixed Sample strategy (achieved in 12 days) versus full accrual. Fewer response differences were found using the Fixed Time strategy (2 of 20 [10%] items). Published data on physician surveys report low response rates with most associated with the sample source selected. FDA REMS assessments should include formal evaluation of survey accrual and response bias.

Assessment of Response Bias Is Neglected in Cross-Sectional Blindness Prevalence Surveys: A Review of Recent Surveys in Low- and Middle-Income Countries

ABSTRACTPurpose: Findings from cross-sectional blindness prevalence surveys are at risk of several biases that cause the study estimate to differ from the ‘true’ population prevalence. For example, response bias occurs when people who participate (‘responders’) differ from those who do not (‘non-responders’) in ways that affect prevalence estimates. This study aimed to assess the extent to which response bias is considered and occurs in blindness prevalence surveys in low- and middle-income countries (LMICs).Methods: We searched MEDLINE, EMBASE and Web of Science for cross-sectional blindness prevalence surveys undertaken in LMICs and published 2009–2017. From included studies, we recorded and descriptively analysed details regarding enumeration processes, response, and non-response, including the impact of non-response on results.Results: Most (95%) of the 92 included studies reported a response rate (median 91.7%, inter-quartile range 85.9–95.6%). Approximately half clearly described enumeration processes (49%), and reported at least one strategy to increase the response rate (53%); a quarter (23%) statistically compared responders and non-responders. When differential response was assessed, men were more likely to be non-responders than women. Two-thirds (65%) of the time a sociodemographic difference was found between responders and non-responders, a difference in blindness prevalence was also found. Only 13 studies (14%) commented on implications of non-response on prevalence estimates.Conclusions: Response rates are commonly reported from blindness prevalence surveys, and tend to be high. High response rates reduce—but do not eliminate—the risk of response bias. Assessment and reporting of potential response bias in blindness prevalence surveys could be greatly improved.

Hospital leadership perspectives on the contributions of Ronald McDonald Houses. Results from an international survey.

The purpose of this paper is to describe an international survey of hospital executives and administrators' perspectives on the contributions of their affiliation with a Ronald McDonald House (RMH) as an example of accommodation in family-centered care to the hospital's mission, operations and patient experience. RMHs worldwide provided the names and e-mail addresses of the people holding key leadership positions in their main hospital partner, who in turn were invited to complete an internet-based survey (530 participants; response rate of 54.5 percent). Hospital leaders reported very positive opinions about the contributions of their RMHs affiliation to their ability to serve seriously ill children and their families. This included such important outcomes as increasing family integrity and family participation in care decisions; and decreasing psychosocial stress and hospital social work resource burdens associated with lodging, food, transportation and sibling support. Hospital chief executive offices (CEOs) and medical directors reported very strong and positive opinions regarding the value-added of their RMHs affiliation in many areas, including enhanced marketing of hospital specialty services and reduced length of stay. Survey response bias is a limitation, although the results are still useful in terms of identifying multiple ways in which RMHs are perceived as contributing to hospitals' operations and provision of family-centered care. Overall, the results suggest that, internationally, hospital leaders believe that RMHs play a key and valued role in their provision of family-centered care to seriously ill children and their families. Family accommodation is more than the simple provision of lodging and plays an integral role how hospitals approach family-centered care. This international study contributes to the general literature on the role of family accommodation in hospitals' provision of family-centered care and the specific and very sparse literature on RMHs in particular.

In today's global environment organizational culture dominates national culture!

Purpose– The specific problem this research addresses is whether cultural differences, national or organizational, impact how effectively the continuous improvement process is received within the supply chain in order to improve supplier performance. The paper aims to discuss these issues.Design/methodology/approach– The research design used a mixed methods approach, combining quantitative and qualitative research.Findings– The research found that Canada was most receptive to continuous improvement, with China being the least receptive. The study found that organizational culture was more influential than national culture. Isomorphism and benchmarking is driving continuous-improvement language and methods to be more universally known within business. Business and management practices appear to take precedence in driving change within organizations.Research limitations/implications– The sample size and countries involved was very small and limited to key medium sized distributed power company (MSDPC) suppliers. This limited diversity and may have introduced supplier selection bias, as well as survey response bias.Practical implications– This research concludes that organizational culture is more dominant over national culture and the influence of leadership within the organization drives the impact of continuous improvement.Originality/value– With isomorphism and the fact that businesses want to be successful, continuous improvement language and methods are becoming more universally known. Business and management practices are now taking precedence in driving change within organizations. Organizational culture is now more influential than national culture.

The Wikipedia Gender Gap Revisited: Characterizing Survey Response Bias with Propensity Score Estimation

Opt-in surveys are the most widespread method used to study participation in online communities, but produce biased results in the absence of adjustments for non-response. A 2008 survey conducted by the Wikimedia Foundation and United Nations University at Maastricht is the source of a frequently cited statistic that less than 13% of Wikipedia contributors are female. However, the same study suggested that only 39.9% of Wikipedia readers in the US were female – a finding contradicted by a representative survey of American adults by the Pew Research Center conducted less than two months later. Combining these two datasets through an application and extension of a propensity score estimation technique used to model survey non-response bias, we construct revised estimates, contingent on explicit assumptions, for several of the Wikimedia Foundation and United Nations University at Maastricht claims about Wikipedia editors. We estimate that the proportion of female US adult editors was 27.5% higher than the original study reported (22.7%, versus 17.8%), and that the total proportion of female editors was 26.8% higher (16.1%, versus 12.7%).

Correcting for Scale Perception Bias in Tourist Satisfaction Surveys

Tourist satisfaction analysis provides important information that contributes to design public and private policies in tourist services and destinations. However, the responses to tourist satisfaction scales can lead to biased results when comparing tourists from different nationalities and backgrounds, since they might use different internal scale interpretations to assess their satisfaction level in a survey. In this article, the anchoring vignette methodology is used to test and correct for scale response bias in tourist satisfaction surveys. The results of the empirical application find out that scale response is not homogeneous across tourists from different nationalities and that the correction for scale differences can lead to an evaluation of tourist satisfaction that more accurately informs and facilitates improvements in policy decisions at tourist destinations and operators.

Open versus Closed

The expression of attitudes is very sensitive to variations in survey structure. With a survey with both open and closed questions of a single sample of respondents, an analysis was done on attitudes about public transportation of peak-hour transit users in Arequipa, Peru. It was found that combining both question types helped identify and compensate for the limitations of each. The results of the open and closed questions were dramatically different, even though the same individuals answered both questions. In response to nondirective, open-ended questioning, respondents emphasized experiential concerns such as comfort and travel times. However, when asked to rank transit-related issues in order of importance, respondents overwhelmingly chose social concerns such as pollution and traffic accidents over the experiential concerns they themselves previously mentioned. The psychological forces that may have led to these apparently contradictory results were explored. The paper drew on the belief-sampling model of survey response and social desirability bias to explain why open questioning may have understated the true value respondents placed on social issues, whereas the closed question may have underemphasized the importance of users’ quotidian travel experience. The participants’ willingness to change answers between the two question forms suggests that confirmation bias may not be a significant impediment to the use of mixed question types.