Quality Of Survey Data Research Articles

SHARE Corona Surveys: study profile

At the onset of the COVID-19 pandemic, the Survey of Health, Ageing, and Retirement in Europe (SHARE) was in a unique position to respond to the need for high quality survey data on people’s changing living situations. Implemented as two telephone interviews in the summer of 2020 and 2021 in 27 European countries and Israel, the SHARE Corona Surveys present a great advantage by their integration into the longitudinal, multidisciplinary and ex-ante harmonised design of the SHARE study. This allows researchers to trace changes from the pre-pandemic period, through the different stages of the pandemic, and the post-pandemic situation. This article lays out the research aims and how the two Corona Surveys fit in the general design of SHARE. It presents the main design features of the SHARE Corona Surveys following the survey life cycle. It starts with information on procurement, contracting, funding, ethics, and data protection and sampling, followed by information on instrument design, translations, questionnaire content and interviewer training. Last, fieldwork, panel care and data processing are described. Focused on topics of health behaviour, health care, economics and social relationships, the balanced panel sample of the two SHARE Corona Surveys comprises more than 48,000 interviews and provides valuable information on how the 50+ population coped with the COVID-19 pandemic. The experience of implementing the SHARE Corona Surveys also offers insights into use of agile project management methods for large survey infrastructures and moving towards a multi-mode design in an ongoing panel data collection project.

An empirical evaluation of probing questions investigating question comprehensibility in web surveys

ABSTRACT It is essential for the quality of survey data that questions are easy to understand. We developed four variants of comprehensibility probes measuring the comprehensibility of a survey question as a whole or of individual aspects (vocabulary, sentence complexity, content). We compared the probes on four experimental questions in terms of response distributions and mean comprehensibility scores to assess their sensitivity to capture comprehensibility problems at different levels of granularity, as well as nonresponse and response times as indicators of respondents’ willingness and required effort to answer the probes. We found no differences in the response distributions and mean comprehensibility scores, while item nonresponse and response times differed significantly between the probe versions.

A quantitative approach to determination of survey specification quality for airborne magnetic and radiometric surveys with relation to large-scale compilations

Regional and nationwide geophysical merges consisting of magnetic and radiometric datasets have become standard practice within Australia and some parts of the world. The best practice for geophysical merges is to ensure the highest quality survey data is incorporated in all given areas. However, the traditional means of accomplishing this determination is quite qualitative without a well-defined methodology. Here, a quantitative algorithmic approach is provided, reducing user “guesswork” regarding which survey supersedes others when occupying the same space within geophysical merges. This is accomplished by assessing a wide range of survey specifications and applying weighted scores to their relative importance. This algorithm is applied to 889 datasets flown in New South Wales between 1957 and 2023 and provides a clear efficacy in determination of data quality based on survey design. Results from this algorithm have demonstrated proficiency in ranking surveys from highest to lowest quality and have formed the basis of all statewide geophysical merges in NSW since 2020.

A dataset on survey designs and quality of social and behavioral science surveys during the COVID-19 pandemic

In the social and behavioral sciences, surveys are frequently used to collect data. During the COVID-19 pandemic, surveys provided political actors and public health professionals with timely insights on the attitudes and behaviors of the general population. These insights were key in guiding actions to fight the pandemic. However, the data quality of these surveys remains unclear because systematic knowledge about how the survey data were collected during the COVID-19 pandemic is lacking. This is unfortunate, since decades of survey research have shown that survey design impacts data. Our Survey Data Collection and the COVID-19 Pandemic (SDCCP) project deals with this research gap. We collected rich metadata on survey design for 717 social and behavioral science surveys carried out in Germany during the first two years of the COVID-19 pandemic. In this data descriptor, we present a unique resource for a systematic assessment of the survey data collection practices and quality of surveys conducted in Germany during the COVID-19 pandemic.

Synthetic Replacements for Human Survey Data? The Perils of Large Language Models

Abstract Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.

Statistical assessment of reliability of anthropometric measurements in the multi-site South African National Dietary Intake Survey 2022.

Anthropometric data quality in large multicentre nutrition surveys is seldom adequately assessed. In preparation for the South African National Dietary Intake Survey (NDIS-2022), this study assessed site leads' and fieldworkers' intra- and inter-rater reliability for measuring weight, length/height, mid-upper arm circumference (MUAC), waist circumference (WC) and calf circumference (CC). Standardised training materials and measurement protocols were developed, and new anthropometric equipment was procured. Following two training rounds (12 site lead teams, 46 fieldworker teams), measurement reliability was assessed for both groups, using repeated measurements of volunteers similar to the survey target population. Reliability was statistically assessed using the technical error of measurement (TEM), relative TEM (%TEM), intra-class correlation coefficient (ICC) and coefficient of reliability (R). Agreement was visualised with Bland-Altman analysis. By %TEM, the best reliability was achieved for weight (%TEM = 0.260-0.923) and length/height (%TEM = 0.434-0.855), and the poorest for MUAC by fieldworkers (%TEM = 2.592-3.199) and WC (%TEM = 2.353-2.945). Whole-sample ICC and R were excellent ( > 0.90) for all parameters except site leads' CC inter-rater reliability (ICC = 0.896, R = 0.889) and fieldworkers' inter-rater reliability for MUAC in children under two (ICC = 0.851, R = 0.881). Bland-Altman analysis revealed no significant bias except in fieldworkers' intra-rater reliability of length/height measurement in adolescents/adults ( + 0.220 (0.042, 0.400) cm). Reliability was higher for site leads vs. fieldworkers, for intra-rater vs. inter-rater assessment, and for weight and length/height vs. circumference measurements. NDIS-2022 site leads and fieldworkers displayed acceptable reliability in performing anthropometric measurements, highlighting the importance of intensive training and standardised measurement protocols. Ongoing reliability assessment during data collection is recommended.

Semi-Automated Nonresponse Detection for Open-Text Survey Data

Open-ended survey questions can enable researchers to gain insights beyond more commonly used closed-ended question formats by allowing respondents an opportunity to provide information with few constraints and in their own words. Open-ended web probes are also increasingly used to inform the design and evaluation of survey questions. However, open-ended questions are more susceptible to insufficient or irrelevant responses that can be burdensome and time-consuming to identify and remove manually, often resulting in underuse of open-ended questions and, when used, potential inclusion of poor-quality data. To address these challenges, we developed and publicly released the Semi-Automated Nonresponse Detection for Survey text (SANDS), an item nonresponse detection approach based on a Bidirectional Transformer for Language Understanding model, fine-tuned using Simple Contrastive Sentence Embedding and targeted human coding, to categorize open-ended text data as valid or likely nonresponse. This approach is powerful in that it uses natural language processing as opposed to existing nonresponse detection approaches that have relied exclusively on rules or regular expressions or used bag-of-words approaches that tend to perform less well on short pieces of text, typos, or uncommon words, often prevalent in open-text survey data. This paper presents the development of SANDS and a quantitative evaluation of its performance and potential bias using open-text responses from a series of web probes as case studies. Overall, the SANDS model performed well in identifying a dataset of likely valid results to be used for quantitative or qualitative analysis, particularly on health-related data. Developed for generalizable use and accessible to others, the SANDS model can greatly improve the efficiency of identifying inadequate and irrelevant open-text responses, offering expanded opportunities for the use of open-text data to inform question design and improve survey data quality.

Does it Pay to Send Multiple Pre-Paid Incentives? Evidence from a Randomized Experiment

To encourage survey participation and improve sample representativeness, the Survey of Consumer Finances (SCF) offers an unconditional pre-paid monetary incentive and separate post-paid incentive upon survey completion. We conducted a pre-registered between-subject randomized control experiment within the 2022 SCF, with at least 1,200 households per experimental group, to examine whether changing the pre-paid incentive structure affects survey outcomes. We assess the effects of: (1) altering the total dollar value of the pre-paid incentive (“incentive effect”), (2) giving two identical pre-paid incentives holding the total dollar value fixed (“reminder effect”), and (3) offering multiple pre-paid incentives of different amounts holding the total dollar value fixed (“slope effect”) on survey response rates, interviewer burden, and data quality. Our evidence indicates that a single $15 pre-paid incentive increases response rates and maintains similar levels of interviewer burden and data quality, relative to a single $5 pre-paid incentive. Splitting the $15 into two pre-paid incentives of different amounts increases interviewer burden though lengthening time in the field without improving response rates, reducing the number of contact attempts needed for a response, or improving data quality, regardless of whether the first pre-paid is larger or smaller than the second.

How Late Reporters Effect Data Quality in Longitudinal Surveys – Experiences From the German Mobility Panel

Survey design always has a significant influence on the outcomes. Therefore, this paper investigates how follow-up campaigns affect survey outcomes and response rates in longitudinal surveys. Furthermore, it is assessed how late reports in the fall affect survey outcomes. The analyses are based on the unique data of the German Mobility Panel. Overall, this paper paints a broad picture of the methodological aspects and overlapping effects that should be considered before starting the fieldwork of longitudinal surveys. The results indicate that people who are reminded to participate positively influence the survey outcomes - even when the report is belated.

Checking Data Quality of Longitudinal Household Travel Survey Data

Ensuring data quality of household travel survey data is often tedious and, thus, time-consuming. To speed up the process of data-checking and to gain an in-depth understanding of the data, data visualization is a practical, fundamental tool. Since 1994, data visualization has been used in the German Mobility Panel (MOP) data-checking process. This paper presents two graphical visualization tools developed for the MOP. Both tools speed up the data checks and ensure high consistency in identifying erroneous data. This paper describes and discusses how the tools provide a continuous data quality assessment.

A United States-based patient-reported adult polyglucosan body disease registry: initial results.

Adult Polyglucosan Body Disease (APBD) is an ultra-rare, genetic neurodegenerative disorder caused by autosomal recessive mutations in the glycogen branching enzyme gene. Knowledge of the demographic and clinical characteristics of APBD patients and the natural history of the disease is lacking. We report here initial results from a patient-reported registry of APBD patients. (1) Maximize the quality of the APBD Registry survey data; (2) provide an initial report on APBD disease progression and natural history using these data; and (3) specify next steps in the process for testing potential new therapies. Data are from members of the APBD Research Foundation (New York), surveyed from 2014 by the Columbia APBD Patient/Family (CAP) Registry. Inclusion criteria are: disease onset at age 18+ and progressive clinical triad of peripheral neuropathy, spasticity, and neurogenic bladder. Genetic testing results were used when available. Respondents found to not have APBD in clinical records were excluded. All changes and exclusions were recorded in a database edit log. Results are reported in frequency tables, bar graphs, time plots, and heat maps. The 96 respondents meeting inclusion criteria were predominantly (96.8%) White, highly educated (89.3% at least some college education), and mostly (85.1%) of Ashkenazi Jewish descent. 57.1% had at least one parent born in the United States, with at least one grandparent from Europe (excluding Russia; 75.4%), the United States (42.1%), or Russia (33.3%). 37.2% reported a family history of APBD, and 33.3% had an affected sibling. Median APBD onset age was 51 [Interquartile range (IQR) 11], and median age of diagnosis 57 (IQR 10.5). The 75 reported prior misdiagnoses were mainly peripheral neuropathy (43, 60.6%) and spinal stenosis (11, 15.1%). Although from a demographically constricted survey, the results provide basic clinical information for future studies to develop treatments for APBD.

Data quality assessment through a preference model

Evaluating the quality of data is a problem of a multi-dimensional nature and quite frequently depends on the perspective of an expected use or final purpose of the data. Numerous works have explored the well-known specification of data quality dimensions in various application domains, without addressing the inter-dependencies and aggregation of quality attributes for decision support. In this work we therefore propose a context-dependent formal process to evaluate the quality of data which integrates a preference model from the field of Multi-Criteria Decision Aiding. The parameters of this preference model are determined through interviews with work-domain experts. We show the interest of the proposal on a case study related to the evaluation of the quality of hydrographical survey data.

Implications for Electronic Surveys in Inpatient Settings Based on Patient Survey Response Patterns: Cross-Sectional Study.

Surveys of hospitalized patients are important for research and learning about unobservable medical issues (eg, mental health, quality of life, and symptoms), but there has been little work examining survey data quality in this population whose capacity to respond to survey items may differ from the general population. The aim of this study is to determine what factors drive response rates, survey drop-offs, and missing data in surveys of hospitalized patients. Cross-sectional surveys were distributed on an inpatient tablet to patients in a large, midwestern US hospital. Three versions were tested: 1 with 174 items and 2 with 111 items; one 111-item version had missing item reminders that prompted participants when they did not answer items. Response rate, drop-off rate (abandoning survey before completion), and item missingness (skipping items) were examined to investigate data quality. Chi-square tests, Kaplan-Meyer survival curves, and distribution charts were used to compare data quality among survey versions. Response duration was computed for each version. Overall, 2981 patients responded. Response rate did not differ between the 174- and 111-item versions (81.7% vs 83%, P=.53). Drop-off was significantly reduced when the survey was shortened (65.7% vs 20.2% of participants dropped off, P<.001). Approximately one-quarter of participants dropped off by item 120, with over half dropping off by item 158. The percentage of participants with missing data decreased substantially when missing item reminders were added (77.2% vs 31.7% of participants, P<.001). The mean percentage of items with missing data was reduced in the shorter survey (40.7% vs 20.3% of items missing); with missing item reminders, the percentage of items with missing data was further reduced (20.3% vs 11.7% of items missing). Across versions, for the median participant, each item added 24.6 seconds to a survey's duration. Hospitalized patients may have a higher tolerance for longer surveys than the general population, but surveys given to hospitalized patients should have a maximum of 120 items to ensure high rates of completion. Missing item prompts should be used to reduce missing data. Future research should examine generalizability to nonhospitalized individuals.

Differential Electronic Survey Response: Does Survey Fatigue Affect Everyone Equally?

IntroductionSurvey fatigue, a phenomenon where respondents lose interest or lack motivation to complete surveys, can undermine rigorously designed studies. Research during the COVID-19 pandemic capitalized on electronic surveys for maximum distribution, but with lower response rates. Additionally, it is unclear how survey fatigue affects surgical education stakeholders. This study aims to determine how response rates to an electronic survey, as a proxy for survey fatigue, differ among medical students (MS), surgery residents, and surgery faculty. MethodsElectronic surveys evaluating the surgical clerkship educational environment were distributed to third year MS, residents, and faculty at three academic institutions. Two reminder emails were sent. Groups with low response rates (<30%) received additional prompting. Response rates were compared using a chi-square test. Demographics of all survey respondents were collected and discussed. Baseline characteristics of the MS class, residency program, and Department of Surgery faculty from one institution were gathered and compared to respondents. ResultsSurveys were sent to 283 third year MS, 190 surgery residents, and 374 surgical faculty. Response rates were 43%, 27%, and 20%, respectively (P < 0.0001). Male respondents, respondents of color, midlevel residents, and assistant professors had lower response rates compared to the baseline cohort. ConclusionsOur results demonstrate a statistically significant difference in survey response rates among MS, residents, and faculty, and have identified various targets for further investigation. Loss of interest in these groups should be further evaluated with a goal of decreasing survey fatigue, increasing survey response rates, and improving the quality of survey data collected.

THE EUROSDR RPAS BENCHMARK: OPEN DATASET DESCRIPTION AND SUMMARY OF KEY RESULTS

Abstract. In 2021 EuroSDR initiated a benchmark study with the aim to evaluate the geometric quality of real-world survey data generated from state-of-practice commercial Remotely Piloted Aircraft System (RPAS) photogrammetry (including DJI P4 RTK and DJI P1) and lidar (including DJI L1 and Riegl MiniVUX). The particular benchmark focus was on achievable data quality from real-world network configurations in the absence of ground control, on-the-fly Real Time Kinematic (RTK) corrections, and/or local GNSS base station information. Successive custom datasets were released to registered benchmark participants who submitted individual outputs that were independently evaluated against reference surveys. Without the inclusion of any supporting ground information, DJI P4 RTK and DJI P1 RPAS solutions were found to deliver m- and dm-level accuracies, respectively, in both plan and height. RTK solutions were found to provide cm-level precisions and accuracies, with some outliers. The introduction of ground control points resulted in similar planimetric accuracy to the RTK solutions, but with slight improvements in height. In terms of lidar datasets, the Riegl MiniVUX solution, using corrections from a local base station, was found to provide smaller discrepancies than the DJI L1 RTK solution, when independently compared against terrestrial laser scanning surveys. This paper provides various quality statistics and demonstrates multiple ways of assessing the geometric quality of RPAS data. The EuroSDR RPAS benchmark datasets are now openly available online in order to support and facilitate further investigation by the community.

A Mixture Model Approach to Assessing Measurement Error in Surveys Using Reinterviews

Abstract Researchers are often unsure about the quality of the data collected by third-party actors, such as survey firms. This may be because of the inability to measure data quality effectively at scale and the difficulty with communicating which observations may be the source of measurement error. Researchers rely on survey firms to provide them with estimates of data quality and to identify observations that are problematic, potentially because they have been falsified or poorly collected. To address these issues, I propose the QualMix model, a mixture modeling approach to deriving estimates of survey data quality in situations in which two sets of responses exist for all or certain subsets of respondents. I apply this model to the context of survey reinterviews, a common form of data quality assessment used to detect falsification and data collection problems during enumeration. Through simulation based on real-world data, I demonstrate that the model successfully identifies incorrect observations and recovers latent enumerator and survey data quality. I further demonstrate the model’s utility by applying it to reinterview data from a large survey fielded in Malawi, using it to identify significant variation in data quality across observations generated by different enumerators.

Are the attention checks embedded in delay discounting tasks a valid marker for data quality?

To ensure good quality delay discounting (DD) data in research recruiting via crowdsourcing platforms, including attention checks within DD tasks have become common. These attention checks are typically identical in format to the task questions but have one sensical answer (e.g., "Would you prefer $0 now or $100 in a month?"). However, the validity of these attention checks as a marker for DD or overall survey data quality has not been directly examined. To address this gap, using data from two studies (total N = 700), the validity of these DD attention checks was tested by assessing performance on other non-DD attention checks and data quality measures both specific to DD and overall survey data (e.g., providing nonsystematic DD data, responding inconsistently in questionnaires). We also tested whether failing the attention checks was associated with degree of discounting or other participant characteristics to screen for potential bias. While failing the DD attention checks was associated with a greater likelihood of nonsystematic DD data, their discriminability was inadequate, and failure was sometimes associated with individual differences (suggesting that data exclusion might introduce bias). Failing the DD attention checks was also not associated with failing other attention checks or data quality indicators. Overall, the DD attention checks do not appear to be an adequate indicator of data quality on their own, for either the DD task or surveys overall. Strategies to enhance the validity of DD attention checks and data cleaning procedures are suggested, which should be evaluated in future research. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Investigating respondents’ willingness to participate in video-based web surveys

Recent developments in communication technology and changes in people’s communication habits facilitate new data collection forms in web surveys. Technical devices, such as computers, tablets, and smartphones, enable researchers to rethink established communication forms and add a human touch to web surveys. Designing web surveys more human-like has the great potential to make communication between researchers and respondents more natural, which may result in higher survey satisfaction and data quality. Considering the existing survey literature, there are only a few studies investigating respondents’ willingness for new communication forms in web surveys. Hence, in the present study, we explore respondents’ willingness to take part in web surveys to have interviewers read questions via pre-recorded videos (question delivery) and in which respondents provide their answers orally via self-recorded videos (question answering). We included two willingness questions – one on question delivery via pre-recorded videos and one on question answering via self-recorded videos – in the non-probability SoSci panel in Germany. The results reveal that respondents’ willingness to have questions read by interviewers is higher than their willingness to self-record video answers. Believing that technology facilitates communication and perceiving the survey as being interesting increases willingness, whereas evaluating the survey topic as sensitive decreases willingness. Personality traits do not play a role when it comes to respondents’ willingness, except for extraversion.

Can EPDS and EPDS-3A be used to replace GAD-7 to screen the anxiety of pregnant women during pregnancy examination?

To examine the screening ability and cut-off scores of the Edinburgh Postnatal Depression Scale (EPDS) and EPDS-3A for pregnancy anxiety in pregnancy examination. This was a prospective repetitive measurement study. A follow-up survey was performed in Chinese pregnant women with the EPDS, the 7-item Generalized Anxiety Disorder (GAD-7) scale and a basic information questionnaire. A total of 609 women participated in all surveys at three trimesters, and all data were complete with no missing values. Receiver operating characteristic curves, Cohen's kappa and cross-tabulation were used to test the single and combined screening ability of EPDS and EPDS-3A for pregnancy anxiety. The areas under the curves of EPDS and EPDS-3A were close to or more than 0.90. EPDS scores greater than 14 in the first trimester and greater than 13 in the second and third trimesters could be used as the cut-off values. EPDS-3A score greater than 4 could be used as the cut-off value in each trimester. Adding the EPDS-3A to the EPDS would lead to correct identification of an additional 6.33%, 6.35%, and 7.25% of anxious women and misdiagnosis of an additional 6.60%, 2.56%, and 2.41% of normal women in each trimester, respectively. The EPDS and EPDS-3A can be used alone or in combination for initially screening of pregnancy anxiety. Under certain conditions, they can even be used to replace GAD-7 for anxiety screening to reduce the adverse influence of excessive screening on pregnant women, and improve the quality of survey data and efficiency of clinical services.

Interviewer-Observed Paradata in Mixed-Mode and Innovative Data Collection

In this research note, we address the potentials of using interviewer-observed paradata, typically collected during face-to-face-only interviews, in mixed-mode and innovative data collection methods that involve an interviewer at some stage (e.g., during the initial contact or during the interview). To this end, we first provide a systematic overview of the types and purposes of the interviewer-observed paradata most commonly collected in face-to-face interviews—contact form data, interviewer observations, and interviewer evaluations—using the methodology of evidence mapping. Based on selected studies, we illustrate the main purposes of interviewer-observed paradata we identified—including fieldwork management, propensity modeling, nonresponse bias analysis, substantive analysis, and survey data quality assessment. Based on this, we discuss the possible use of interviewer-observed paradata in mixed-mode and innovative data collection methods. We conclude with thoughts on new types of interviewer-observed paradata and the potential of combining paradata from different survey modes.