Respondent Screening and Revealed Preference Axioms: Testing Quarantining Methods for Enhanced Data Quality in Web Panel Surveys

Abstract

Online survey methods and online access panels are commonly used in both academic and private industry research. However, many industry reviews indicate the presence of fraudulent and inattentive online panel participants who pose a threat to data validity, with estimates of 8–25 percent of the sample being impacted. Although these challenges are not unique to online surveying, the nature of online panels may offer possibilities to improve data quality through real-time filters, rather than in data analysis. In this study, we retain all panel respondents passing and failing common web-survey quarantining methods to directly analyze filtering results on data quality. Through a choice experiment, we build a mathematical model to measure respondent irrationality through the classic strong axiom of revealed preference (SARP), which serves as the primary data-quality metric to test the efficacy of quarantining methods. We show that inattentive respondents failing “trap” or “red herring” questions, as well as fraudulent respondents heavily selecting low-probability screening questions, have significantly lower data quality compared to passing respondents. Removing failing respondents significantly improves data quality when filters are strategically executed. We also simulate the data-quality benefits of real-time filtering techniques during web-data collection. We conclude by advocating the use of low-probability filters as qualifiers, as well as strategic placement of trap questions just before the most crucial sections of a survey