Device effects on survey response quality: A comparison of smartphone, tablet and PC responses on a cross sectional probability sample

Abstract

The increasing use of web surveys and different devices for survey completion calls for the examination of device effects on survey response quality. Whereas most existing studies are based on web panels, subgroups (e.g., students), or short questionnaires designed for device experiments, which compels participants to respond through specific devices, this study is based on two large, nationally representative cross-sectional samples (ISSP 2018 and ISSP 2019) in which the completion device was chosen by the respondent. Seven indicators of response quality are applied, which allows comparison among survey participants answering the questionnaire on a smartphone, tablet, or PC. The results are in line with previous findings: Respondents’ self-evaluated engagement in survey completion does not differ across devices, and only small, non-systematic differences between devices on satisficing indicators, such as the tendency to agree regardless of question content (acquiescence), non-substantive answers, selection of mid-point response options and primacy effects, and straightlining are identified. Controlling the associations between response device and response quality indicators for self-selection biases did not change the overall result.