The effect of assigning sample members to their preferred device on nonresponse and measurement in Web surveys

Abstract

Web surveys are completed on a range of different devices and even if Web surveys encourage sample members to respond with one specific device, some sample members do not follow the instructions and complete the Web survey with a different device. The prevalence of non-conforming respondents indicates that people have a device preference for Web survey participation. Thus, most Web surveys use a responsive questionnaire design that accommodates all devices. Web surveys using a responsive questionnaire design, also called mixed-device Web surveys, give sample members the opportunity to choose the device for participation at their own convenience. The assumption so far is that the opportunity to choose their preferred device for Web survey participation increases response rates compared to Web surveys that encourage one specific device for participation. Mixed-device Web surveys are a unique type of concurrent mixed-mode surveys and findings of previous research revealed that response rates of concurrent mixed-mode surveys were lower than response rates of surveys using a unimode design. These results can be explained by the paradox of choice. The opportunity to choose from a range of modes/devices may increase the complexity and burden of responding. Thus, the choice between modes/devices may dissuades sample members from responding. Accordingly, the allocation to one device may decrease burden of responding further even if Web surveys use a responsive questionnaire design. However, the assumption is that device instructions only help, if sample members are assigned to their preferred device. In particular, this thesis examines if the allocation to the preferred device can decrease nonresponse compared to being assigned to the non-preferred device. Furthermore, people are expected to prefer devices for Web survey participation that are less burdensome and more motivating. Thus, the task difficulty of answering questions is lower and the respondents’ degree of motivation is higher, if sample members respond with their preferred device instead of their non-preferred device. According to the satisficing framework, task difficulty increases the likelihood of satisficing and the degree of motivation decreases the likelihood of satisficing. Thus, respondents who complete the Web survey with their preferred device are less likely to shortcut the question-answer process resulting in higher data quality compared to respondents who answer the Web survey with their non-preferred device. The second aim of this thesis is to determine the effect of responding with the preferred device on measurement. The findings of this thesis revealed that respondents who were assigned to their preferred device were more likely to respond with the requested device than respondents who were assigned to their non-preferred device. However, being assigned to the preferred device did not affect unit nonresponse rates of sample members. Thus, higher conformance rates of sample members assigned to their preferred device were primary due to the decrease of non-conforming respondents. Findings of the effect of responding with the preferred device on data quality were inconclusive. Seven indicators of data quality (survey breakoff, item nonresponse, response time, survey focus, degree of differentiation, length of answers and primacy effects) were analyzed. Two indicators (survey breakoff and degree of differentiation) revealed that data quality of respondents who completed the Web survey with their preferred device was lower than data quality of respondents who answered the Web survey with their non-preferred device. However, findings on the respondents’ survey focus and their response time at question level indicated that data quality of respondents who completed the Web survey with their preferred device was higher than data quality of respondents who answered the Web survey with their non-preferred device. No effects were found for the remaining indicators. In conclusion, if sample members are assigned to one specific device for Web survey participation the sample members’ device preference should be considered, because non-conformance of device instructions can be reduced to a great extent resulting in higher conformance rates. Responding with the preferred device did not affect most indicators of data quality and effects on the remaining indicators of data quality were contradictory. Thus, overall responding with the preferred device should not affect data quality. The effect on conformance rates was stronger than the effect on data quality. Thus, in mixed-device Web surveys, it seems worth considering assigning sample members to their preferred device.