Impacts of Implementing an Automatic Advancement Feature in Mobile and Web Surveys

Abstract

As more surveys are administered online and taken on mobile devices, researchers need to continue to work toward defining best practices. A relatively new technical feature is an automatic advancement feature, which automatically scrolls the screen to the next item on a page of a survey instrument once a respondent selects an answer to the current question. This and other related forms of automatic advancement methods, such as the horizontal scrolling matrix, have so far produced mixed effects in terms of their impact on data quality, break off rates, and other survey measures. We present findings from an experiment conducted in 2017 using a survey of law school students in which we randomly assigned an automatic advancement feature to half of the respondents in a sample of nearly 40,000 students in the United States and Canada. We do not find significant differences in survey duration time, straight-lining, breakoff rates, or item nonresponse (for mobile users) between the two experimental groups, but desktop users without the automatic advancement feature had higher item nonresponse. More significantly, we find that respondents receiving the automatic advancement treatment on average changed about 50% fewer answers across the survey instrument than those who did not receive the automatic advancement design. We incorporate qualitative data to help us understand why this might have happened. Finally, respondents rated the automatic advancement feature as being easier to use and having a better visual design. This research extends the limited amount of work on automatic advancement features for surveys, helps move the field closer to best practices for survey design for mobile devices, and provides context and understanding about a particular respondent behavior related to data quality.