Horizontal or Vertical? The Effects of Visual Orientation of Categorical Response Options on Survey Responses in Web Surveys

Abstract

Prior research has studied the impact of horizontal versus vertical orientation of rating scales on survey responses, but not the effects of orientation of categorical options. The present study attempts to fill this gap by investigating the orientation effects of categorical options on response burden and response quality and by studying how the effects of orientation of response categories interact with cognition capacity. The data are drawn from an experiment embedded in the 2013 Internet Survey of the Health and Retirement Study (HRS), in which respondents were randomly assigned to a question with either horizontally or vertically aligned response categories. Response burden is measured by response time, and response quality is measured by reliability, primacy effect, and item missing on the question. Horizontal orientation is found to be more burdensome than vertical orientation on the web for HRS respondents, but there is no significant difference between the two orientations on primacy effect and item missing, and the two orientations are equally reliable. The effects of cognition capacity are found to interact with orientation on both response burden and response quality on the web. Compared to vertical orientation, horizontal orientation leads to longer response time but less primacy effect for people with lower cognition capacity versus people with higher cognition capacity. However, the effects of the orientation and cognition capacity do not interact on item missing. Vertical orientation for categorical response options in web surveys is recommended in general, but more caution should be exercised if respondents are elderly and primacy effect is a main concern.