The impact of visual design and response formats on data quality in a web survey of MOOC students

Abstract

The objective of this paper is to gain more evidence regarding how the design of the rating scales and open-ended questions influence data quality in Web surveys of MOOC (Massive Open Online Course) students. We present the results of four full-factorial randomized experiments that investigate the impact of the following factors: 1) order of response options; 2) user interface for rating questions 3) layout of question's options; and 4) size of answer boxes in open-ended questions. We found that responses to scalar questions with ascending (from negative to positive) or descending (from positive to negative) order of response options do not differ substantially. The use of the radio button format allows a reduction in the percentage of respondents who choose the “Don't know” option and makes responding to questions less challenging in comparison with slider and text box interfaces. There are no significant differences in the answers of respondents who completed questionnaires with a vertical or horizontal orientation of the questions' options. In addition, respondents who answer the questions with larger answer boxes are more likely to write longer comments.