Web survey experiments on matrix questions

Abstract

Matrix or grid questions are a very common question type in web surveys although previous studies have demonstrated several measurement errors associated with this type of question. This study reports findings from four web survey experiments on testing a new matrix, called a menu matrix. A menu matrix groups several matrix questions into one large question and presents the response options using dropdown menus. The four experiments collectively show that the item nonresponse rate for the menu matrix is higher than for the regular matrix and multiple-choice questions. Also, respondents found the menu matrix to be more difficult than the regular matrix and multiple-choice questions. The time to complete is similar across the three question types. Also, the regular matrix and multiple-choice showed very similar missing data rates. The evidence speaks against the use of complex survey questions, such as the menu matrix, since they reduce data quality and the respondents' survey taking experience.