Item-by-item Versus Matrix Questions

Mingnan Liu,Alexandru Cernat

doi:10.1177/0894439316674459

Mingnan Liu, Alexandru Cernat

Open Access

https://doi.org/10.1177/0894439316674459

Copy DOI

Abstract

While the choice of matrix versus item-by-item questions has received considerable attention in the literature, it is still unclear in what situation one is better than the other. Building upon the previous findings, this study expands this line of research by examining whether the difference between the two question types is moderated by the number of response options. Through a web survey experiment, this study compares matrix and item-by-item questions with 2, 3, 4, 5, 7, 9, and 11 response options. Additionally, we also investigate the impact of the device used to complete the survey on data quality. The results show that straight lining and response time are similar between the two question types across all response lengths, but item nonresponse tends to be higher for matrix than item-by-item question, especially among mobile respondents. Also measurement models reveal measurement equivalence between the two question types when there are fewer than seven response options. For matrices with 9 or 11 response options, analyses reveal substantial differences compared to item-by-item questions.

Highlights

While the choice of matrix versus item-by-item questions has received considerable attention in the literature, it is still unclear in what situation one is better than the other
The pattern of missing data became somewhat mixed: Matrix questions produced more missing data for four, five, seven, and nine-category questions, while for the other response option formats, the missing rates were higher for Item-by-item questions
Matrix questions consistently had higher missing data than item-by-item question, and the difference was significant for questions with three categories (p 1⁄4 .03), five categories (p 1⁄4 .01), and seven categories (p < .01)

Summary

Introduction

While the choice of matrix versus item-by-item questions has received considerable attention in the literature, it is still unclear in what situation one is better than the other. When asking multiple questions using the same set of response options, researchers often have two choices: either format them as item-byitem questions (asking them item by item) or group them into a matrix ( called grid) format and present them together While the latter is more succinct, it can impact data quality (e.g., see Couper, Traugott, & Lamias, 2001). In this context, survey researchers have to make informed decisions regarding which approach to use in different situations. Survey researchers have to make informed decisions regarding which approach to use in different situations They have to counterbalance the potential time savings and ease of application with the quality of the answers and the satisfaction of respondents.

Objectives

Methods

Results

Conclusion