The Impact of Question and Respondent Characteristics on Comprehension and Mapping Difficulties

Abstract

Behavior coding is one technique researchers use to detect problems in survey questions, but it has been primarily explored as a practical tool rather than a source of insight into the theoretical understanding of the cognitive processes by which respondents answer survey questions. The latter is the focus of the current investigation. Using data from a large study in which face-to-face interviews were taped and extensive behavior coding was done, we tested whether sets of respondent behavior codes could be used to distinguish respondent difficulties with comprehension of the question from difficulties associated with mapping a judgment onto the response format provided, and whether characteristics of the survey questions and respondents could be used to predict when and for whom such difficulties would occur. Sets of behavior codes were identified that reflected comprehension and mapping difficulties, and these two types of difficulties were associated with different question and respondent characteristics. This evidence suggests that behavior coding shows promise as a tool for researchers studying the cognitive processes involved in answering survey questions.