The Use of Cognitive Interviewing Methods to Evaluate Mode Effects in Survey Questions

Abstract

This article focuses on a novel use of cognitive interviewing as a follow-up rather than as a pretesting methodology to explore mode effects. Respondent from a quantitative mixed-mode experiment took part in cognitive interviews where questions were administered face to face, by telephone, and by web, followed by a retrospective think-aloud. The focus of the think-aloud (used in combination with some prescripted probes) was not on what respondents understood by certain words or answer categories but on how they had answered questions in the different data collection modes. This article discusses the methods used in these mixed-mode cognitive interviews (which involved an element of mode mimicking) and how the interviews differed from standard cognitive interviewing for question testing. The benefits and limitations of our approach are discussed as well as lessons learned for using cognitive interviewing to explore patterns observed in survey data.