Survey Questions About the Frequency of Vaguely Defined Events: The Effects of Response Alternative

Abstract

Experimental research has shown that the choice of response alternatives can influence responses to questions about the frequency of vaguely defined target events (e.g., feeling annoyed); the set of response alternatives is treated as information that contributes to the interpretation of the question. In a series of split ballot experiments we investigate whether such effects occur in a large-scale survey context for sets of response alternatives that might be used interchangeably by survey researchers. The predicted response shifts were found in our field experiments. Those presented with response alternatives discriminating at low frequencies reported fewer of the target episodes than those presented with higher-frequency response alternatives. However, the size of the observed shifts varied from zero to 13 percent and depended on a number of characteristics of the survey, for example, the presence of “priming” questions and the orientation of the response scales. Response alternatives are not neutral and therefore must be carefully considered when constructing a survey question.