Assessing Measurement Error in Hypothetical Questions

Abstract

The hazard of asking survey participants hypothetical questions is that most often, responses are only an inferential vehicle for understanding preference and may not reflect the truth. Often it is necessary to make a reasoned assumption about the alignment of an individual's attitudes and behaviors. In this paper, we discuss results from a Survey of Doctorate Recipients (SDR) pilot study experiment that found significant response bias to hypothetical questions even when the hypothetical context was unambiguous and included salient properties which were familiar to respondents from prior experience. The experiment asked Treatment group participants to report their reaction to slight alterations in the SDR instrument; meanwhile, those in the Control group were asked hypothetically how they *would react* to the protocol if they had experienced the same changes. We identify the presence of response bias across 10 respondent experience questions using both categorical and quantitative methods. Our results show that the Control group consistently exhibited less preference for the hypothetical protocol than the Treatment group. We believe these differences are representative of the measurement error resulting from the poor correspondence between the Control group's hypothetical attitudes and the Treatment group's reported experience. This work shows that hypothetical bias is a concern in opinion research even when hypothetical conditions may be clearly understood by respondents and similar to their previous experiences.