Response Order Biases in Economic Surveys

Abstract

Survey data are ubiquitous in the economics literature, ranging from unemployment rate and CPI to surveys of professional forecasts and consumer finances. However, their potential biases are rarely discussed. By randomizing the order of responses to the questions in an economic survey, we document a pervasive response order bias: respondents tend to select answers at or near the top of the lists, leading to a systematic bias in survey results. This bias is smaller when respondents are more certain about their answers and disappears for “objective” questions (e.g., questions on demographics and recent experiences). Our evidence directly shows the bias in the levels of many survey-based variables and indirectly implies a bias in their changes, the latter likely correlated with the uncertainty in the economy. To assess bias magnitude, we examine two salient features at the time of our survey: the COVID pandemic and the approaching 2020 presidential election. We find that respondents’ expectations are shaped by their political leanings and personal experience during the pandemic: when forecasting stock returns, GDP growth, or COVID vaccine development, respondents are more pessimistic if they lean Democratic or personally know someone with COVID, but are more optimistic if they expect their preferred presidential candidate to win. The magnitude of these effects is comparable to the size of the response order bias. The implications of our evidence for financial services, public health policy, and political elections are discussed.