Assessing general attentiveness to online panel surveys: the use of instructional manipulation checks

Abstract

ABSTRACT In online surveys, the use of manipulation checks aimed at measuring respondents’ attentiveness has become common. More than being measures of attentiveness pertaining to a specific survey, instructional manipulation checks (IMC) could work as generic measures of the quality of the answers a person gives when completing a questionnaire. By using several waves of the ITANES (Italian National Election Study) – University of Milan online panel survey, the article shows that the outcome of an IMC predicts the quality of the answers a respondent gives in previous and subsequent waves of the panel. Moreover, through a survey experiment that randomizes the length of an IMC we show that, overall, the answers’ quality of ‘attentive’ respondents assigned to different IMCs do not substantially vary. Findings also show that IMCs are reliable measures, as the outcome of two IMCs placed in two consecutive waves proved to be highly associated.