Attention checks and how to use them: Review and practical recommendations

Abstract

Web surveys dominate contemporary data collection in numerous disciplines within the broadly understood social sciences. However, this mode of data collection comes with additional challenges, particularly related to careless or insufficient effort responding (C/IER), which can distort study results and poses a direct threat to the validity. One of the recommended approaches to address this problem is using attention checks, which are additional tasks or items with objective answers that indicate attentive responding. Despite the potential benefits of attention checks, recent evidence suggests that they are still not sufficiently researched to justify their uncritical use in screening out inattentive participants. This article provides an abridged review of the attention checks literature, offers evidence-based practical recommendations, and highlights crucial gaps in research regarding attention checks. Evidence-based recommendations concerning the type, number, and placement of attention checks in a survey are presented. Generally, including more than one attention check in a survey is advisable, especially for longer surveys. Long instructed manipulation checks should be avoided, instead, covert attention checks, which are difficult for participants to identify, are recommended to reduce negative side effects such as noncompliance. In addition to attention checks, other criteria, such as item-level response time analysis, should be used in combination to identify inattentive participants. It is crucial to carefully analyse all data before making decisions about participant elimination. Ethical considerations related to the use of attention checks are also discussed, recognizing the importance of maintaining participant trust and understanding the potential impact on survey completion rates and data quality. Overall, attention checks hold certain promise as a tool to enhance data quality, but further research and a thoughtful implementation are necessary to maximise their effectiveness.