Cognitive misers on the web: An online-experiment of incentives, cheating, and cognitive reflection

Abstract

The Cognitive Reflection Test (CRT) has become one of the most widely used psychological tests in decision research. It also acquired considerable fame in the news, social media, and classrooms around the globe. Given its popularity, it comes as no surprise that correct answers on the CRT questions can be easily found online. How does the availability of information about the test influence its validity as a measure of cognitive reflection, particularly, if it is deployed in an online study? We present evidence that participants searched for answers on the internet. We compared two CRT versions. The crucial difference was that correct responses could be easily found online for the standard CRT, but not its modified variant. In addition, we examined if performance-based payment affected test performance and participants’ propensity to cheat. The latter was operationalized as the online search for correct answers on the CRT questions. Our results suggested that cheating occurred during the experiment, and that doing so improved performance if correct responses were easily found on the web. When the CRT items were re-phrased so that they could not be found online, performance was not improved by incentives. Hence, incentives boosted cheating rather than cognitive reflection.