The importance of being earnest: How truth and evidence affect participants’ judgments

Abstract

Truth-value judgments are one of the most common measures in experimental semantics and pragmatics, yet there is no standardized way to elicit such judgments. Despite anecdotal remarks on how proper choice of prompts or response options could help disentangle pragmatic from semantic effects, little is known regarding the relation between parameters of the task and what it actually measures. We tested a range of prompts and two response options for their sensitivity to truth of the target sentence, prior evidence, and the interaction between these two factors. We found that participants attribute high value to true statements, even when they are not backed by evidence. Moreover, our results confirm that prompts vary wildly in their sensitivity to pragmatic factors, and should allow researchers to make an informed choice depending on what they want to test. There was no difference between the results generated by the response options, although the Likert scale required fewer participants and may therefore be preferable. In addition, we discuss some theoretical consequences of our results for pragmatics, philosophy of language, and social psychology.