Validation of a multiple-choice implicature test: insights from Chinese EFL learners’ cognitive processes

Abstract

Abstract In the past decades, implicature has been recognized as an indispensable topic in L2 pragmatics research. While various instruments have been used to test implicature knowledge, scant research has verified their construct validity of implicature items by examining test takers’ cognitive processes while responding to them. In response to this research gap, Roever, Carsten. 2005. Testing ESL pragmatics: Development and validation of a web-based assessment battery. Frankfurt am Main, Germany: Peter Lang implicature test was scrutinized. Twelve L2 test takers with high and low proficiency levels were asked to verbalize their thoughts retrospectively. Content analyses were employed to examine whether their cognitive processes were (ir)relevant to what the test intended to elicit and how well their processes aligned with the test results. The findings revealed that relevant processes were associated with idiosyncratic implicature questions, while irrelevant processes were related to formulaic implicature ones. The high-proficiency test takers generally reported engaging in relevant processes, whereas the low-proficiency test takers reported using for the most part irrelevant processes. Moreover, test takers’ cognitive processes aligned with the test results on the whole. In other words, when the respondents’ cognitive processes corresponded to those the test designed to elicit, the test results were normally correct. One the other hand, when the cognitive processes deviated from the expected ones, the test results were generally incorrect. However, fourteen misalignments were identified and classified as irrelevant conceptual blending, contextual judgment, insufficient effort responding, intuition, and wild guessing. These misalignments were seen as construct irrelevant variance and pose potential threats to the construct validity.