Investigating a failed novel test of socio-pragmatic knowledge

Abstract

This article is premised on the assumption that reporting failure is as important as reporting success in a research endeavour. The Social Variables Test that we developed was intended to provide a novel way of assessing second language learners’ explicit socio-pragmatic knowledge. To establish the validity of the test we conducted an evaluation using Kane's (2012) argument-based framework. The evaluation resulted in the recognition that the test had largely failed in its purpose. We report the evidence we collected from 24 native speakers and 66 Japanese university students majoring in English. We conclude that there was insufficient evidence to support the underlying rationale for the test and for most of the claims needed to establish a validity argument. We conclude with a number of suggestions for redeveloping the test.