Operationalizing uncertainty in language testing: an argument in favour of content validity

Abstract

It is argued that five variables associated with language proficiency are all subject to unreliability. They are: the native speaker, the cut-off, the criterion score, the test and the language. The native speaker exists only in an idealized form. The cut-off has at best a local definition and at worst is hugely subjective. The criterion used for validation is itself often less reliable than its predictor. Tests are difficult, perhaps impossible to replicate and therefore to generalise from. Language itself balances the general and the particular (this is the 'language': 'a language' distinction). In all these areas the object of description is elusive and fugitive. Language testing has these problems to an even greater extent because measurement is by its nature prone to error. The elusiveness of precision in language testing is discussed in relation to three major issues of linguistic analysis: universality, combining power and the incorporation of extra language data, and results from a recent validation study of the English Language Testing Service (ELTS) Test are quoted to exemplify that relation ship. It is suggested that the test has attempted overspecification and concluded that language testing should accept the centrality of uncertainty and make it explicit.