Test dimensionality: IRT analysis of an ESP listening test1

Abstract

The use of Item Response Theory (IRT) in the examination of the qualities of language tests is a comparatively recent development, and one that has proved controversial. Oller (1983) contains no reference to IRT in a wide-ranging col lection. By contrast, IRT, particularly involving the family of models deriving from the work of Rasch (1960 [1980]), has featured in a number of studies since the early 1980s. Although Rasch IRT has sometimes been discussed primarily as a tool for improved investigation of the reliability of tests (Skehan, 1989), its potential for investigating aspects of the validity of language tests has also been demonstrated (McNamara, 1990a). The application of Rasch IRT in this latter role has in some cases met with objections based on what are claimed to be its unsatisfactory theoretical assumptions, in particular the so-called 'unidimensionality' assumption (Hamp-Lyons, 1989). In this paper, this issue will be discussed in the context of the analysis of data from an ESP listening test for health professionals, part of a larger test, the Occupational English Test (OET), developed on behalf of the Australian Government (McNamara, 1989b). The paper is in three sections. First, there is a brief description of the listening subtest of the OET. Second, the appropriateness of the use of Rasch IRT in language testing research is discussed. Third, the use of Rasch IRT in the validation of the listening subtest of the OET is reported. In this part of the paper, the issue of unidimensionality is considered in the context of analysis of data from the two parts of this test.