Comments on Antony John Kunnan's "DIF in Native Language and Gender Groups in an ESL Placement Test". The Limits of Biased Item Analysis

Abstract

Differential item functioning (DIF) occurs when some special ability comes into play in answering an item on an examination other than the one that the item was intended to test. A typical example of this is when the examinees' native languages enable them to identify the correct answer to a question on a vocabulary test because there is a cognate in L1 with meaning similar to the word being tested. In this situation, it is common to say that the item is towards speakers of a particular language. A more neutral preferred term these days is DIF, but both terms will be used interchangeably here. Being able to detect DIF and to eliminate test items that exhibit it is important, especially when test results contribute to important decisions such as admission to school, licensing, job qualification, and promotion. We would like to know that the decisions we make are based on sound data that represent actual measurements of the quality that we are intending to measure. Antony John Kunnan's brief report in the TESOL Quarterly (Vol. 24, No. 4, Winter 1990) is admirable in what is intended, but the fact that statistical and subjective procedures for identifying DIF cannot be relied upon dilutes the significance of his conclusions and of the conclusions of nearly all studies which investigate the causes of DIF. There is nothing wrong with Kunnan's method of identifying biased items, which relies on selecting items which lie outside the 95% confidence interval. In fact, most statistical tests of significance