Testing the test of advanced EFL reading comprehension: To what extent does the difficulty of a multiple-choice comprehension test reflect the difficulty of the text?

Abstract

This paper examines the faulty assumption that the difficulty of a multiple-choice comprehension test depends mainly on the difficulty of the text. Instead, the basic assumption stated in this paper is that the difficulty of a multiple-choice test depends on two components, a text and questions, with results varying according to the level of English proficiency of the students tested. Two multiple-choice comprehension tests on the advanced level were administered: one based on a relatively easy text, the other on a more difficult one. The difficulty of the texts was measured by teacher evaluation and by discourse analysis. For each text, nine multiple-choice questions are examined. Test 1 was administered to 178 candidates, and Test 2 to 132. The two tests show no significant differences in scores, despite the fact that one of the texts is clearly easier. The results indicate that teachers/test-constructors cannot assume that a text of a particular level of difficulty will automatically yield questions reflecting the level of the text.