Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis

Abstract

A critical issue in cognitive diagnostic assessment (CDA) lies in the dearth of research in developing diagnostic tests for cognitive diagnostic purposes. Most research thus far has been mainly carried out on large-scale tests, e.g., Test of English as a Foreign Language (TOEFL), Michigan English Language Assessment Battery (MELAB), International English Language Testing System (IELTS), etc. In particular, CDA of formative language assessment that aims to inform instruction and to discover strengths and weaknesses of students to guide instruction has not been conducted in a foreign (i.e., second) language-learning context. This study explored how developing a reading comprehension test based on a cognitive framework could be used for such diagnostic purposes. To achieve this, initially, a list of 9 reading attributes was prepared by experts based on the literature, and then the targeted attributes were used to construct a 20-item reading comprehension test. Second, a tentative “Q-matrix” that specified the relationships between test items and the target attributes required by each item was developed. Third, the test was administered to seven language-testing experts who were asked to identify which of the 9 attributes were required by each item of the test. Fourth, on the basis of the overall agreement of the experts’ judgments concerning the choices of attributes, review of the literature and results of student think-aloud protocols, the tentative Q-Matrix was refined and used for statistical analyses. Finally, the test was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. To examine the CDA of the test, the Reparameterized Unified Model (RUM) (also known as the Fusion Model), a type of cognitive diagnostic measurement model (CDM), was used for further refining the Q-Matrix for future data analyses and, most importantly, for diagnosing the participants' strengths and weaknesses. Data analysis results confirmed that the nine proposed reading attributes are involved in the reading comprehension test items. Such diagnostic information could be helpful for teachers and practitioners to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas that need to be emphasized in class in order to plan for better L2 reading instruction. Further, such information could inform individualized student instruction and produce improved diagnostic tests for future use.