Retrofitting Non-diagnostic Reading Comprehension Assessment: Application of the G-DINA Model to a High Stakes Reading Comprehension Test

Abstract

ABSTRACTWith the advancement of Cognitive Diagnostic Assessment (CDA) and the pertinent statistical models, different domains of large-scale testing and assessment have been examined for the sake of reporting more diagnostic information. Applying the generalized deterministic input, noisy, “and” gate (G-DINA) model, the current study analyzed a high-stakes L2 reading comprehension test as an integral section of the PhD degree’s entrance exam in Iran. It aimed at examining the reading comprehension attributes underlying this high stakes test in an attempt to check the capability of CDA models, (in this case G-DINA), in providing diagnostic information for test developers and users. The items’ response data were analyzed in R, “GDINA” package, version 1.4.2. With data collected from multiple sources including the current literature on the sub-skills of reading comprehension, test specifications, test-takers’ think-aloud verbal protocols, and expert panel’s judgments, an initial Q-matrix, including five sub-skills, was developed and then validated. Data analysis, using the validated version of Q-matrix, showed the sub-skill (attribute) prevalence and its difficulty. Code-related sub-skills were the easiest and the most prevalent ones and the connecting/synthesizing sub-skills were the most difficult and the least prevalent ones for the test-takers. Skill mastery profile results verified the relationship among the sub-skills of reading skill and also the prominence of some of these sub-skills over others.