Modeling Item Difficulty in a Dynamic Test

Abstract

The Concept Formation subtest of the Woodcock Johnson Tests of Cognitive Abilities represents a dynamic test due to continual provision of feedback from examiner to examinee. Yet, the original scoring protocol for the test largely ignores this dynamic structure. The current analysis applies a dynamic adaptation of an explanatory item response theory model to evaluate the impact of feedback on item difficulty. Additionally, several item features (rule type, number of target shapes) are considered in the item difficulty model. Results demonstrated that all forms of feedback significantly reduced item difficulty, with the exception of corrective feedback that could not be directly applied to the next item in the series. More complex and compound rule types also significantly predicted item difficulty, as did increasing the number of shapes, thereby supporting the response process aspect of validity. Implications for continued use of the Concept Formation subtest for educational programming decisions are discussed.