Designing Cognitive Complexity in Mathematical Problem-Solving Items

Abstract

Cognitive complexity level is important for measuring both aptitude and achievement in large-scale testing. Tests for standards-based assessment of mathematics, for example, often include cognitive complexity level in the test blueprint. However, little research exists on how mathematics items can be designed to vary in cognitive complexity level. In fact, determining the cognitive complexity level of items is usually based on correspondence to definitions rather than on empirically and theoretically justifiable variables that can predict item difficulty. In the current study, mathematical problem-solving items were designed for varying cognitive complexity levels based on a cognitive model of item processing. Structural variants of item models were designed to vary on two aspects of the cognitive model, the equation source and the number of subgoals. Participants were randomly assigned to test forms that contained different structural variants of the item models. Results from the linear logistic test model, the two-parameter-logistic—constrained model, and a corresponding linear mixed modeling procedure indicated that the item design variables affected both item difficulty and response time. Implications of the results for using structural variants in item generation and for the plausibility of the hypothesized cognitive model are discussed.