Measuring Cognitive Processing Capabilities in Solving Mathematical Problems

Abstract

Understanding the sources of processing complexity in mathematical problem solving items is an important aspect of test validity. The sources of cognitive complexity may be either construct relevant or construct irrelevant. Studies have shown that the levels and sources of cognitive complexity predict item difficulty (e.g., S.E. Embretson, R.C. Daniel, Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychol. Sci. 50, 328–344 (2008)) and, further, that items can be selected or designed for difficulty in different sources of cognitive complexity. Although these results are relevant to the response processes aspect of construct validity, potential impact on the other aspects of validity was not addressed. That is, the modeling procedures did not include multidimensional measurement of individual differences in processing capabilities. In the current study, the multicomponent latent trait model for diagnosis (MLTM-D; S.E. Embretson, X. Yang, A multicomponent latent trait model for diagnosis. Psychometrika 78, 14–36 (2013)) was applied to measure cognitive processing capabilities in processing mathematical items. Individual differences in patterns of processing capabilities were significantly related to examinee background variables, thus indicating potential impact on the consequential aspect of validity. Implications of the findings for item design and test development are discussed.