Testing Expert-Based and Student-Based Cognitive Models: An Application of the Attribute Hierarchy Method and Hierarchy Consistency Index

Abstract

The objective of the present investigation was to compare the adequacy of two cognitive models for predicting examinee performance on a sample of algebra I and II items from the March 2005 administration of the SAT™. The two models included one generated from verbal reports provided by 21 examinees as they solved the SAT™ items, and the other generated from the judgment of a content expert. Using the attribute hierarchy method (Leighton, Gierl, & Hunka, 2004) and the hierarchy consistency index (Cui, Leighton, Gierl, & Hunka, 2006), the predictive adequacy of the two cognitive models was tested with three samples of varying size. Results indicated that the student-based model predicted the test performance of examinees of moderately high ability with excellent accuracy but not the test performance of examinees of moderate (or average) ability. In contrast, the expert-based model was more consistent, predicting the test performance of both moderately high ability examinees and moderate ability examinees with reasonable accuracy. Expert-based models may be more predictive of the general response processes used by diverse groups of examinees, whereas student-based models may be preferred when making inferences about examinees at specific ability levels.