Using Algebra Word Problems to Assess Quantitative Ability: Attributes, Strategies, and Errors

Abstract

Changing goals in mathematics education have encouraged more open-ended problem solving in assessment. However, the use of these less constrained approaches has been limited by a lack of demonstrated relations between the underlying cognitive models and measurement consequences. In order to begin to characterize the cognitive basis for this emerging approach to measurement in a small domain-algebra word problemsdetailed analyses of solutions to 20 problems that had appeared on the Graduate Record Examination General Test were collected from 51 undergraduates. Problems were characterized in terms of their major attributes, and solutions were described by students' strategies and errors. Regression analyses indicated that models including attributes such as the need to apply algebraic concepts, problem complexity, and problem content could account for 37% to 62% of the variance in problem difficulty. Protocol analyses identified four major solution strategies-equation formulation, ratio setup, simulation, and other (unsystematic) approaches-as well as a number of collateral strategies, including the use of pictures, formulae, and verbal descriptions. Higher achieving students used more equation strategies, more collateral strategies, and fewer unsystematic approaches than lower achieving students. Student errors tended to be idiosyncratic but could be classified into six principal categories that were used to identify sources of performance. Overall, the results support the notion that constructed responses capture strategy formulation and high-level planning, as do more traditional measures of quantitative reasoning. At the same time, constructed responses are more sensitive to individual problem characteristics and procedural errors that may be helpful in instruction but are a potential source of bias in assessment. A preliminary theoretical framework for describing performance on algebra word problems is proposed, and its usefulness for instruction and for more systematic design of tests is discussed.