Using sequence mining to study students’ calculator use, problem solving, and mathematics achievement in the National Assessment of Educational Progress (NAEP)

Abstract

Using appropriate tools strategically to aid in problem solving is a crucial skill identified in K-12 mathematics curriculum standards. As more assessments transition from paper-and-pencil to digital formats, a variety of interactive tools have been made available to test takers in digital testing platforms. Using onscreen calculators as an example, this study illustrates how process data obtained from student interactions with a digitally-based large-scale assessment can be leveraged to explore how and how well test takers use interactive tools and unveil their mathematical problem-solving processes and strategies. Specifically, sequence mining techniques using the longest common subsequence were applied on process data collected from a nationally representative sample who took the National Assessment of Educational Progress (NAEP) mathematics assessment to examine patterns of eighth-grade students’ calculator-use behaviors and the content of calculator input across a series of items. Sequences of keystrokes executed on the onscreen calculator by test takers were compared to reference sequences identified by content experts as proficient and efficient use to infer how well and how consistently the calculator was used. Results indicated that calculator-use behaviors and content differed by item characteristics. Students were more likely to use calculators on calculation-demanding items that involve intensive and complex computations than on items that involve simple or no computation. Using the calculator on more calculation-demanding items and using it in a manner that is more efficient and more similar to reference sequences on these items were related to higher mathematical proficiency. Findings have implications for assessment design and can be used in educational practices to provide educators with actionable process-related information on tool use and problem solving.