First among equals: Log data indicates ability differences despite equal scores

Abstract

Analyzing test-taking behavior allows researchers to investigate the steps and actions resulting in the specific test outcome. The underlying assumption is that test-taking behavior is a valid indicator of the tested ability. The aim of this paper is to scrutinize this assumption in the context of complex problem solving (CPS) by analyzing individual differences in test-taking behavior and their relation to individual differences in established correlates of CPS ability. We investigated a sample of Finnish students who achieved the maximum score on five CPS tasks and worked on an additional intelligence measure. We logged the number of interactions with the CPS tasks and time-on-task. Students showed significant variance in both time-on-task (s2 = 260.09, p = .005) and the number of interactions (s2 = 0.381, p < .001) despite having no variance in their CPS test scores. Using structural equation modeling, both time-on-task (β = 0.17, p = .015) and the number of interactions (β = −0.17, p < .001) significantly predicted students' GPA, which has frequently shown to be associated with CPS ability. When adding intelligence – another established correlate of CPS – as predictor of GPA (β = 0.55, p < .001), the relation both between time-on-task (β = −0.09, p = .053) and the number of interactions (β = 0.09, p = .100) and students’ GPA was reduced to negligible effects. Taken together, our study supports the assumption of test-taking behavior actually representing differences in CPS ability – over and above test scores.