The role of rapid guessing and test‐taking persistence in modelling test‐taking engagement

Abstract

AbstractBackgroundItem response times in computerized assessments are frequently used to identify rapid guessing behaviour as a manifestation of response disengagement. However, non‐rapid responses (i.e., with longer response times) are not necessarily engaged, which means that response‐time‐based procedures could overlook disengaged responses. Therefore, the identification of disengaged responses could be improved by considering additional indicators of disengagement. We investigated the extent to which decreases in individuals' item solution probabilities over the course of a test reflect disengaged response behaviour.ObjectivesTo disentangle different types of possibly disengaged responses and better understand non‐effortful test‐taking behaviour, we augmented responses‐time‐based procedures for identifying rapid guessing with strategies for detecting disengaged responses on the basis of performance declines in non‐rapid responses.MethodsWe combined item response theory (IRT) models for rapid guessing and test‐taking persistence to examine the capability of response times and item positions to capture response disengagement. We used a computerized assessment in which science items were randomly distributed across positions for each student. This allowed us to estimate individual differences in test‐taking persistence (i.e., the duration for which the initial level of performance is maintained) while accounting for rapid responses.Results and ConclusionsResponse times did not fully explain disengagement; item responses reflected test‐taking persistence even when rapid responses were accounted for. This interpretation was supported by a strong correlation of test‐taking persistence with decreases in self‐reported test‐taking effort. Furthermore, our results suggest that IRT models for test‐taking persistence can effectively account for the undesirable impact of low test‐taking effort even when response times are unavailable.