Modeling the interaction between resilience and ability in assessments with allowances for multiple attempts

Abstract

Many computer-based assessments, in particular those administered in learning settings, provide immediate item-level feedback and permit respondents to retry questions following incorrect responses. Doing so supports learning and facilitates the measurement of partial mastery. However, persistence through attempts may introduce new factors, such as resilience, affecting a student's test score. The current study proposes a tree-based approach to jointly model item responses and reattempt decisions in multiple-attempt assessments, allowing for the separate measurement of latent ability and reattempt propensity, or resilience. The proposed model is implemented on two datasets collected from online homework assignments, one from a university for-credit course and the other from a massive open online course. The results shed light on the relationship between resilience and ability both at the individual level and at the item level, where threshold parameters can be interpreted as difficulty and repeatability. Importantly, resilience was found to affect certain commonly used performance scores, such as eventual correctness proportion, posing a validity threat for summative assessment use-cases. The tree-based model not only resolves the problem but also opens the door to ecologically valid measures of resilience that are not reliant on self-reports.