Comparing Examinee-Based and Response-Based Motivation Filtering Methods in Remote Low-Stakes Testing

Abstract

ABSTRACT Large-scale educational assessments are sometimes considered low-stakes, increasing the possibility of confounding true performance level with low motivation. These concerns are amplified in remote testing conditions. To remove the effects of low effort levels in responses observed in remote low-stakes testing, several motivation filtering methods can be used to purify the data. We estimated scores from assessment data collected remotely in Spring 2021 six ways, applying examinee-based filtering methods (filtering examinees based on total time) and response-based filtering methods (filtering responses using the effort-moderated IRT model), varying the thresholds selected to separate effortful and non-effortful responses. We compared the 2021 scores (estimated six ways) to those obtained in previous cohorts tested in person. The results support the use of motivation filtering regardless of which method is used. Practical implications regarding method and threshold selection are discussed.