Motivation Filtering on a Multi-Institution Assessment of General College Outcomes

Abstract

Possible lack of motivation is a perpetual concern when tests have no stakes attached to performance. Specifically, the validity of test score interpretations may be compromised when examinees are unmotivated to exert their best efforts. Motivation filtering, a procedure that filters out apparently unmotivated examinees, was applied to the Collegiate Learning Assessment (CLA), an open-ended test of general college outcomes. Filtering based on self-reported effort was compared to a new approach based on CLA performance relative to expected. Both approaches had expected positive effects on aggregate CLA scores, but the new approach achieved these gains while filtering out fewer students and maintaining sample representativeness. Motivation filtering had negligible effects on the relative standings of schools based on their average CLA scores, although the impact of filtering was slightly greater on lower-performing schools. Motivation filtering increased longitudinal estimates of growth on the CLA because of apparently lower motivation among second-semester sophomores compared to entering freshmen.