You Can't Compare If You Don't Prepare: Differential Item Functioning In Measures of Grit, STEM Self-Regulation, and Motivation

Abstract

Researchers developing questionnaire measures of personality, motivation, and self-regulation constructs related to students’ achievement and persistence in STEM or other fields rarely have examined whether the items on the measures used are functioning differently across groups, which is necessary for accurate group comparison. The present study assessed differential item functioning (DIF) in two frequently-used measures of such constructs, the Grit-S, the Motivated Strategies for Learning Questionnaire (MSLQ), and a new measure of long-term grit, using a sample of 1,146 ninth and tenth-grade students in the U.S.A. We also assessed gender and racial differences in the patterns of the relationships among the latent constructs, something rarely reported in the literature. Differences in such patterns indicate that the concurrent validity of the construct scores could vary across different groups. Results indicated that numerous items exhibited gender DIF on the Grit-S but not on the MSLQ or long-term grit measure. There were fewer DIF items across racial groups. Patterns of relations among the variables, in general, were weaker for females than males. There were fewer racial groups differences in the patterns of correlations. We discuss the implications of the DIF findings for future research on group comparisons.