Assessing Differential Item Functioning and Differential Test Functioning in an Academic Motivation Scale using Item Response Theory methods

Abstract

Social work researchers and practitioners who use measurement instruments to make data-informed decisions need to ensure those decisions are based on items and scales that are free from possible bias or undesirable differential functioning. In this study, we provide an example of how a set of Item Response Theory (IRT) statistical methods and tools can be used by social work measurement researchers to assess differential item (DIF) and scale (DTF) functioning. For the example, we explored the possible race, gender, and family composition differential functioning of a scale—the Academic Motivation Scale (AMS)—developed for use by school social workers. The data used in this analysis were collected from 3,221 seventh grade students in multiple school districts in a large urban mid-western U.S. county. We used IRT methods and a multiple-step framework to assess possible race, gender, and family composition DIF/DTF. Results indicated there was minimal race, gender, or family composition differential functioning at both the item and scale level. While the AMS is recommended for use by school social workers, further research is needed to examine possible DIF/DTF by other factors such as parent involvement and family background, gender identity, sexual preference, and cultural attributes and ethnic factors.