Gender fairness within the Force Concept Inventory

Adrienne Traxler,John Stewart,Rebecca Lindell,Rachel Henderson,Alexis Papak,Gay Stewart

doi:10.1103/physrevphyseducres.14.010103

Abstract

Research on the test structure of the Force Concept Inventory (FCI) has largely ignored gender, and research on FCI gender effects (often reported as “gender gaps”) has seldom interrogated the structure of the test. These rarely crossed streams of research leave open the possibility that the FCI may not be structurally valid across genders, particularly since many reported results come from calculus-based courses where 75% or more of the students are men. We examine the FCI considering both psychometrics and gender disaggregation (while acknowledging this as a binary simplification), and find several problematic questions whose removal decreases the apparent gender gap. We analyze three samples (total Npre=5391, Npost=5769) looking for gender asymmetries using classical test theory, item response theory, and differential item functioning. The combination of these methods highlights six items that appear substantially unfair to women and two items biased in favor of women. No single physical concept or prior experience unifies these questions, but they are broadly consistent with problematic items identified in previous research. Removing all significantly gender-unfair items halves the gender gap in the main sample in this study. We recommend that instructors using the FCI report the reduced-instrument score as well as the 30-item score, and that credit or other benefits to students not be assigned using the biased items.Received 1 September 2017DOI:https://doi.org/10.1103/PhysRevPhysEducRes.14.010103Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAssessmentDiversity & inclusionResearch methodologyPhysics Education Research

Highlights

The Force Concept Inventory (FCI) [1] has been studied using tools such as factor analysis [2,3], item response theory [4,5], and network analysis [6]
We adopt the classical test theory (CTT) and item response theory (IRT) measures used by Jorion et al We extend the framework to include item fairness analysis using differential item functioning as discussed below
The FCI was revised after its initial publication; this work uses the revised instrument published with Mazur [59] which is available at PhysPort [60]

Summary

Introduction

The Force Concept Inventory (FCI) [1] has been studied using tools such as factor analysis [2,3], item response theory [4,5], and network analysis [6] Though these investigations have probed the structure and validity of the test, they have primarily treated student data as a single undifferentiated sample and have not studied gender effects. A largely separate branch of research has explored gender differences in scores on the FCI and other conceptual inventories [7,8] These studies have documented a ubiquitous advantage for men on pretest questions, which often persists to the post-test.

Methods

Results

Discussion

Conclusion