How do we go about investigating test fairness?

Abstract

Previous test fairness frameworks have greatly expanded the scope of fairness, but do not provide a means to fully integrate fairness investigations and set priorities. This article proposes an approach to guide practitioners on fairness research and practices. This approach treats fairness as an aspect of validity and conceptualizes it as comparable validity for all relevant groups. Anything that weakens fairness compromises the validity of a test. This conceptualization expands the scope and enriches the interpretations of fairness by drawing on well-defined validity theories while enhancing the meaning of validity by integrating fairness in a principled way. TOEFL® iBTTM is then used to illustrate how a fairness argument may be established and supported in a validity argument. The fairness argument consists of a series of rebuttals to the validity argument that would compromise the comparability of score-based interpretations and uses for relevant groups, and it provides a logical mechanism for identifying critical research areas and setting research priorities. This approach will hopefully inspire more investigations motivated by and built on a central fairness argument. It may also foster a deeper understanding and expanded explorations of actions based on test results and social consequences, as impartiality and justice of actions and comparability of test consequences are at the core of fairness.