A Reaction to “Moderating Possibly Irrelevant Multiple Mean Score Differences on a Test of Mathematical Reasoning”

Abstract

Stocking, Jirele, Lewis, and Swanson (1998) provide an intriguing application of automated test assembly (ATA) to the problem of reducing potential performance differentials among population subgroups. ATA offers test developers the advantage of using computers to help construct test forms that have consistent features and that exhibit desired statistical properties. That is, ATA uses optimization algorithms or heuristics to select items from a pool that meet some desired statistical requirements such as a targeted mean difficulty, a particular level of reliability, or a minimum amount of precision at various regions of the score scale. Additionally, the item selection software usually needs to match content specifications and other categorical or quantitative features deemed important by the test developers (see, e.g., van der Linden & Boekkooi-Timminga, 1989; Stocking, Swanson, & Pearlman, 1993). From a purely technical perspective, reducing impact by moderating differences in observed item proportion correct scores (i.e., p-values) is not complicated. It simply involves the introduction of additional statistical criteria or a new set of constraints to be considered as part of a particular ATA item selection optimization model. Stocking et al. (1998) demonstrate their ATA application via the weighted deviations model (Swanson & Stocking, 1993). They further provide some instruction about how to incorporate these types of constraints into the ATA problem, including alternatives for weighting those criteria in the item selection heuristic. There are, however, potential pitfalls inherent in creating an ATA machine to unconditionally eliminate or reduce impact. The machine understands neither the cause of the impact nor the purpose or policy behind the attempt to reduce it. Because of the importance of impact from an ethical and social perspective, it seems reasonable to point out some caveats for consideration by readers and to suggest other alternatives to the approach advocated by Stocking et al. (1998) that future research might wish to explore.