Framing Effects on the Setting of Critical Scores for Content Valid Tests

Abstract

The setting of critical scores frequently relies on judgments of item difficulty made by subject matter experts. However, the setting of these critical scores may be susceptible to various judgmental biases. Using a written driver's test, I conducted 2 experiments that examined how the framing of instructions to judges may affect estimates of item difficulty. Participants who were instructed to estimate the percentage of average drivers who would get each item correct (positive frame) provided item estimates that were significantly lower (more difficult) than participants who were instructed to estimate the percentage of average drivers who would get each item incorrect (negative frame). Though the framing conditions did not differ in accuracy of the item difficulty estimates, an average of the 2 framing conditions resulted in the most accurate estimates of item difficulty.