The effect of question order on evaluations of test performance: how does the bias evolve?

Abstract

Weinstein and Roediger (Memory & Cognition 38:366-376, 2010) found that manipulating the order of questions on a general knowledge quiz resulted in differing evaluations of performance at the end of the quiz: Irrespective of their actual performance, participants were consistently more optimistic about their performance when questions were given in an easy-to-hard order. In the present experiment, the participants were stopped 10 times throughout a 100-item test and asked to evaluate their performance on the last 10 questions they had answered, as well as rating their impressions of the test so far and predicting their final performance. Arranging the questions from the easiest to the hardest produced more optimistic performance evaluations on each block than did an analogous hard-easy question order, even though performance on the two versions did not differ significantly as a function of question order. Furthermore, the ratings of item difficulty on each block of 10 questions were asymmetrical in the two conditions, with a higher sensitivity to increasing as compared to decreasing question difficulty. On the other hand, the item-by-item ratings and predictions remained unaffected by question order. Our findings are best explained by an anchoring interpretation, which suggests that students fail to adjust their evaluations of performance as the difficulty of the questions changes across the test.