Item Writer Judgments of Item Difficulty Versus Actual Item Difficulty: A Case Study

Abstract

This study investigates how accurate one item writer can be on item difficulty estimates and whether factors affecting item writer judgments correspond to predictors of actual item difficulty. The items were based on conversational dialogs (presented as videos online) that focus on pragmatic functions. Thirty-five 2nd-, 3rd-, and 4th-year learners of Russian participated. The results indicated that the hypothesized item difficulty was a weak, although significant, predictor of actual item difficulty. Intermediate and advanced items did not differ in difficulty, which may suggest that it is more challenging to predict higher level item difficulty or to design higher level items. In general, the item writer was accurate on predictions involving linguistic focus and response format. However, the effect of topical knowledge was not accounted for. This study provides implications for training item writers and for conducting future studies on item difficulty estimation.