Difficulty of Summarization Items for Japanese Learners: Effects of Passages, Distractors, and Response Formats

Abstract

This study aims to examine factors affecting the difficulty of summarization items for Japanese learners. In the process of item development, a connection between cognitive features related to target construct and the difficulty of test items is necessary to check empirically in order to define the abilities to be measured. Previous studies have mainly focused on local reading comprehension, while this study addressed summarization skills at the paragraph levels. The study originally developed items for the experiment to elicit three macrorules to represent a gist of the paragraph and text; deletion, generalization, and integration. And this study evaluated the influences that passages, distractor characteristics central to summarization processes, and response formats have on item difficulty, using the item difficulty modeling. When editing distractors, characteristics of L2 learners’ summarization were carefully reviewed and reflected. The participants included 150 freshmen from Japan, who were asked to answer experimental summarization items. The results of the linear logistic test model indicated that the main source of difficulty in summarization items was distractor characteristics. In particular, summaries with unnecessary information or lacking necessary information increased difficulty. In addition, summaries with detailed information, such as episodes and examples, and with a viewpoint that differed from the author’s also increased difficulty. The effect of passage differences was found to be minimal. Difference in response formats moderately affected item difficulty, and the extended-matching format was slightly difficult than the conventional multiple-choice format. This study suggested that test developers and item writers should care about the distractor development, to contain students’ errors when measuring summarization skills of L2 learners.