A Comparison of Distractor Selection Among Proficiency Levels in Reading Tests: A Focus on Summarization Processes in Japanese EFL Learners

Takahiro Terao,Hidetoki Ishii

doi:10.1177/2158244020902087

Abstract

This study aimed to compare selection patterns of distractors (incorrect options) according to test taker proficiency regarding Japanese students’ summarization skills of an English paragraph. Participants included 414 undergraduate students, and the test comprised three summarization process types—deletion, generalization, and integration. Within the questions, which represented summary candidates for a final version of a test, distractors were created reflecting typical student errors related to each summarization process. Six distractor types were tested. Results showed that distractors that were missing important information for the summary functioned well for determining low-, middle-, and high-proficiency students regarding deletion items. For generalization items, both distractor types, those containing examples and those with inappropriate superordinates, were attractive for low- and middle-proficiency students. Regarding integration items, it was found that distractors missing the author’s viewpoint in the summary were more attractive only for less-proficient students. Several tips to guide future item writing are provided.

Highlights

In educational testing, test items play an important role in gathering evidence of students’ skills and abilities
We revealed the attractiveness of each distractor in test items to measure L2 summarization skills
This study examined several types of distractors related to cognitive processes in summarization, and the same type of distractors were edited in several items

Summary

Introduction

Test items play an important role in gathering evidence of students’ skills and abilities. To obtain rich information from test items and options, it is necessary to define the skills and abilities to be measured and reflect these components in the items and options. The test development procedure includes several steps before item writing, including the overall planning of the test, content and construct definition and description, and item specification. These are all initial and essential parts of ensuring the validity of test items. In construct definition and description, the exploration of the target construct is a principal and influential step that guides item writing. Construct definition and description are called domain description and refer to how performance in the target domain is linked to the observation of performance in the test (Chapelle et al, 2008)

Objectives

Methods

Results

Conclusion