Method Bias in Cloze Tests as Reading Comprehension Measures

Purya Baghaei,Hamdollah Ravand

doi:10.1177/2158244019832706

Purya Baghaei, Hamdollah Ravand

Open Access

https://doi.org/10.1177/2158244019832706

Copy DOI

Abstract

In many reading comprehension tests, different test formats are employed. Two commonly used test formats to measure reading comprehension are sustained passages followed by some questions and cloze items. Individual differences in handling test format peculiarities could constitute a source of score variance. In this study, a bifactor Rasch model is applied to separate the cloze-specific variance in a reading comprehension test composed of sustained passages (plus questions) and a cloze passage. The results are compared with a unidimensional Rasch model where all items load on a single dimension. The inclusion of the cloze-specific dimension, that is, the method factor, improved the fit and resulted in substantially lower item difficulty estimates for the cloze items. Findings indicate that reading comprehension tests comprising sustained passages and cloze items are not unidimensional and contain a cloze-specific nuisance dimension that contaminates the latent construct variance.

Highlights

Mixed-format items are gaining increasing popularity in high-stakes testing
The term mixed format has commonly been associated with tests that are composed of multiple choice (MC) and constructed response (CR) items
A combination of MC reading comprehension items and MC cloze tests, which are used in some high-stakes tests, is an instance of mixed format tests

Summary

Introduction

Mixed-format items are gaining increasing popularity in high-stakes testing. The term mixed format has commonly been associated with tests that are composed of multiple choice (MC) and constructed response (CR) items. It is believed that tests composed of multimethod items are psychometrically more advantageous (Wang, Drasgow, & Liu, 2016). In tests, where multimethod items are employed, it is believed that items of different formats complement each other. The depth of the knowledge tested by CR items, for example, can complement the breadth of knowledge covered by MC items. This advantage might come at a price. Use of mixed format items within the same test intended to be a measure of a single construct might lead to multidimensionality

Objectives

Results

Conclusion