Estimating Reliability Under a Generalizability Theory Model for Test Scores Composed of Testlets

Abstract

Previous studies have indicated that the reliability of test scores composed of testlets might be overestimated by conventional item-based reliability estimation methods (Anastasi, 1988; Sireci, Thissen, & Wainer, 1991; Thorndike, 1951; Wainer, 1995; Wainer & Thissen, 1996). We designed this study to investigate the appropriateness and implications of using a generalizability theory (G-theory) approach to estimating the reliability of scores from tests composed of testlets. The magnitude of overestimation from using Cronbach's alpha based on item scores in this situation was found to be about 0.04 relative to the testlet approach with G-theory. The generalizability coefficients based on varying numbers of passages and a fixed total number of items were found to be more variable than when the number of passages was fixed, the total number of items was fixed, and the number of items per passage varied. Therefore, manipulating the number of passages is a more productive way to obtain efficient measurement procedures than manipulating the number of items within each passage.