Generalizing the Reliability of Tests Comprised of Testlets

Abstract

It is common to find educational tests that are comprised of testlets (clusters of items that relate to a specific topic or content area). Testlet-based tests often have within-testlet covariation that is additional to the interitem covariation explained by the general trait that the test is designed to measure, which, if ignored, results in an estimate of reliability (usually coefficient a) that, depending on the inference associated with the test score, is either an underestimate or overestimate of the true reliability. Different methods for the estimation of reliability have been proposed (e.g., polytomous item response theory, stratified-a) depending on whether the testlets are fixed or random factors. The purpose of this article is to provide a general framework to estimate either total test or testlet reliability with either fixed or random factors. This general framework, which uses hierarchical factor analysis as its basis, (a) allows for a testable model to ensure that the variation is partitioned consistently with the data, (b) requires a single analysis to provide all the all the information to estimate reliability with fixed or random factors, and (c) provides for an estimate of reliability that is superior to coefficient a. An example based on real data illustrates the general framework.