“Rassling the Hog”: The Influence of Correlated Item Error on Internal Consistency, Classical Reliability, and Congeneric Reliability

Abstract

The properties of internal consistency (α), classical reliability (ρ), and congeneric reliability (ω) for a composite test with correlated item error are analytically investigated. Possible sources of correlated item error are contextual effects, item bundles, and item models that ignore additional attributes or higher-order attributes. The relation between reliability and internal consistency is determined by the deviance from true-score equivalence. The influence of correlated item error on α, ρ, and ω is conveyed strictly through the total item error covariance. As the total item error covariance increases, ρ and ω decrease, but α increases. The necessary and sufficient condition for α to be a lower bound to ρ and to ω is that the total item error covariance not exceed the deviance from true-score equivalence. Coefficient α will uniformly exceed ρ or ω in true-score equivalent tests with positively correlated item error. Index terms: classical reliability, coefficient alpha, coefficient omega, compound symmetry, congeneric reliability, correlated item error, deviance from true-score equivalence, internal consistency, reliability, total item error covariance, true-score equivalence. The purpose of this article is to analytically compare internal consistency to reliability for composite tests in which the item errors might be correlated. The psychometric literature variously presentsinternalconsistencyasidenticaltoreliability,asanotherkindofreliability(internalconsistency reliability), as a lower bound to reliability, as an approximator to reliability, or as an estimator of reliability. This explication, however, begins atfirst principles and makes no assumptions regarding the relationship between internal consistency and reliability. The development focuses entirely on internal consistency and reliability as parameters; their estimation is only briefly discussed near the end. The notation and model specifications, except for the notation for the true score, are borrowed freely from a special case (continuous variables, no exogenous variables) of Muth´ (2002) structuralequationmodel(seealso,Muth´