Grouping Effects on Jackknifed Variance Estimation for Item Response Theory Scaling and Equating With Cluster‐Based Assessment Data

Abstract

AbstractEducational assessment data are often collected from a set of test centers across various geographic regions, and therefore the data samples contain clusters. Such cluster‐based data may result in clustering effects in variance estimation. However, in many grouped jackknife variance estimation applications, jackknife groups are often formed by a random grouping method that ignores the cluster structures of the data. In this study, we constructed both random and cluster‐based jackknife groups for data known to have cluster structures and compared the jackknifed standard errors, yielded by two different grouping methods, of item response theory (IRT) scaling coefficient estimates and equated scores. Three independent data samples from an international test of English were used for the study. The cluster‐based jackknife group results showed relatively larger jackknifed standard errors of scaling coefficient estimates and scale scores than the results of the random jackknife groups for all three data samples. For cluster‐based assessment data, the cluster‐based jackknife approach provides a more appropriate way to estimate the standard errors of the parameters of IRT calibration, scaling, and equating analyses.