Estimating the Variability of Estimated Variance Components for Generalizability Theory

Abstract

Generalizability theory is widely applied in psychological and educational measurement.The variability of estimated variance component,which is constrained by sampling,is the "Achilles heel" of generalizability theory. Therefore,estimating the variability of estimated variance components needs to be further explored. In previous literature,some problems remain to be settled:first,the previous studies failed to compare the variability of estimated variance components among different methods simultaneously:traditional,bootstrap,jackknife and Markov Chain Monte Carlo (MCMC); second,some studies only focused on such variability of estimated variance components as the standard error,while neglected other variability such as confidence interval; last but not least,MCMC method which can be used in generalizability theory hasn't gained sufficient exploration. There are different methods to estimate the variability of variance components:standard error and confidence interval,including traditional,bootstrap,jackknife and MCMC. Based on these four methods,the study adopts Monte Carlo data simulation technique to compare the variability of estimated variance components for normal distribution data. For traditional method,ANOVA is used to estimate the variance components and their standard errors. Satterthwaite and TBGJL are used to estimate the confidence intervals. For bootstrap method,twelve bootstrap strategies are adopted,but only three strategies are considered in jackknife method and only two strategies,i.e.,informative and non-informative priors,for MCMC method. Some criteria are set to compare the four methods. The bias is cared about when variance components and their standard errors are estimated. The smaller the absolute bias is,the more reliable the result is. The criterion of confidence intervals is "80% interval coverage". If the "80% interval coverage" is more closed to 0.80,the confidence interval is more reliable. The simulation is implemented in R statistical programming environment. To link R program with WinBUGS,R2WinBUGS and Coda package are adopted. And the simulation results are as follows. First,it is more accurate to use traditional,jackknife,adjusted bootstrap and MCMC method with informative priors to estimate variance components. But unadjusted bootstrap and MCMC method with non-informative priors are not adequate. Second,traditional and MCMC method with informative priors are accurate to estimate standard errors of three variance components,while jackknife method is not. Bootstrap method needs to adopt a "divide-and-conquer" strategy to obtain good estimated standard errors:the most accurate estimation of standard error for person is consistently provided by adjusted boot-p or boot-ir; adjusted boot-pi is the clear winner in estimating standard error for item; adjusted boot-ir,boot-pir or boot-pr give good estimate of standard error for person and item. Finally,using traditional and MCMC method to estimate confidence intervals is suitable because their respective interval coverages are close to 0.80. But jackknife method is not accurate in estimating confidence intervals. There is no great difference between Bootstrap-PC and Bootstrap-BCa method that are used to estimate confidence intervals of three variance components. Bootstrap method should apply the "divide-and-conquer" strategy to get desirable estimated confidence intervals as following:adjusted boot-p for person; adjusted boot-pi for item; adjusted boot-ir,boot-pr or boot-p for person and item. This study shows that jackknife method is not accurate to estimate the variability of estimated variance components. If the "divide-and-conquer" strategy is not used in bootstrap method,it is advisable to use traditional method or MCMC method with informative priors.