The Effects of Small Sample Size on Identifying Polytomous DIF Using the Liu–Agresti Estimator of the Cumulative Common Odds Ratio

Abstract

This study is an evaluation of the behavior of the Liu—Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu—Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no simulation study focusing on small samples has analyzed this estimator regarding effect size, Type I error, and power rates. The article begins with a description of this estimator in the context of polytomous DIF. Then it presents the methods and results for a simulation study in which three factors are manipulated: between-group difference in ability distribution, form of DIF introduced into the polytomous item, and sample size. The results of this study indicate that for samples smaller than 200, very little power was observed for statistical significance testing; however, Type I error rates were close to nominal levels, and the recovery of the log odds ratio as an effect size was relatively unaffected by sample size for high discriminating items. Implications for practice are discussed.