The Development of a Standardized Effect Size for the SIBTEST Procedure

Abstract

In this study a standardized effect size was created for use with the SIBTEST procedure. Using this standardized effect size, a single set of heuristics was developed that are appropriate for data fitting different item response models (e.g., 2-parameter logistic, 3-parameter logistic). The standardized effect size rescales the raw beta-uni value using a pooled variation that incorporates the beta-uni inclusion factor. Although the heuristics for the standardized and unstandardized effect sizes provide similar true-positive and false-positive rates in most conditions, the standardized effect size provides higher true-positive rates for conditions where item response variability is smaller in proportion to raw score differences. Inflated false-positive rates were solely impacted by smaller sample sizes, whereas larger sample sizes improved true-positive rates. An empirical application is provided to demonstrate how the standardized effect size provides for a more consistent comparison across items with varying response distributions. This study lays the foundation for the utilization of a standardized effect size for both dichotomous and polytomous item response models using the suite of SIBTEST procedures.