DIF Detection Research Articles

Quantifying ‘problematic’ DIF within an IRT framework: application to a cancer stigma index

DIF detection within an IRT framework is highly powerful, often identifying significant DIF that is of little clinical importance. This paper introduces two metrics for IRT DIF evaluation that can discern potentially problematic DIF among items flagged with statistically significant DIF. Computation of two DIF metrics-(1) a weighted area between the expected score curves (wABC) and (2) a difference in expected a posteriori scores across item response categories (dEAP)-is described. Their use is demonstrated using data from a 27-item cancer stigma index fielded to four adult samples: (1) Arabic (N = 633) and (2) English speakers (N = 324) residing in Jordan and Egypt, and (3) English (N = 500) and (4) Mandarin speakers (N = 500) residing in China. We used IRTPRO's DIF module to calculate IRT-based Wald chi-square DIF statistics according to language within each region. After standard p value adjustments for multiple comparisons, we further evaluated DIF impact with wABC and dEAP. There were a total of twenty statistically significant DIF comparisons after p value adjustment. The wABCs for these items ranged from 0.13 to 0.90. Upon inspection of curves, DIF comparisons with wABCs >0.3 were deemed potentially problematic and were considered further for removal. The dEAP metric was also informative regarding impact of DIF on expected scores, but less consistently useful for narrowing down potentially problematic items. The calculations of wABC and dEAP function as DIF effect size indicators. Use of these metrics can substantially augment IRT DIF evaluation by discerning truly problematic DIF items among those with statistically significant DIF.

A REVIEW OF ETS DIFFERENTIAL ITEM FUNCTIONING ASSESSMENT PROCEDURES: FLAGGING RULES, MINIMUM SAMPLE SIZE REQUIREMENTS, AND CRITERION REFINEMENT

ABSTRACTDifferential item functioning (DIF) analysis is a key component in the evaluation of the fairness and validity of educational tests. The goal of this project was to review the status of ETS DIF analysis procedures, focusing on three aspects: (a) the nature and stringency of the statistical rules used to flag items, (b) the minimum sample size requirements that are currently in place for DIF analysis, and (c) the efficacy of criterion refinement. The main findings of the review are as follows:• The ETS C rule often displays low DIF detection rates even when samples are large.• With improved flagging rules in place, minimum sample size requirements could probably be relaxed. In addition, updated rules for combining data across administrations could allow DIF analyses to be performed in a broader range of situations.• Refinement of the matching criterion improves detection rates when DIF is primarily in one direction but can depress detection rates when DIF is balanced. If nothing is known about the likely pattern of DIF, refinement is advisable.Each of these findings is discussed in detail, focusing on the case of dichotomous items.

Nonuniform DIF Detection using Discriminant Logistic Analysis and Multinomial Logistic Regression: A comparison for polytomous items

This study focused on the effectiveness in nonuniform polytomous item DIF detection using Discriminant Logistic Analysis (DLA) and Multinomial Logistic Regression (MLR). A computer simulation study was conducted to compare the effect of using DLA and MLR, applying either an iterative test purification procedure or non-iterative to detect nonuniform DIF. The conditions under study were: DIF effect size (0.5, 1.0 and 1.5), sample size (500 and 1000), percentage of DIF items in the test (0, 10 and 20%) and DIF type (nonuniform). The results suggest that DLA is more accurate than MLR in detecting DIF. However, the purification process only improved the correct detection rate when MLR was applied. The false positive rates for both procedures were similar. Moreover, when the test purification procedure was used, the proportion of non-DIF items that were detected as DIF decreased for both procedures, although the false positive rates were smaller for DLA than for MLR.

DIF Detection and Interpretation in Large-Scale Science Assessments: Informing Item Writing Practices

DIF Detection and Interpretation in Large-Scale Science Assessments: Informing Item Writing Practices

Applying the Breslow-Day Test of Trend in Odds Ratio Heterogeneity to the Analysis of Nonuniform DIF

This article applies the Breslow-Day test of trend in odds ratio heterogeneity (BD) to the detection of nonuniform DIF. A simulation study was conducted to assess the power and Type I error rate of BD, as well as a combined decision rule (CDR) whereby a decision of the existence of DIF was based on a combination of the decisions made using BD and the Mantel-Haenszel chi-square. The results indicated that CDR displayed good Type I error rate and power across a variety of conditions. Comparing these results with those of earlier research indicates that CDR may yield more accurate decisions about DIF than other commonly used DIF detection procedures.

Applying the Breslow-Day Test of Trend in Odds Ratio Heterogeneity to the Analysis of Nonuniform DIF

This article applies the Breslow-Day test of trend in odds ratio heterogeneity (BD) to the detection of nonuniform DIF. A simulation study was conducted to assess the power and Type I error rate of BD, as well as a combined decision rule (CDR) whereby a decision of the existence of DIF was based on a combination of the decisions made using BD and the Mantel-Haenszel chi-square. The results indicated that CDR displayed good Type I error rate and power across a variety of conditions. Comparing these results with those of earlier research indicates that CDR may yield more accurate decisions about DIF than other commonly used DIF detection procedures.

Using Loss Functions for DIF Detection: An Empirical Bayes Approach

We investigated a DIF flagging method based on loss functions. The approach builds on earlier research that involved the development of an empirical Bayes (EB) enhancement to Mantel-Haenszel (MH) DIF analysis. The posterior distribution of DIF parameters was estimated and used to obtain the posterior expected loss for the proposed approach and for competing classification rules. Under reasonable assumptions about the relative seriousness of Type I and Type II errors, the loss-function-based DIF detection rule was found to perform better than the commonly used "A, " "B, " and "C" DIF classification system, especially in small samples.

Using Loss Functions for DIF Detection: An Empirical Bayes Approach

Using Loss Functions for DIF Detection: An Empirical Bayes Approach

Improved Type I Error Control and Reduced Estimation Bias for DIF Detection Using SIBTEST

One emphasis in the development and evaluation of SIBTEST has been the control of Type I error (false flagging of non-differential item functioning [DIF] items) inflation and estimation bias. SIBTEST has performed well in comparative simulation studies of Type I error and estimation bias relative to other procedures such as the Mantel-Haenszel and Logistic Regression. Nevertheless it has for a minority of cases that might occur in applications displayed sizable Type I error inflation and estimation bias. A vital part of SIBTEST is the regression correction, which adjusts for the Type I error-inflating and estimation-biasing influence of group target ability differences by using the linear regression of true on observed matching subtest scores from Classical Test Theory. In this paper, we propose a new regression correction, using essentially a two-segment piecewise linear regression of the true on observed matching subtest scores. A realistic simulation study of the new approach shows that when there is a clear group ability distributional difference, the new approach displays improved SIBTEST Type I error performance; when there is no group ability distributional difference, its Type I error rate is comparable to the current SIBTEST. We have also conducted a power study which indicates that the new approach has on average similar power as the current SIBTEST. We concluded that the new version of SIBTEST, although not perfectly robust, seems appropriately robust against sizable Type I error inflation, while retaining other desirable features of the current version.

Improved Type I Error Control and Reduced Estimation Bias for DIF Detection Using SIBTEST

Improved Type I Error Control and Reduced Estimation Bias for DIF Detection Using SIBTEST

Test Fairness: Internal and External Investigations of Gender Bias in Mathematics Testing

What two major approaches have been used to study gender bias in test scores? How do statistical DIF detection methods differ? How does DIF screening of items affect mean score differences?

A Kernel-Smoothed Version of SIBTEST With Applications to Local DIF Inference and Function Estimation

Smoothed SIBTEST, a nonparametric DIF detection procedure, amalgamates SIBTEST and kernel-smoothed item response function estimation. This procedure assesses DIF as a function of the latent trait θ that the test is designed to measure. Smoothed SIBTEST estimates this Junction with increased efficiency, as compared to SIBTEST, while providing hypothesis tests of local and global DIF. By means of kernel smoothing, smoothed SIBTEST reduces noise in local DIF estimation while retaining SIBTEST’s reduction of group-ability-difference-induced DIF estimation bias via use of regression-corrected estimates of ability as design points in the kernel smoothing. By contrast with most nonparametric procedures, matched examinee score cells are not needed, so sparse cell problems are avoided. The performance of smoothed SIBTEST is studied via simulation and real data analysis.

Detecting DIF for Polytomously Scored Items: An Adaptation of the SIBTEST Procedure

Shealy and Stout (1993) proposed a DIF detection procedure called SIBTEST and demonstrated its utility with both simulated and real data sets'. Current versions of SIBTEST can be used only for dichotomous items. In this article, an extension to handle polytomous items is developed. Two simulation studies are presented which compare the modified SIBTEST procedure with the Mantel and standardized mean difference (SMD) procedures. The first study compares the procedures under conditions in which the Mantel and SMD procedures have been shown to perform well (Zwick, Donoghue, & Grima, 1993). Results of Study I suggest that SIBTEST performed reasonably well, but that the Mantel and SMD procedures performed slightly better. The second study uses data simulated under conditions in which observed‐score DIF methods for dichotomous items have not performed well. The results of Study 2 indicate that under these conditions the modified SIBTEST procedure provides better control of impact‐induced Type I error inflation than the other procedures.