Differential Item Functioning Detection Research Articles

A Synthesis of 15 Years of Research on DIF in Language Testing: Methodological Advances, Challenges, and Recommendations

This article reviews research on differential item functioning (DIF) in language testing conducted primarily between 1990 and 2005 with an eye toward providing methodological guidelines for developing, conducting, and disseminating research in this area. The article contains a synthesis of 27 studies with respect to five essential sets of characteristics: (a) features of the tests utilized, (b) features of the learner groups employed, (c) features of the DIF detection methods applied, (d) features of the reporting of DIF effects, and (e) features of the explanations for and consequences drawn from DIF results. Methodological recommendations for improving the conceptualization and the reporting of DIF research in the field of language testing is provided with an eye toward developing statistical meta-analyses that aid practitioners in interpreting the relevance of DIF results more precisely.

Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going

The purpose of this article is to reflect on the state of the theorizing and praxis of DIF in general: where it has been; where it is now; and where I think it is, and should, be going. Along the way the major trends in the differential item functioning (DIF) literature are summarized and integrated providing some organizing principles that allow one to catalog and then contrast the various DIF detection methods and to shine a light on the future of DIF analyses. The three generations of DIF are introduced and described with an eye toward issues on the horizon for DIF.

A comparison of three sets of criteria for determining the presence of differential item functioning using ordinal logistic regression

Several techniques have been developed to detect differential item functioning (DIF), including ordinal logistic regression (OLR). This study compared different criteria for determining whether items have DIF using OLR. To compare and contrast findings from three different sets of criteria for detecting DIF using OLR. General distress and physical functioning items were evaluated for DIF related to five covariates: age, marital status, gender, race, and Hispanic origin. Cross-sectional study. 1,714 patients with cancer or HIV/AIDS. A total of 23 items addressing physical functioning and 15 items addressing general distress were selected from a pool of 154 items from four different health-related quality of life questionnaires. The three sets of criteria produced qualitatively and quantitatively different results. Criteria based on statistical significance alone detected DIF in almost all the items, while alternative criteria based on magnitude detected DIF in far fewer items. Accounting for DIF by using demographic-group specific item parameters had negligible effects on individual scores, except for race. Specific criteria chosen to determine whether items have DIF have an impact on the findings. Criteria based entirely on statistical significance may detect small differences that are clinically negligible.

Iterative Purification and Effect Size Use With Logistic Regression for Differential Item Functioning Detection

Two unresolved implementation issues with logistic regression (LR) for differential item functioning (DIF) detection include ability purification and effect size use. Purification is suggested to control inaccuracies in DIF detection as a result of DIF items in the ability estimate. Additionally, effect size use may be beneficial in controlling Type I error rates. The effectiveness of such controls, especially used in combination, requires evaluation. Detection errors were evaluated through simulation across iterative purification and no purification procedures with and without the use of an effect size criterion. Sample size, DIF magnitude and percentage, and ability differences were manipulated. Purification was beneficial under certain conditions, although overall power and Type I error rates did not substantially improve. The LR statistical test without purification performed as well as other classification criteria and may be the practical choice for many situations. Continued evaluation of the effect size guidelines and purification are discussed.

SMALL‐SAMPLE DIF ESTIMATION USING LOG‐LINEAR SMOOTHING: A SIBTEST APPLICATION

ABSTRACTThe purpose of the current study was to examine whether log‐linear smoothing of observed score distributions in small samples results in more accurate differential item functioning (DIF) estimates under the simultaneous item bias test (SIBTEST) framework. Data from a teacher certification test were analyzed using White candidates in the reference group and African American candidates in the focal group. Smoothed and raw DIF estimates from 100 replications under seven different sample‐size conditions were compared to a criterion to determine the effect of smoothing on small‐sample DIF estimation. Root‐mean‐squared deviation and bias were used to evaluate the accuracy of DIF detection in the smoothed versus raw data conditions. Results indicate that, for most studied items, smoothing the raw score distributions reduced variability and bias of the DIF estimates especially in the small‐sample‐size conditions. Implications of these results for actual testing programs and future directions for research are discussed.

Evaluating measurement equivalence using the item response theory log-likelihood ratio (IRTLR) method to assess differential item functioning (DIF): applications (with illustrations) to measures of physical functioning ability and general distress

Methods based on item response theory (IRT) that can be used to examine differential item functioning (DIF) are illustrated. An IRT-based approach to the detection of DIF was applied to physical function and general distress item sets. DIF was examined with respect to gender, age and race. The method used for DIF detection was the item response theory log-likelihood ratio (IRTLR) approach. DIF magnitude was measured using the differences in the expected item scores, expressed as the unsigned probability differences, and calculated using the non-compensatory DIF index (NCDIF). Finally, impact was assessed using expected scale scores, expressed as group differences in the total test (measure) response functions. The example for the illustration of the methods came from a study of 1,714 patients with cancer or HIV/AIDS. The measure contained 23 items measuring physical functioning ability and 15 items addressing general distress, scored in the positive direction. The substantive findings were of relatively small magnitude DIF. In total, six items showed relatively larger magnitude (expected item score differences greater than the cutoff) of DIF with respect to physical function across the three comparisons: "trouble with a long walk" (race), "vigorous activities" (race, age), "bending, kneeling stooping" (age), "lifting or carrying groceries" (race), "limited in hobbies, leisure" (age), "lack of energy" (race). None of the general distress items evidenced high magnitude DIF; although "worrying about dying" showed some DIF with respect to both age and race, after adjustment. The fact that many physical function items showed DIF with respect to age, even after adjustment for multiple comparisons, indicates that the instrument may be performing differently for these groups. While the magnitude and impact of DIF at the item and scale level was minimal, caution should be exercised in the use of subsets of these items, as might occur with selection for clinical decisions or computerized adaptive testing. The issues of selection of anchor items, and of criteria for DIF detection, including the integration of significance and magnitude measures remain as issues requiring investigation. Further research is needed regarding the criteria and guidelines appropriate for DIF detection in the context of health-related items.

DIF Detection and Effect Size Measures for Polytomously Scored Items

Data from a large‐scale performance assessment (N= 105,731) were analyzed with five differential item functioning (DIF) detection methods for polytomous items to examine the congruence among the DIF detection methods. Two different versions of the item response theory (IRT) model‐based likelihood ratio test, the logistic regression likelihood ratio test, the Mantel test, and the generalized Mantel–Haenszel test were compared. Results indicated some agreement among the five DIF detection methods. Because statistical power is a function of the sample size, the DIF detection results from extremely large data sets are not practically useful. As alternatives to the DIF detection methods, four IRT model‐based indices of standardized impact and four observed‐score indices of standardized impact for polytomous items were obtained and compared with the R2 measures of logistic regression.

Differential item functioning and health assessment

Establishing measurement equivalence is important because inaccurate assessment may lead to incorrect estimates of effects in research, and to suboptimal decisions at the individual, clinical level. Examination of differential item functioning (DIF) is a method for studying measurement equivalence. An item (i.e., one question in a longer scale) exhibits DIF if the item response differs across groups (e.g., gender, race), controlling for an estimate of the construct being measured. A distinction between applications in health, as contrasted with other settings such as educational and aptitude testing, is that there are many health-related constructs and multiple measures of each, few of which have received much critical evaluation. Discussed in this article are several methods for detection of differential item functioning (DIF), including non-parametric and parametric methods such as logistic regression, and those based on item response theory. Basic definitions and criteria for DIF detection are provided, as are steps in performing the analyses. Recommendations are presented and future directions discussed.

Odds Ratio, Delta, ETS Classification, and Standardization Measures of DIF Magnitude for Binary Logistic Regression

Previous methodological and applied studies that used binary logistic regression (LR) for detection of differential item functioning (DIF) in dichotomously scored items either did not report an effect size or did not employ several useful measures of DIF magnitude derived from the LR model. Equations are provided for these effect size indices. Using two large data sets, the authors demonstrate the usefulness of these effect sizes for judging practical importance: the LR adjusted odds ratio and its conversions to the delta metric, the Educational Testing Service (ETS) classification system, and the p metric; the LR model-based standardization indices, using various weights for averaging stratum-specific differences in fitted probabilities; and a p metric classification system. Pros and cons of these effect sizes are discussed. Recommendations are offered. These LR effect sizes will be valuable to practitioners, particularly for preventing flagging of statistically significant but practically unimportant DIF in large samples.

Influence of equal or unequal comparison group sample sizes on the detection of differential item functioning using the Mantel–Haenszel and logistic regression techniques

In recent decades several methods have been developed for detecting differential item functioning (DIF), and many studies have aimed to identify both the conditions under which items may or may not be adequate and the factors which affect their power and Type I error. This paper describes a Monte Carlo experiment that was carried out in order to analyse the effect of reference group sample size, focal group sample size and the interaction of the two on the power and Type I error of the Mantel–Haenszel (MH) and Logistic regression (LR) procedures. The data were generated using a three-parameter logistic model, the design was fully-crossed factorial with 12 experimental conditions arising from the crossing of the two main factors, and the dependent variables were power and the rate of false positives calculated across 100 replications. The results enabled the significant factors to be identified and the two statistics to be compared. Practical recommendations are made regarding use of the procedures by psychologists interested in the development and analysis of psychological tests.

Rapid detection of differential item functioning in assessments of health-related quality of life: The Functional Assessment of Cancer Therapy

Differential item functioning (DIF) occurs when a test item functions differently in different groups when controlling for the level of the underlying construct measured by the test. DIF assessment is a first step in the evaluation of test bias. We sought to demonstrate a rapid hybrid approach to DIF detection by determining the presence and scale-level impact of DIF related to eight covariates in four domains measured by the Functional Assessment of Cancer Therapy (FACT). The number of items found with DIF in each domain depended on the criterion chosen to define the presence of DIF. With a few exceptions, scale-level differential functioning was similar regardless of the criteria chosen. For physical well-being, there was relevant scale-level differential functioning related only to race. For social and family well-being, there was relevant scale-level differential functioning related to each of the covariates. For emotional well-being, there was relevant scale-level differential functioning related to ethnicity, language, and race. For functional well-being, there was relevant scale-level differential functioning related to ethnicity, race, education, and self- vs. interviewer-administration. Our rapid hybrid approach to DIF detection may be broadly applicable in other studies of health-related quality of life.

Overview of Quantitative Measurement Methods

Reviewed in this article are issues relating to the study of invariance and differential item functioning (DIF). The aim of factor analyses and DIF, in the context of invariance testing, is the examination of group differences in item response conditional on an estimate of disability. Discussed are parameters and statistics that are not invariant and cannot be compared validly in crosscultural studies with varying distributions of disability in contrast to those that can be compared (if the model assumptions are met) because they are produced by models such as linear and nonlinear regression. The purpose of this overview is to provide an integrated approach to the quantitative methods used in this special issue to examine measurement equivalence. The methods include classical test theory (CTT), factor analytic, and parametric and nonparametric approaches to DIF detection. Also included in the quantitative section is a discussion of item banking and computerized adaptive testing (CAT). Factorial invariance and the articles discussing this topic are introduced. A brief overview of the DIF methods presented in the quantitative section of the special issue is provided together with a discussion of ways in which DIF analyses and examination of invariance using factor models may be complementary. Although factor analytic and DIF detection methods share features, they provide unique information and can be viewed as complementary in informing about measurement equivalence.

Differential Item Functioning Analysis With Ordinal Logistic Regression Techniques

We present an ordinal logistic regression model for identification of items with differential item functioning (DIF) and apply this model to a Mini-Mental State Examination (MMSE) dataset. We employ item response theory ability estimation in our models. Three nested ordinal logistic regression models are applied to each item. Model testing begins with examination of the statistical significance of the interaction term between ability and the group indicator, consistent with nonuniform DIF. Then we turn our attention to the coefficient of the ability term in models with and without the group term. If including the group term has a marked effect on that coefficient, we declare that it has uniform DIF. We examined DIF related to language of test administration in addition to self-reported race, Hispanic ethnicity, age, years of education, and sex. We used PARSCALE for IRT analyses and STATA for ordinal logistic regression approaches. We used an iterative technique for adjusting IRT ability estimates on the basis of DIF findings. Five items were found to have DIF related to language. These same items also had DIF related to other covariates. The ordinal logistic regression approach to DIF detection, when combined with IRT ability estimates, provides a reasonable alternative for DIF detection. There appear to be several items with significant DIF related to language of test administration in the MMSE. More attention needs to be paid to the specific criteria used to determine whether an item has DIF, not just the technique used to identify DIF.

Different Approaches to Differential Item Functioning in Health Applications

Reviewed in this article are topics related to the study of invariance and differential item functioning (DIF) that have received relatively little attention in the literature. Several factors influence DIF detection; these include (1) model fit, (2) model assumptions, (3) disability distributions, (4) purification, (5) cutoff values for magnitude measures, and (6) sample and scale size. Approaches to DIF detection are discussed in terms of model assumptions, purification, magnitude and impact, and possible advantages and disadvantages of each method. An integrated approach to the examination of measurement equivalence, invariance, and DIF is necessary for measurement in an increasingly multi-ethnic society. Ideally, qualitative analyses should be performed in an iterative fashion to inform about findings of DIF. However, if an already-developed measure is being evaluated, then the steps might be to focus first on dimensional invariance using factor analytic methods, followed by DIF analyses examining both significance and magnitude of DIF, accompanied by formal tests of the impact of DIF. The DIF analytic method selected in the second step might be determined based on the findings summarized in the table presented within this paper.

Comparing Methods of Assessing Differential Item Functioning in a Computerized Adaptive Testing Environment

Mantel‐Haenszel and SIBTEST, which have known difficulty in detecting non‐unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item‐response‐theory‐likelihood‐ratio test (IRT‐LRT), capable of detecting both unidirectional and non‐unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT‐LRT procedures are more powerful than the CAT version of SIBTEST for non‐unidirectional DIF detection. The good Type I error control provided by IRT‐LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed.

Power and Type I Error of the Meanand Covariance Structure Analysis Model for Detecting Differential Item Functioning in Graded Response Items

In this simulation study, we investigate the power and Type I error rate of a procedure based on the mean and covariance structure analysis (MACS) model in detecting differential item functioning (DIF) of graded response items with five response categories. The following factors were manipulated: type of DIF (uniform and non-uniform), DIF magnitude (low, medium and large), equality/inequality of latent trait distributions, sample size (100, 200, 400, and 800) and equality or inequality of the sample sizes across groups. The simulated test was made up of 10 items, of which only 1 contained DIF. One hundred replications were generated for each simulated condition. Results indicate that the MACS-based procedure showed acceptable power levels (≥ .70) for detecting medium-sized uniform and non-uniform DIF, when both groups' sample sizes were as low as 200/200 and 400/200, respectively. Power increased as sample sizes and DIF magnitude increased. The analyzed procedure tended to better control for its Type I error when both groups' sizes and latent trait distribution were equal across groups and when magnitude of DIF and sample size were small.

Fixed Effects IRT Model

A fixed effect item response theory (IRT) model is developed for modeling group specific item parameters. Two applications are presented. The first application is that the proposed model can be used to detect whether a response mechanism is ignorable using the splitter item technique. The second application is the detection of differential item functioning. In the latter application, the fixed effect item parameters can model item parameter differences between groups. Simulation studies are presented to show the feasibility and performance of the method on both applications.

Identification of Differential Item Functioning Using Item Response Theory and the Likelihood-Based Model Comparison Approach

An important part of examining the adequacy of measures for use in ethnically diverse populations is the evaluation of differential item functioning (DIF) among subpopulations such as those administered the measure in different languages. A number of methods exist for this purpose. The objective of this study was to introduce and demonstrate the identification of DIF using item response theory (IRT) and the likelihood-based model comparison approach. Data come from a sample of community-residing elderly who were part of a dementia case registry. A total of 1578 participants were administered either an English (n = 913) or Spanish (n = 665) version of the 21-item Mini-Mental State Examination. IRT was used to identify language DIF in these items with the likelihood-based model comparison approach. : Fourteen of the 21 items exhibited significant DIF according to language of administration. However, because the direction of the identified DIF was not consistent for one language version over the other, the impact at the scale level was negligible. IRT and the likelihood-based model comparison approach comprise a powerful tool for DIF detection that can aid in the development, refinement, and evaluation of measures for use in ethnically diverse populations.

USING DIF DISSECTION METHOD TO ASSESS EFFECTS OF ITEM DELETION

Statistical procedures for detecting differential item functioning (DIF) are often used as an initial step to screen items for construct irrelevant variance. This research applies a DIF dissection method and a two-way classification scheme to SAT Reasoning Test™ verbal section data and explores the effects of deleting sizable DIF items on reported scores after re-equating. The DIF dissection approach and the two-way classification scheme may yield new and detailed insight into item functioning at the subgroup level. Two hypotheses are studied: (1) whether or not the deletion of a sizeable DIF item that is the most disadvantageous to a particular subgroup will affect the scores for that subgroup the most; and (2) whether or not the effects of item deletion on scores can be predicted by the standardization method. Both hypotheses are supported by the results of this research.

A Comparison of Four Methods for Detecting Differential Item Functioning in Ordered Response Items

Item bias is a major threat to measurement validity. Methods for detecting differential item functioning (DIF) are now commonly used to identify potentially biased items. DIF detection methods for dichotomous items are well developed, but those for ordinal items are less well developed. In this article, the authors compare four methods for detecting DIF in ordinal items: the Mantel, generalized Mantel-Haenszel (GMH), logistic discriminant function analysis (LDFA), and unconstrained cumulative logits ordinal logistic regression (UCLOLR). Factors varied include type of DIF, group ability differences, studied item discrimination, skewness in ability distributions, and sample size ratio. All procedures had good Type I error control as well as high power for detecting uniform DIF. However, the Mantel could not detect nonuniform DIF, and the LDFA also performed poorly in detecting nonuniform DIF, particularly when item discrimination was high. The UCLOLR and GMH performed extremely well under conditions simulated in this study. Implications for research and practice are discussed.