Differential Item Functioning Detection Research Articles

Assessment of Differential Item Functioning in Health-Related Outcomes: A Simulation and Empirical Analysis with Hierarchical Polytomous Data.

Background The purpose of this study was to evaluate the effectiveness of two methods of detecting differential item functioning (DIF) in the presence of multilevel data and polytomously scored items. The assessment of DIF with multilevel data (e.g., patients nested within hospitals, hospitals nested within districts) from large-scale assessment programs has received considerable attention but very few studies evaluated the effect of hierarchical structure of data on DIF detection for polytomously scored items. Methods The ordinal logistic regression (OLR) and hierarchical ordinal logistic regression (HOLR) were utilized to assess DIF in simulated and real multilevel polytomous data. Six factors (DIF magnitude, grouping variable, intraclass correlation coefficient, number of clusters, number of participants per cluster, and item discrimination parameter) with a fully crossed design were considered in the simulation study. Furthermore, data of Pediatric Quality of Life Inventory™ (PedsQL™) 4.0 collected from 576 healthy school children were analyzed. Results Overall, results indicate that both methods performed equivalently in terms of controlling Type I error and detection power rates. Conclusions The current study showed negligible difference between OLR and HOLR in detecting DIF with polytomously scored items in a hierarchical structure. Implications and considerations while analyzing real data were also discussed.

The Effect of Small Sample Size on Measurement Equivalence of Psychometric Questionnaires in MIMIC Model: A Simulation Study

Evaluating measurement equivalence (also known as differential item functioning (DIF)) is an important part of the process of validating psychometric questionnaires. This study aimed at evaluating the multiple indicators multiple causes (MIMIC) model for DIF detection when latent construct distribution is nonnormal and the focal group sample size is small. In this simulation-based study, Type I error rates and power of MIMIC model for detecting uniform-DIF were investigated under different combinations of reference to focal group sample size ratio, magnitude of the uniform-DIF effect, scale length, the number of response categories, and latent trait distribution. Moderate and high skewness in the latent trait distribution led to a decrease of 0.33% and 0.47% power of MIMIC model for detecting uniform-DIF, respectively. The findings indicated that, by increasing the scale length, the number of response categories and magnitude DIF improved the power of MIMIC model, by 3.47%, 4.83%, and 20.35%, respectively; it also decreased Type I error of MIMIC approach by 2.81%, 5.66%, and 0.04%, respectively. This study revealed that power of MIMIC model was at an acceptable level when latent trait distributions were skewed. However, empirical Type I error rate was slightly greater than nominal significance level. Consequently, the MIMIC was recommended for detection of uniform-DIF when latent construct distribution is nonnormal and the focal group sample size is small.

An Item Analysis of the French Version of the Test for Reception of Grammar Among Children and Adolescents With Down Syndrome or Intellectual Disability of Undifferentiated Etiology.

An item analysis of Bishop's (1983) Test for Reception of Grammar (TROG) in its French version (F-TROG; Lecocq, 1996) was conducted to determine whether the difficulty of items is similar for participants with or without intellectual disability (ID). In Study 1, responses to the 92 F-TROG items by 55 participants with Down syndrome (DS), 55 with ID of undifferentiated etiology (UND), and 55 typical children (TYP) matched on their F-TROG total score were compared using the transformed item difficulties method, a statistical approach designed to detect differential item functioning (DIF) between groups. In Study 2, an additional comparison involving 526 TYP participants and 526 participants with UND was conducted to increase the statistical power of the analysis. The difficulty of items was highly similar whatever the sample size or clinical status of participants. Fewer than 3.5% of the items were flagged as showing DIF. Tests such as the TROG can be used with confidence in clinical practice as well as in research studies comparing participants with or without ID. Methods designed for investigating potential internal test bias-such as done here-should be more regularly employed in the developmental disability field to affirm the absence of DIF.

Comparing DIF methods for data with dual dependency

The current study compared four differential item functioning (DIF) methods to examine their performances in terms of accounting for dual dependency (i.e., person and item clustering effects) simultaneously by a simulation study, which is not sufficiently studied under the current DIF literature. The four methods compared are logistic regression accounting neither person nor item clustering effect, hierarchical logistic regression accounting for person clustering effect, the testlet model accounting for the item clustering effect, and the multilevel testlet model accounting for both person and item clustering effects. The secondary goal of the current study was to evaluate the trade-off between simple models and complex models for the accuracy of DIF detection. An empirical example analyzing the 2011 TIMSS Mathematics data was also included to demonstrate the differential performances of the four DIF methods. A number of DIF analyses have been done on the TIMSS data, and rarely had these analyses accounted for the dual dependence of the data. Results indicated the complex models did not outperform simple models under certain conditions, especially when DIF parameters were considered in addition to significance tests. Results of the current study could provide supporting evidence for applied researchers in selecting the appropriate DIF methods under various conditions.

Does Differential Item Functioning Occur Across Respondents’ Characteristics in Safety Attitudes Questionnaire?

encouraged give a lower score to an item about self-confidence than those from a country where more self-assured people are well-respected, even when their underground trait levels of selfconfidence are the same. In more psychometric terminology, this phenomenon—namely, unequal responding patterns among groups—is called differential item functioning (DIF), which is a profound bias threatening survey-based research [3]. If DIF is in doubt, we naturally question whether a difference in survey scores between two groups stems from the real difference in the trait that we want to measure or DIF between the groups, at least to a certain degree [4]. Thus, it is essential to ensure equivalence in the responding pattern for survey items among groups before moving forward to any group-to-group comparison of survey scores and more sophisticated analysis. Unfortunately, however, more often than not, this step is omitted in survey-based studies [5]. In this series of Safety Attitudes Questionnaire–Korean Version (SAQ-K) articles, we have intentionally postponed the discussion on DIF [6-9] because we planned to utilize item response theory (IRT) for DIF detection. Using IRT is known to be a superior method given its conditional invariance property, which enables better decisions on DIF than traditional sum scores of a questionnaire [10]. We waited for the successful application

Detection of Uniform and Nonuniform Differential Item Functioning by Item-Focused Trees

Detection of differential item functioning (DIF) by use of the logistic modeling approach has a long tradition. One big advantage of the approach is that it can be used to investigate nonuniform (NUDIF) as well as uniform DIF (UDIF). The classical approach allows one to detect DIF by distinguishing between multiple groups. We propose an alternative method that is a combination of recursive partitioning methods (or trees) and logistic regression methodology to detect UDIF and NUDIF in a nonparametric way. The output of the method are trees that visualize in a simple way the structure of DIF in an item showing which variables are interacting in which way when generating DIF. In addition, we consider a logistic regression method, in which DIF can be induced by a vector of covariates, which may include categorical but also continuous covariates. The methods are investigated in simulation studies and illustrated by two applications.

A Monte Carlo Study of an Iterative Wald Test Procedure for DIF Analysis.

This study examined the performance of a proposed iterative Wald approach for detecting differential item functioning (DIF) between two groups when preknowledge of anchor items is absent. The iterative approach utilizes the Wald-2 approach to identify anchor items and then iteratively tests for DIF items with the Wald-1 approach. Monte Carlo simulation was conducted across several conditions including the number of response options, test length, sample size, percentage of DIF items, DIF effect size, and type of cumulative DIF. Results indicated that the iterative approach performed well for polytomous data in all conditions, with well-controlled Type I error rates and high power. For dichotomous data, the iterative approach also exhibited better control over Type I error rates than the Wald-2 approach without sacrificing the power in detecting DIF. However, inflated Type I error rates were found for the iterative approach in conditions with dichotomous data, noncompensatory DIF, large percentage of DIF items, and medium to large DIF effect sizes. Nevertheless, the Type I error rates were substantially less inflated in those conditions compared with the Wald-2 approach.

Using a Multidimensional IRT Framework to Better Understand Differential Item Functioning (DIF): A Tale of Three DIF Detection Procedures.

The theoretical reason for the presence of differential item functioning (DIF) is that data are multidimensional and two groups of examinees differ in their underlying ability distribution for the secondary dimension(s). Therefore, the purpose of this study was to determine how much the secondary ability distributions must differ before DIF is detected. Two-dimensional binary data sets were simulated using a compensatory multidimensional item response theory (MIRT) model, incrementally varying the mean difference on the second dimension between reference and focal group examinees while systematically increasing the correlation between dimensions. Three different DIF detection procedures were used to test for DIF: (1) SIBTEST, (2) Mantel-Haenszel, and (3) logistic regression. Results indicated that even with a very small mean difference on the secondary dimension, smaller than typically considered in previous research, DIF will be detected. Additional analyses indicated that even with the smallest mean difference considered in this study, 0.25, statistically significant differences will almost always be found between reference and focal group examinees on subtest scores consisting of items measuring the secondary dimension.

A Comparison of Linking Methods for Estimating National Trends in International Comparative Large‐Scale Assessments in the Presence of Cross‐National DIF

Trend estimation in international comparative large‐scale assessments relies on measurement invariance between countries. However, cross‐national differential item functioning (DIF) has been repeatedly documented. We ran a simulation study using national item parameters, which required trends to be computed separately for each country, to compare trend estimation performances to two linking methods employing international item parameters across several conditions. The trend estimates based on the national item parameters were more accurate than the trend estimates based on the international item parameters when cross‐national DIF was present. Moreover, the use of fixed common item parameter calibrations led to biased trend estimates. The detection and elimination of DIF can reduce this bias but is also likely to increase the total error.

Differential Item Functioning in Online Learning Instrument (EPFun)

Providing young learners with high-quality digital primary education and literacy skills are necessary for them to excel academically. The development of the online Fun Learning Instrument known as English Primary Fun (EPFun) is based on the Piaget Constructivism Development theory. This theory could be applied to young learners as it includes cognitive, behaviourist and second-language acquisition theory as well. The purpose of this study is to detect Differential Item Functioning (DIF) in performances between genders among Malaysian young learners in rural primary schools. The data were obtained from the randomly selected respondents of 106 male and 144 female young learners from the target group. The EPFun instrument consists of four constructs: a) Usefulness (USFN), b) Ease of Use (EOU), c) Ease of Learning (EOL) and d) Satisfaction (SAT). To analyse the responses to the instruments, 40 items overall were made up using 5 levels of coded smileys from the 4 constructs. The data were analysed using SPSS and Winsteps version 3.68.2, a Rasch-based item analysis programme. The findings indicate there is no significant difference in DIF performance between male and female young learners. This conclusion is derived from the t-test for Equality of Means, which revealed the results of the Sig. (2 tailed) for every construct (USFN; Sig = 0.558; EOU; Sig = 0.638; EOL; Sig = 0.628 and SAT; Sig = 0.500) larger than 0.5. The findings also revealed that there were only 7 items detected as DIF. However, there is no significant DIF based on the size range of the logits scales. Female young learners excel in one construct-Satisfaction (SAT). However, male young leaners are on a par with female young learners with regard to other constructs. Thus, this instrument is free from DIF and could be used as an indicator to gauge young learners’ literacy skills, especially in using hypermedia reading materials in English.

A New Measurement Equivalence Technique Based on Latent Class Regression as Compared with Multiple Indicators Multiple Causes.

Background:Measurement equivalence is an essential prerequisite for making valid comparisons in mental health questionnaires across groups. In most methods used for assessing measurement equivalence, which is known as Differential Item Functioning (DIF), latent variables are assumed to be continuous.Objective:To compare a new method called Latent Class Regression (LCR) designed for discrete latent variable with the multiple indicators multiple cause (MIMIC) as a continuous latent variable technique to assess the measurement equivalence of the 12-item General Health Questionnaire (GHQ-12), which is a cross deferent subgroup of Iranian nurses.Methods:A cross-sectional survey was conducted in 2014 among 771 nurses working in the hospitals of Fars and Bushehr provinces of southern Iran. To identify the Minor Psychiatric Disorders (MPD), the nurses completed self-report GHQ-12 questionnaires and sociodemographic questions. Two uniform-DIF detection methods, LCR and MIMIC, were applied for comparability when the GHQ-12 score was assumed to be discrete and continuous, respectively.Results:The result of fitting LCR with 2 classes indicated that 27.4% of the nurses had MPD. Gender was identified as an influential factor of the level of MPD.LCR and MIMIC agree with detection of DIF and DIF-free items by gender, age, education and marital status in 83.3, 100.0, 91.7 and 83.3% cases, respectively.Conclusions:The results indicated that the GHQ-12 is to a great degree, an invariant measure for the assessment of MPD among nurses. High convergence between the two methods suggests using the LCR approach in cases of discrete latent variable, e.g. GHQ-12 and adequate sample size.

An NCME Instructional Module on Latent DIF Analysis Using Mixture Item Response Models

The purpose of this ITEMS module is to provide an introduction to differential item functioning (DIF) analysis using mixture item response models. The mixture item response models for DIF analysis involve comparing item profiles across latent groups, instead of manifest groups. First, an overview of DIF analysis based on latent groups, called latent DIF analysis, is provided and its applications in the literature are surveyed. Then, the methodological issues pertaining to latent DIF analysis are described, including mixture item response models, parameter estimation, and latent DIF detection methods. Finally, recommended steps for latent DIF analysis are illustrated using empirical data.

Detection of Differential Item Functioning in the Cornell Critical Thinking Test Between Turkish and United States Students

Abstract. Critical thinking is a broad term that includes core elements such as reasoning, evaluating, and metacognition that should be transferred to students in educational systems. The integration of such skills into models of student success is increasing on an international scale. The Cornell Critical Thinking Test is an internationally used tool to assess critical thinking skills. However, limited validity evidence of the translated versions of the instrument exists to support the inferences based on the CCTT scores. This study examined the CCTT Turkish version. Specifically, translated items were examined for measurement equivalence by determining if items function differently across students from United States and Turkey. Differential Item Functioning (DIF) analysis via logistic regression was employed. Results demonstrated that each subtest contained DIF items and 10% of the items in the instrument were identified as DIF. Mean differences between students in each country were not influenced by these items. A critical content review of the translated item gave insight as to why items may be functioning differently.

Transforming SIBTEST to Account for Multilevel Data Structures

SIBTEST is a differential item functioning (DIF) detection method that is accurate and effective with small samples, in the presence of group mean differences, and for assessment of both uniform and nonuniform DIF. The presence of multilevel data with DIF detection has received increased attention. Ignoring such structure can inflate Type I error. This simulation study examines the performance of newly developed multilevel adaptations of SIBTEST in the presence of multilevel data. Data were simulated in a multilevel framework and both uniform and nonuniform DIF were assessed. Study results demonstrated that naïve SIBTEST and Crossing SIBTEST, ignoring the multilevel data structure, yield inflated Type I error rates, while certain multilevel extensions provided better error and accuracy control.

A Comparative Analysis of the Efficacy of Differential Item Functioning Detection for Dichotomously Scored Items among Logistic Regression, SIBTEST and Raschtree Methods

A variety of statistical methods is available for detecting Differential Item Functioning (DIF) in the Rasch model. Most of these methods consist of two approaches. The first is based on the comparison of the item parametersestimate pre-specified groups of subjects. The second is based on a comparison between all possible groups of subjects, regardless of person covariates. The purpose of this research is to compare the efficiency of Differential Item Functioning (DIF) detection between three methods: (1) Logistic regression based on classical test theory; (2) SIBTEST based on item response theory and (3) Raschtree based on model-based recursive partitioning. Detection of DIF was collected through simulation. The first phase of the study revealed the advantages and disadvantages of these three methods from a literature review. The second phase of the research compared the efficiency of DIF detection using data simulation of four factors: type of item, number of DIF, test length and sample size.

DIF Detection Using Multiple‐Group Categorical CFA With Minimum Free Baseline Approach

The aim of this study is to assess the efficiency of using the multiple‐group categorical confirmatory factor analysis (MCCFA) and the robust chi‐square difference test in differential item functioning (DIF) detection for polytomous items under the minimum free baseline strategy. While testing for DIF items, despite the strong assumption that all but the examined item are set to be DIF‐free, MCCFA with such a constrained baseline approach is commonly used in the literature. The present study relaxes this strong assumption and adopts the minimum free baseline approach where, aside from those parameters constrained for identification purpose, parameters of all but the examined item are allowed to differ among groups. Based on the simulation results, the robust chi‐square difference test statistic with the mean and variance adjustment is shown to be efficient in detecting DIF for polytomous items in terms of the empirical power and Type I error rates. To sum up, MCCFA under the minimum free baseline strategy is useful for DIF detection for polytomous items.

The Matching Criterion Purification for Differential Item Functioning Analyses in a Large-Scale Assessment.

The current study investigated the impact of matching criterion purification on the accuracy of differential item functioning (DIF) detection in large-scale assessments. The three matching approaches for DIF analyses (block-level matching, pooled booklet matching, and equated pooled booklet matching) were employed with the Mantel-Haenszel procedure. Five factors-the length of a test, the proportion of items exhibiting DIF, a sample size, a ratio of a reference and focal group, and the existence of an average ability difference between two groups-were manipulated. The three matching approaches were used with and without purification. Also, a systematic test form difference was considered. The results indicated that overall, matching criterion purification in the three approaches contributed to the improvement of power in the detection of DIF. Depending on the psychometric characteristics of items exhibiting DIF and the existence of an average ability difference, the amount of power improvement due to matching criterion purification was different across the three approaches. The purification of a matching criterion contributed to the slight reduction of Type I error rates in the three approaches when no mean ability difference existed between the two groups. Considering power improvement with the control of Type I error rates, the purification of a matching criterion in the pooled booklet matching and the equated pooled booklet matching approaches can be recommended for DIF analyses in large-scale assessments.

Detection of Differential Item Functioning Using the Lasso Approach

This article proposes a novel approach to detect differential item functioning (DIF) among dichotomously scored items. Unlike standard DIF methods that perform an item-by-item analysis, we propose the “LR lasso DIF method”: logistic regression (LR) model is formulated for all item responses. The model contains item-specific intercepts, an effect of the sum score, and item-group interaction (i.e., DIF) effects, with a lasso penalty on all DIF parameters. Optimal penalty parameter selection is investigated through several known information criteria (Akaike information criterion, Bayesian information criterion, and cross validation) as well as through a newly developed alternative. A simulation study was conducted to compare the global performance of the suggested LR lasso DIF method to the LR and Mantel–Haenszel methods (in terms of false alarm and hit rates). It is concluded that for small samples, the LR lasso DIF approach globally outperforms the LR method, and also the Mantel–Haenszel method, especially in the presence of item impact, while it yields similar results with larger samples.

The Mediated MIMIC Model for Understanding the Underlying Mechanism of DIF.

Due to its flexibility, the multiple-indicator, multiple-causes (MIMIC) model has become an increasingly popular method for the detection of differential item functioning (DIF). In this article, we propose the mediated MIMIC model method to uncover the underlying mechanism of DIF. This method extends the usual MIMIC model by including one variable or multiple variables that may completely or partially mediate the DIF effect. If complete mediation effect is found, the DIF effect is fully accounted for. Through our simulation study, we find that the mediated MIMIC model is very successful in detecting the mediation effect that completely or partially accounts for DIF, while keeping the Type I error rate well controlled for both balanced and unbalanced sample sizes between focal and reference groups. Because it is successful in detecting such mediation effects, the mediated MIMIC model may help explain DIF and give guidance in the revision of a DIF item.

A Methodology for Zumbo’s Third Generation DIF Analyses and the Ecology of Item Responding

Methods for detecting differential item functioning (DIF) and item bias are typically used in the process of item analysis when developing new measures; adapting existing measures for different populations, languages, or cultures; or more generally validating test score inferences. In 2007 in Language Assessment Quarterly, Zumbo introduced the concept of Third Generation DIF. In the current article we introduce a new methodology, latent class logistic regression, for Zumbo’s Third Generation DIF, whose foundation is a novel ecological model of item responding. The ecological model and the new statistical methodology are introduced, and a proof-of-concept is provided, in the context of an example of an international reading test focusing on DIF due to testing language. The new DIF framework is described and contrasted with other methods, Mplus code is provided, and the new method is shown to have potential for application in assessment.