Differential Item Functioning Effect Size Research Articles

Measuring Food Security among American Indian and Alaska Native Adults: Validity Evidence Supports the Use of the US Department of Agriculture Module

BackgroundInequities in access, availability, and affordability of nutritious foods produced by settler colonialism contribute to high rates of food insecurity among American Indian and Alaska Native (AI/AN) households. Efforts to understand the influences of food security programming among AI/AN individuals in the United States are constrained by the absence of validity evidence for food security assessments for this population. ObjectiveThis study assessed whether AI/AN adult responses on the Food Security Survey Module provide an accurate assessment of food security prevalence, especially when compared with other racial and ethnic groups. DesignA correlational design with the cross-sectional 2019 National Health Interview Survey was used to address the research objective. Participants and settingThe 2019 National Health Interview Survey contains a sample (N = 30,052) representative of the resident civilian noninstitutionalized population. Main outcome measuresThe primary outcome was food security, as characterized by the 10-item US Department of Agriculture Adult Food Security Survey Module. The module evaluates whether insufficient finances result in perceived food shortages and a reduction in the quantity and/or quality of food intake during the prior 30 days. Statistical analyses performedData were analyzed by racial and ethnic subsamples to assess scale dimensionality (confirmatory factor analysis), Item Response Theory item analysis, differential item functioning, and external validity (χ2 tests). ResultsResults supported the use of the 10-item module for racial and ethnic groups. However, differential item functioning effect sizes exceeded criteria for the Asian, AI/AN, and Hispanic respondents when compared with White respondents. Food security was not significantly related to all expected correlates in the AI/AN subsample. ConclusionsCompelling evidence is presented for validity of the FSSM scores in determining food security status of AI/AN adults. Qualitative inquiry that explores how culture influences the way food security is conceptualized and experienced is warranted.

Effect Sizes for Estimating Differential Item Functioning Influence at the Test Level

Differential item functioning (DIF) is a critical step in providing evidence to support a scoring inference in building a validity argument for a psychological or educational assessment. Effect sizes can assist in understanding the accumulation of DIF at the test score level. The current simulation study investigated the performance of several proposed effect size measures under a variety of conditions. Conditions under study included varied sample sizes, DIF effect sizes, the proportion of items with DIF, and the type of DIF (additive vs. non-additive). DIF effect sizes under study included sDTF%, uDTF%, τ^w2, d, R¯Δ2, IDIF2*, and S−DIF−V. The results of this study suggest that across study conditions, τ^w2, IDIF2*, and d were consistently the most accurate measures of the DIF effects. The effect sizes were also estimated in an empirical example. Recommendations and implications for practice are discussed.

Investigating the Distractors to Explain DIF Effects Across Gender in Large-Scale Tests With Non-Linear Logistic Regression Models

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

A Comparative Study of Mathematics Self-Beliefs between Students in Shanghai-China and the USA

According to PISA 2012, mathematics self-beliefs involve three dimensions: mathematics self-efficacy, mathematics self-concept and mathematics anxiety. This study utilized the multiple indicators, multiple causes (MIMIC) method, an approach for detecting whether there is a lack of measurement invariance or differential item functioning (DIF), to deal with multiple covariates, multiple dimensions, and ordered categorical variables with threshold structures to study DIF among mathematics self-beliefs items across Shanghai-China (N = 5177) and the USA (N = 4978) from Program for International Student Assessment (PISA) 2012. The MIMIC approach with mediators was also applied in the study, which helped to detect variables that could account for meaningful partial or complete DIF effects. Confirmatory factor analysis (CFA), a statistic method that can verify the number of latent traits of the dataset, indicated that the three-factor structure worked for the data. Both robust weighted least square (WLSMV) estimator and robust maximum likelihood (MLR) estimator were used in the parameter estimation to identify items with DIF and quantify the DIF effect size. It was found in the study with MIMIC method that in-school mathematics class periods and out-of-school study hours in PISA 2012 had partial effects on most items with meaningful DIF effects.

Measurement Invariance and Differential Item Functioning Across Gender Within a Latent Class Analysis Framework: Evidence From a High-Stakes Test for University Admission in Saudi Arabia.

The main aim of the present study was to investigate the presence of Differential Item Functioning (DIF) using a latent class (LC) analysis approach. Particularly, we examined potential sources of DIF in relation to gender. Data came from 6,265 Saudi Arabia students, who completed a high-stakes standardized admission test for university entrance. The results from a Latent Class Analysis (LCA) revealed a three-class solution (i.e., high, average, and low scorers). Then, to better understand the nature of the emerging classes and the characteristics of the people who comprise them, we applied a new stepwise approach, using the Multiple Indicator Multiple Causes (MIMIC) model. The model identified both uniform and non-uniform DIF effects for several items across all scales of the test, although, for the majority of them, the DIF effect sizes were negligible. Findings from this study have important implications for both measurement quality and interpretation of the results. Particularly, results showed that gender is a potential source of DIF for latent class indicators; thus, it is important to include those direct effects in the latent class regression model, to obtain unbiased estimates not only for the measurement parameters but also of the structural parameters. Ignoring these effects might lead to misspecification of the latent classes in terms of both the size and the characteristics of each class, which in turn, could lead to misinterpretations of the obtained latent class results. Implications of the results for practice are discussed.

The Power of Crossing SIBTEST.

Crossing SIBTEST or CSIB is designed to detect crossing differential item functioning (DIF) as well as unidirectional DIF. A theoretical formula for the power of CSIB is derived based on the asymptotic distribution of the test statistic under the null and alternative hypotheses. The derived power formula provides insights on the factors that influence the CSIB power, including DIF effect size, standard error, and sample size. The power formula and those influencing factors are further discussed in the context of the item response theory (IRT) three parameter logistic model (3PL) model. Simulation results show the consistency between the theoretical power and the observed rejection rate. The power of CSIB is compared with the unidirectional SIBTEST in theory and through simulation.

Differential Item Functioning (DIF) in composite health measurement scale: Recommendations for characterizing DIF with meaningful consequences within the Rasch model framework.

ObjectiveThe aims were to review practices concerning Differential Item Functioning (DIF) detection in composite measurement scales, particularly those used in health research, and to provide guidance on how to proceed if statistically significant DIF is detected.MethodsThis work specifically addressed the Rasch model which is the subject of growing interest in the field of health owing to its particularly advantageous properties. There were three steps: 1) Literature review to describe current practices; 2) Simulation study to determine under which conditions encountered in health research studies can erroneous conclusions be drawn from group comparisons when a scale is affected by DIF but which is not considered; 3) Based on steps 1 and 2, formulation of recommendations that were subsequently reviewed by leading internationally recognized experts.ResultsFour key recommendations were formulated to help researchers to determine whether statistically significant DIF is meaningful in practice, according to the kind of DIF (uniform or non-uniform) and the DIF effect size.ConclusionThis work provides the first recommendations on how to deal in practice with the presence of DIF in composite measurement scales used in health research studies.

Item Response Theory With Covariates (IRT-C): Assessing Item Recovery and Differential Item Functioning for the Three-Parameter Logistic Model.

In large-scale testing, the use of multigroup approaches is limited for assessing differential item functioning (DIF) across multiple variables as DIF is examined for each variable separately. In contrast, the item response theory with covariate (IRT-C) procedure can be used to examine DIF across multiple variables (covariates) simultaneously. To assess the utility of the IRT-C procedure, we conducted a simulation study. Using SAT data for realistic parameters, uniform DIF on three covariates were simulated: gender (dichotomous), race/ethnicity (categorical), and income (continuous). Simulations were conducted across several conditions: two test lengths (14 items, 21 items), four sample sizes (5,000, 10,000, 20,000, 40,000), and two DIF effect sizes (medium, large). It was found that the IRT-C procedure could accurately recover the latent means and the three-parameter logistic model parameters well with a substantial sample size of 20,000. There was good control of Type I error rates to the nominal rates across the sample sizes. Good power to detect DIF across all covariates (>.80) was observed when the sample size was 20,000 for large DIF effect size and 40,000 for medium DIF effect size. Practical implications for the use of the IRT-C procedure are discussed.

Evaluation of Two Types of Differential Item Functioning in Factor Mixture Models With Binary Outcomes

Conventional differential item functioning (DIF) detection methods (e.g., the Mantel–Haenszel test) can be used to detect DIF only across observed groups, such as gender or ethnicity. However, research has found that DIF is not typically fully explained by an observed variable. True sources of DIF may include unobserved, latent variables, such as personality or response patterns. The factor mixture model (FMM) is designed to detect unobserved sources of heterogeneity in factor models. The current study investigated use of the FMM for detecting between-class latent DIF and class-specific observed DIF. Factors that were manipulated included the DIF effect size and the latent class probabilities. The performance of model fit indices (Akaike information criterion [AIC], Bayesian information criterion [BIC], sample size–adjusted BIC, and consistent AIC) were assessed for their detection of the correct DIF model. The recovery of DIF parameters was also assessed. Results indicated that use of FMMs with binary outcomes performed well in terms of the DIF detection and for recovery of large DIF effects. When class probabilities were unequal with small DIF effects, performance decreased for fit indices, power, and the recovery of DIF effects compared with equal class probability conditions. Inflated Type I errors were found for non-DIF items across simulation conditions. Results and future research directions for applied and methodological are discussed.

Detecting Differential Item Functioning of Polytomous Items for an Ideal Point Response Process

There has been growing use of ideal point models to develop scales measuring important psychological constructs. For meaningful comparisons across groups, it is important to identify items on such scales that exhibit differential item functioning (DIF). In this study, the authors examined several methods for assessing DIF on polytomous items generated by an ideal point process. Two paradigms (i.e., null hypothesis significance testing [NHST] and effect size quantification) were utilized, and three test statistics (i.e., the log-likelihood ratio [LR], the Akaike information criterion [AIC], and Lord’s chi-square) and two approaches to DIF testing (i.e., the constrained and free baseline methods) were evaluated. In addition, the authors investigated three levels of impact. The results revealed that DIF effect sizes were moderately large for the .50 uniform DIF conditions and small for nonuniform DIF; moreover, the LR test in general yielded the best results. When there was small to moderate impact, the free baseline approach combined with an item linking implementation produced the most satisfactory results.

Equated Pooled Booklet Method in DIF Testing

Differential item functioning (DIF) analysis is an important step in the data analysis of large-scale testing programs. Nowadays, many such programs endorse matrix sampling designs to reduce the load on examinees, such as the balanced incomplete block (BIB) design. These designs pose challenges to the traditional DIF analysis methods. For example, as difficulty levels often vary across booklets, examinees with same booklet scores may be disparate in ability. Consequently, DIF procedures based on matching total scores at the booklet level may cause misplacement of examinees and inflation in measurement errors. Therefore, modification to traditional DIF procedures to better accommodate the BIB design becomes important. This article introduces modification of current simultaneous item bias test (SIBTEST) procedure for the DIF analysis method when multiple booklets are used. More specifically, examinees will be pooled across booklets, and the matching will be based on transformed booklet scores after common block equating/linking. Simulations are conducted to compare the performance of this new method, the equated pooled booklet method against that of the current pooled booklet method, in terms of both Type I error control and power. Four factors are considered in the simulation—the DIF effect size, item difficulty, impact, and the length of common block. Results show that the equated pooled booklet method in general improves power while keeping Type I error under control. The advantage of the new method is the most pronounced when the traditional method struggles, for example, when the item is difficult or there is impact.

Differential Item Functioning classification for polytomously scored items

Differential item functioning (DIF) refers to a difference in the way a test item functions for comparable groups of test takers. When there are DIF items in the test, it is necessary to make decisions concerning items revision or removal which cannot be based only on the outcome of DIF test statistic; a interpretable measure of the amount of DIF can help in decisions regarding DIF items. Due to the increasing use of polytomous item formats, this paper reviews the systems of DIF classification for polytomously scored items. Keywords : DIF effect size, polytomous items; Liu-Agresti estimate; Signed Area; Standardized Mean Difference; polytomous SIBTEST; ordinal logistic model.

The differential item functioning and structural equivalence of a nonverbal cognitive ability test for five language groups

Orientation: For a number of years, eliminating a language component in testing by using nonverbal cognitive tests has been proposed as a possible solution to the effect of groups’ languages (mother tongues or first languages) on test performance. This is particularly relevant in South Africa with its 11 official languages.Research purpose: The aim of the study was to determine the differential item functioning (DIF) and structural equivalence of a nonverbal cognitive ability test (the PiB/SpEEx Observance test [401]) for five South African language groups.Motivation for study: Cultural and language group sensitive tests can lead to unfair discrimination and is a contentious workplace issue in South Africa today. Misconceptions about psychometric testing in industry can cause tests to lose credibility if industries do not use a scientifically sound test-by-test evaluation approach.Research design, approach and method: The researcher used a quasi-experimental design and factor analytic and logistic regression techniques to meet the research aims. The study used a convenience sample drawn from industry and an educational institution.Main findings: The main findings of the study show structural equivalence of the test at a holistic level and nonsignificant DIF effect sizes for most of the comparisons that the researcher made.Practical/managerial implications: This research shows that the PIB/SpEEx Observance Test (401) is not completely language insensitive. One should see it rather as a language-reduced test when people from different language groups need testing.Contribution/value-add: The findings provide supporting evidence that nonverbal cognitive tests are plausible alternatives to verbal tests when one compares people from different language groups.

Lordif: AnRPackage for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations

Logistic regression provides a flexible framework for detecting various types of differential item functioning (DIF). Previous efforts extended the framework by using item response theory (IRT) based trait scores, and by employing an iterative process using group-specific item parameters to account for DIF in the trait scores, analogous to purification approaches used in other DIF detection frameworks. The current investigation advances the technique by developing a computational platform integrating both statistical and IRT procedures into a single program. Furthermore, a Monte Carlo simulation approach was incorporated to derive empirical criteria for various DIF statistics and effect size measures. For purposes of illustration, the procedure was applied to data from a questionnaire of anxiety symptoms for detecting DIF associated with age from the Patient-Reported Outcomes Measurement Information System.

A COMPARISON OF METHODS FOR ESTIMATING CONDITIONAL ITEM SCORE DIFFERENCES IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ASSESSMENTS

ABSTRACTThis study compared the accuracies of four differential item functioning (DIF) estimation methods, where each method makes use of only one of the following: raw data, logistic regression, loglinear models, or kernel smoothing. The major focus was on the estimation strategies' potential for estimating score‐level, conditional DIF. A secondary focus was on assessing the accuracy of strategies' overall DIF effect sizes and statistical significance tests. A real data simulation was used to evaluate the estimation strategies with 6 items representing DIF and No DIF situations, and with 4 sample size combinations for the reference and focal group data. Results showed that the logistic regression estimation strategy was the most highly recommended strategy in terms of the bias and variability of its estimates and the power of its statistical significance test. The loglinear models strategy had flexibility advantages, but these advantages only offset the greater variability of its estimates and its reduced statistical power when sample sizes were large. The kernel smoothing estimation strategy was the least accurate of the considered strategies due to estimation problems when the reference and focal groups differed in overall ability.

Item-fit evaluation in biased tests: a study under Rasch model

In this paper, the power rates and distributional properties of the Outfit, Infit, Lz, ECI2z and ECI4z statistics when they are used in tests with biased or differential item functioning (DIF) were explored. In this study, different conditions of sample size, sample size ratio focal and reference group, impact between groups, DIF effect size, and percentage of DIF items were manipulated. In addition, examinee responses were generated to simulate uniform DIF. Results suggest that item fit statistics generally detected medium percents of DIF in large samples (1000/500 or 1000/1000) only when DIF effect size was relatively high and when the mean of focal and reference group was different. Moreover, when groups had equal mean, low correct identification rates were found in the five item-fit indices. In general, the results showed adequate control of false positive rates. These findings lead to the conclusion that all indices used in this study are partially adequate fit measures for detecting biased items, mainly when impact between groups is present and sample size is large.

Impact of Missing Data on the Detection of Differential Item Functioning

This article describes the results of a simulation study to investigate the impact of missing data on the detection of differential item functioning (DIF). Specifically, it investigates how four methods for dealing with missing data (listwise deletion, zero imputation, two-way imputation, response function imputation) interact with two methods of DIF detection (Mantel-Haenszel statistic, logistic regression analysis) under three mechanisms of missingness (data missing completely at random, data missing at random, and data missing not at random) to produce over- or underestimates of the DIF effect sizes and detection rates. Results show that the interaction effects between missingness mechanism, treatment, and rate are most influential for explaining variation in bias, root mean square errors, and rejection rates. An incorrect treatment of missing data can thus lead to severe increases of Type I and Type II error rates. However, the choice between the two DIF detection methods investigated in this study is not important.

Item–trait association, scale multidimensionality, and differential item functioning identification in personality assessment

This report documents relationships between differential item functioning (DIF) identification and: (1) item–trait association, and (2) scale multidimensionality in personality assessment. Applying [Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.] logistic regression model, DIF effect size is found to become increasingly inflated as investigated item associations with trait scores decrease. Similar patterns were noted for the influence of scale multidimensionality on DIF identification. Individuals who investigate DIF in personality assessment applications are provided with estimates regarding the impact of the magnitude of item and trait association and scale multidimensionality on DIF occurrence and effect size. The results emphasize the importance of excluding investigated items in focal trait identification prior to conducting DIF analyses and reporting item and scale psychometric properties in DIF reports.

A Monte Carlo Examination of the Sensitivity of the Differential Functioning of Items and Tests Framework for Tests of Measurement Invariance With Likert Data

This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted.

DIF Detection and Effect Size Measures for Polytomously Scored Items

Data from a large‐scale performance assessment (N= 105,731) were analyzed with five differential item functioning (DIF) detection methods for polytomous items to examine the congruence among the DIF detection methods. Two different versions of the item response theory (IRT) model‐based likelihood ratio test, the logistic regression likelihood ratio test, the Mantel test, and the generalized Mantel–Haenszel test were compared. Results indicated some agreement among the five DIF detection methods. Because statistical power is a function of the sample size, the DIF detection results from extremely large data sets are not practically useful. As alternatives to the DIF detection methods, four IRT model‐based indices of standardized impact and four observed‐score indices of standardized impact for polytomous items were obtained and compared with the R2 measures of logistic regression.