Identification Of Differential Item Functioning Research Articles

Comparison of Methods for Identifying Differential Step Functioning with Polytomous Item Response Data

ABSTRACT Much research has been devoted to identification of differential item functioning (DIF), which occurs when the item responses for individuals from two groups differ after they are conditioned on the latent trait being measured by the scale. There has been less work examining differential step functioning (DSF), which is present for polytomous items when the conditional likelihood of responses to specific categories differ between groups. DSF impacts estimation of the measured trait and reduces the effectiveness of standard DIF detection methods. The purpose of this simulation study was to extend upon earlier work by comparing several methods for detecting the presence of DSF in polytomous items, including an approach based on the lasso estimation of the generalized partial credit model. Results show that the lasso GPCM technique controlled the Type I error rate while yielding power rates somewhat lower than logistic regression and the MIMIC model, which were not able to control the Type I error rate in some conditions. An empirical example is also presented, and implications of this study for practice are discussed.

Measuring autism in males and females with a differential item functioning approach: Results from a nation-wide population-based study

Existing screening instruments for Autism Spectrum Disorder (ASD) might be prone to detect a male manifestation of ASD. Here, we examined the 17 items from the ASD domain in the Autism–Tics, ADHD and other Comorbidities inventory (A–TAC) for Differential Item Functioning (DIF). Data were obtained from the Child and Adolescent Twin Study in Sweden (CATSS) in which parents have responded to the A–TAC. Information regarding a registered diagnosis of ASD were retrieved from the National Patient Register. The cohort was divided into a developmental sample for evaluation of DIF, and a validation sample for examination of the diagnostic accuracy of the total ASD domain, and a novel male and female short form. Our main finding included the identification of DIF for six items, three favouring males and three favouring females. The full, 17 item, ASD domain and the male and female short form showed excellent ability to capture ASD diagnoses in both males and females up to the age of nine years. The full ASD domain in A–TAC is psychometrically largely equivalent across sex and the limited differences between males and females diminish the need for a sex-specific scoring when utilizing the 17 item total score.

Identification of differential item functioning by race and ethnicity in the Childhood Trauma Questionnaire

BackgroundThe accurate assessment of childhood maltreatment (CM) is important in medical and mental health settings given its association to adverse psychological and physical outcomes. Reliable and valid assessment of CM is also of critical importance to research. Due to the potential of measurement bias when comparing CM across racial and ethnic groups, invariant measurement is an important psychometric property of such screening tools. ObjectiveIn this study, differential item function (DIF) by race and ethnicity was tested. Uniform DIF refers to the influence of bias on scores across all levels of childhood maltreatment, and non-uniform DIF refers to bias in favor of one group. MethodParticipants were N=1,319 women and men (Mage=36.77, SDage=10.37) who completed the Child Trauma Questionnaire-Short Form; 42.7% were women, 57.3% were male; 58.9% were White-American, 22.1% Black-American, and 8.0% as other; 26.3% were Hispanic. ResultsUsing empirical thresholds, non-uniform DIF was identified in five items by race, and no items by ethnicity. ConclusionsUniform DIF is less problematic given that mathematical corrections can be made to adjust scores for DIF. However, non-uniform DIF can usually only be corrected by removing the DIF items from the scale. Further methodological research is needed to minimize measurement bias to effectively assess racially diverse populations.

Impact of unbalanced DIF item proportions on group-specific DIF identification

Differences in type I error and power rates for majority and minority groups are investigated when differential item functioning (DIF) contamination in a test is unbalanced. Typically, type I error and power rates are aggregated across groups, however cumulative results can be misleading if subgroups are affected differently by study conditions. With unbalanced DIF contamination, type I error and power rates are reduced for groups with more DIF items favoring them, and increased for groups with less DIF contamination. Even when aggregated impacts appear small, differing subgroup impacts can result in a larger proportional bias than in the original data.

Item-focussed Trees for the Identification of Items in Differential Item Functioning.

A novel method for the identification of differential item functioning (DIF) by means of recursive partitioning techniques is proposed. We assume an extension of the Rasch model that allows for DIF being induced by an arbitrary number of covariates for each item. Recursive partitioning on the item level results in one tree for each item and leads to simultaneous selection of items and variables that induce DIF. For each item, it is possible to detect groups of subjects with different item difficulties, defined by combinations of characteristics that are not pre-specified. The way a DIF item is determined by covariates is visualized in a small tree and therefore easily accessible. An algorithm is proposed that is based on permutation tests. Various simulation studies, including the comparison with traditional approaches to identify items with DIF, show the applicability and the competitive performance of the method. Two applications illustrate the usefulness and the advantages of the new method.

Detection of differential item functioning in Rasch models by boosting techniques.

Methods for the identification of differential item functioning (DIF) in Rasch models are typically restricted to the case of two subgroups. A boosting algorithm is proposed that is able to handle the more general setting where DIF can be induced by several covariates at the same time. The covariates can be both continuous and (multi-)categorical, and interactions between covariates can also be considered. The method works for a general parametric model for DIF in Rasch models. Since the boosting algorithm selects variables automatically, it is able to detect the items which induce DIF. It is demonstrated that boosting competes well with traditional methods in the case of subgroups. The method is illustrated by an extensive simulation study and an application to real data.

Inconsistencies in DIF Detection for Sub-Groups in Heterogeneous Language Groups

Diversity and heterogeneity among language groups have been well documented. Yet most fairness research that focuses on measurement comparability considers linguistic minority students such as English language learners (ELLs) or Francophone students living in minority contexts in Canada as a single group. Our focus in this research is to examine the degree to which measurement comparability, as indicated by differential item functioning (DIF), is consistent for sub-groups among linguistic minority Francophone students in Canada. The findings suggest that the linguistic minority Francophone students who speak French at home and those who do not speak French at home should not be grouped together for investigating measurement comparability or for examining performance gaps. We identified a great degree of differences in DIF identification with a consistency of 7–10% in DIF identification in the separate analyses for the two groups. The findings highlight methodological problems with investigating fairness for diverse linguistic groups that are treated as a single group.

Type I Error Inflation in DIF Identification With Mantel–Haenszel

It is known that sum score-based methods for the identification of differential item functioning (DIF), such as the Mantel–Haenszel (MH) approach, can be affected by Type I error inflation in the absence of any DIF effect. This may happen when the items differ in discrimination and when there is item impact. On the other hand, outlier DIF methods have been developed that are robust against this Type I error inflation, although they are still based on the MH DIF statistic. The present article gives an explanation for why the common MH method is indeed vulnerable to the inflation effect whereas the outlier DIF versions are not. In a simulation study, we were able to produce the Type I error inflation by inducing item impact and item differences in discrimination. At the same time and in parallel with the Type I error inflation, the dispersion of the DIF statistic across items was increased. As expected, the outlier DIF methods did not seem sensitive to impact and differences in item discrimination.

A Penalty Approach to Differential Item Functioning in Rasch Models

A new diagnostic tool for the identification of differential item functioning (DIF) is proposed. Classical approaches to DIF allow to consider only few subpopulations like ethnic groups when investigating if the solution of items depends on the membership to a subpopulation. We propose an explicit model for differential item functioning that includes a set of variables, containing metric as well as categorical components, as potential candidates for inducing DIF. The ability to include a set of covariates entails that the model contains a large number of parameters. Regularized estimators, in particular penalized maximum likelihood estimators, are used to solve the estimation problem and to identify the items that induce DIF. It is shown that the method is able to detect items with DIF. Simulations and two applications demonstrate the applicability of the method.

Analysis of Sources of Latent Class Differential Item Functioning in International Assessments

In this study, we investigated differential item functioning (DIF) and its sources using a latent class (LC) modeling approach. Potential sources of LC DIF related to instruction and teacher-related variables were investigated using substantive and three statistical approaches: descriptive discriminant function, multinomial logistic regression, and multilevel multinomial logistic regression analyses. Results revealed that differential response patterns, as indicated by identification of LCs, were most strongly associated with student achievement levels and teacher-related variables rather than manifest characteristics such as gender, test language, and country, which are the focus of typical measurement comparability research. Findings from this study have important implications for measurement comparability and validity research. Evidence of within-group heterogeneity in the test data structure suggests that the identification of DIF and its sources may not apply to all examinees in the group and that measurement incomparability may be greater among groups that are not defined by manifest variables such as gender and ethnicity. Results suggest that alternative variables that may be more closely related to the investigated construct should be examined when conducting measurement comparability research.

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

Even if national and international assessments are designed to be comparable, subsequent psychometric analyses often reveal differential item functioning (DIF). Central to achieving comparability is to examine the presence of DIF, and if DIF is found, to investigate its sources to ensure differentially functioning items that do not lead to bias. In this study, sources of DIF were examined using think-aloud protocols. The think-aloud protocols of expert reviewers were conducted for comparing the English and French versions of 40 items previously identified as DIF (N = 20) and non-DIF (N = 20). Three highly trained and experienced experts in verifying and accepting/rejecting multi-lingual versions of curriculum and testing materials for government purposes participated in this study. Although there is a considerable amount of agreement in the identification of differentially functioning items, experts do not consistently identify and distinguish DIF and non-DIF items. Our analyses of the think-aloud protocols identified particular linguistic, general pedagogical, content-related, and cognitive factors related to sources of DIF. Implications are provided for the process of arriving at the identification of DIF, prior to the actual administration of tests at national and international levels.

The Langer-Improved Wald Test for DIF Testing With Multiple Groups

Differential item functioning (DIF) occurs when the probability of responding in a particular category to an item differs for members of different groups who are matched on the construct being measured. The identification of DIF is important for valid measurement. This research evaluates an improved version of Lord’s χ2 Wald test for comparing item response model parameter estimates between two groups. The improved version uses better approaches for computation of the covariance matrix and equating the item parameters across groups. There are two equating algorithms implemented in IRTPro and flexMIRT software: Wald-1 (one-stage) and Wald-2 (two-stage), only one of which has been studied in simulations before. The present study evaluates for the first time the Wald-1 algorithm and Wald-1 and Wald-2 for three groups simultaneously. A comparison to two-group IRT-LR-DIF is included. Results indicate that Wald-1 performs very well and is recommended, whereas Type I error is extremely inflated for Wald-2. Performance of IRT-LR-DIF and Wald-1 was similar, even for three groups.

Identification of Differential Item Functioning in Multiple-Group Settings: A Multivariate Outlier Detection Approach

We focus on the identification of differential item functioning (DIF) when more than two groups of examinees are considered. We propose to consider items as elements of a multivariate space, where DIF items are outlying elements. Following this approach, the situation of multiple groups is a quite natural case. A robust statistics technique is proposed to identify DIF items as outliers in the multivariate space. For low dimensionalities, up to 2–3 groups, a simple graphical tool is derived. We illustrate our approach with a reanalysis of data from Kim, Cohen, and Park (1995) on using calculators for a mathematics test.

A Robust Outlier Approach to Prevent Type I Error Inflation in Differential Item Functioning

The identification of differential item functioning (DIF) is often performed by means of statistical approaches that consider the raw scores as proxies for the ability trait level. One of the most popular approaches, the Mantel–Haenszel (MH) method, belongs to this category. However, replacing the ability level by the simple raw score is a source of potential Type I error inflation, not only in the presence of DIF but also when DIF is absent and in the presence of impact. The purpose of this article is to present an alternative statistical inference approach based on the same measure of DIF but such that the Type I error inflation is prevented. The key notion is that for DIF items, the measure has an outlying value that can be identified as such with inference tools from robust statistics. Although we use the MH log odds ratio as a statistic, the inference is different. A simulation study is performed to compare the robust statistical inference with the classical inference method, both based on the MH statistic. As expected, the Type I error rate inflation is avoided with the robust approach, although the power of the two methods is similar.

Identification of Differential Item Functioning in Assessment Booklet Designs With Structurally Missing Data

The Mantel—Haenszel procedure is a popular technique for determining items that may exhibit differential item functioning (DIF). Numerous studies have focused on the strengths and weaknesses of this procedure, but few have focused the performance of the Mantel—Haenszel method when structurally missing data are present as a result of test booklet design. The current research compares the Type I error rate, power rate, and accuracy of DIF statistics when using the Mantel—Haenszel methods to detect DIF in three different test booklet designs that produce sparse data. Data were simulated for three different total sample sizes, fully crossed with three different test lengths, and two focal to reference group ratios. In general, the Mantel—Haenszel procedure was not adversely affected by sparseness of data under any of the booklet designs considered when sample size was adequately large. As sample sizes decrease, both the amount of sparseness and features of the booklet design may influence the ability to detect DIF.

The Use of Multilevel Item Response Theory Modeling in Test Development

Both item response theory(IRT) and a multilevel model are used in a variety of social science research applications.The use of IRT allows researchers to link the observed categorical responses provided by students with an underlying unobservable trait,such as ability or attitude.A multilevel model allows the natural multilevel structure that is widely present in social science data to be represented formally in data analysis.In some cases,researchers may wish to study the effects of the covariates on the latent trait.These covariates may include information pertaining to responses as well as contextual information.Traditionally,manifest variables are used in a multilevel analysis as fixed and known entities.An important deficiency is that the measurement error associated with the test scores is ignored.In general,the use of unreliable test scores leads to a biased estimation of the regression coefficients;consequently,the resulting statistical inference can be rather misleading. In this paper,a multilevel IRT model is presented where some of the variables cannot be observed directly but are measured using tests or questionnaires.A two-level model can be defined using a two-level formulation in which level 1 is the item-level model and level 2 is the person-level model.In such a model,the latent ability and item parameters can be estimated simultaneously.Further,we can consider the effects of the person-level covariates on the person-level ability.Similar to a traditional two-level model,the two-level IRT model can be expanded to a three-level model,which includes a higher site level.The threelevel IRT model proposed in this paper yields the additional benefit of being able to accommodate data that are collected in a hierarchical setting.This expansion of multilevel IRT models to three levels allows not only the dependency typically found in hierarchical data to be accommodated but also the estimation of latent traits at different levels as well as the estimation of relationships between predictor variables and latent traits at different levels. The purpose of this paper is to provide both a theoretic description and a practical application of the multilevel IRT model.First,we demonstrate the algebraic equivalence between the parameterizations of an IRT model and the multilevel IRT model.Second,we illustrate the method,using an application that involves students' achievements on a mathematics test and test results regarding the characteristics of students and schools.Third,we expand the multilevel IRT model to the identification of differential item functioning(DIF).Finally,we present a discussion on the advantages and disadvantages of using multilevel IRT models in applied research and provide the scope for future research.

Item–trait association, scale multidimensionality, and differential item functioning identification in personality assessment

This report documents relationships between differential item functioning (DIF) identification and: (1) item–trait association, and (2) scale multidimensionality in personality assessment. Applying [Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, ON: Directorate of Human Resources Research and Evaluation, Department of National Defense.] logistic regression model, DIF effect size is found to become increasingly inflated as investigated item associations with trait scores decrease. Similar patterns were noted for the influence of scale multidimensionality on DIF identification. Individuals who investigate DIF in personality assessment applications are provided with estimates regarding the impact of the magnitude of item and trait association and scale multidimensionality on DIF occurrence and effect size. The results emphasize the importance of excluding investigated items in focal trait identification prior to conducting DIF analyses and reporting item and scale psychometric properties in DIF reports.

O funcionamento diferencial dos itens

A dificuldade de um item deveria ser uma propriedade independentemente de uma população em particular, da mesma forma que a habilidade de uma pessoa deveria ser uma característica sem referência a qualquer conjunto particular de itens. Nesse contexto, a análise do funcionamento diferencial dos itens (DIF) busca detectar os itens cuja probabilidade de acertos difere entre distintos grupos, cujas pessoas possuem o mesmo nível de habilidade na variável medida. Para a análise dessa questão são discutidos principalmente, os tipos de abordagem para sua detecção e as propostas de procedimentos para identificação de DIF. Além disso, é comentado que o modelo de Rasch fornece uma estrutura que permite a comparação dos resultados observados com os previstos, avaliando a probabilidade do resultado observado, equacionando problemas não solucionados pelos outros modelos.

DIF Identification Using HGLM for Polytomous Items

The relationship between the hierarchical generalized linear model (HGLM) and item response theory (IRT) models has been demonstrated for dichotomous items. The current study demonstrated the use of the HGLM for polytomous items (termed PHGLM) for identification of differential item functioning (DIF). First, the algebraic equivalence between parameterizations of a constrained form of Muraki's rating scale model and PHGLM was demonstrated. Next, as a precursor to using the PHGLM for DIF identification, a simulation was conducted to compare estimation of item and person latent trait parameters using HLM 5 and PARSCALE software. The resulting PHGLM and IRT parameter estimates were comparable. Last, a simulation was conducted that compared the performance of PHGLM with that of the generalized Mantel-Haenszel statistic for DIF identification. Both approaches performed similarly. Benefits and limitations of using PHGLM for DIF identification are presented. Index terms: hierarchical linear modeling, item response theory, polytomous items, parameter estimation, differential item functioning, attitude measurement.

Identification of Differential Item Functioning Using Item Response Theory and the Likelihood-Based Model Comparison Approach

An important part of examining the adequacy of measures for use in ethnically diverse populations is the evaluation of differential item functioning (DIF) among subpopulations such as those administered the measure in different languages. A number of methods exist for this purpose. The objective of this study was to introduce and demonstrate the identification of DIF using item response theory (IRT) and the likelihood-based model comparison approach. Data come from a sample of community-residing elderly who were part of a dementia case registry. A total of 1578 participants were administered either an English (n = 913) or Spanish (n = 665) version of the 21-item Mini-Mental State Examination. IRT was used to identify language DIF in these items with the likelihood-based model comparison approach. : Fourteen of the 21 items exhibited significant DIF according to language of administration. However, because the direction of the identified DIF was not consistent for one language version over the other, the impact at the scale level was negligible. IRT and the likelihood-based model comparison approach comprise a powerful tool for DIF detection that can aid in the development, refinement, and evaluation of measures for use in ethnically diverse populations.