Differential Item Functioning Detection Research Articles

Accuracy of DIF Estimates and Power in Unbalanced Designs Using the Mantel–Haenszel DIF Detection Procedure

This study examined how much improvement was attainable with respect to accuracy of differential item functioning (DIF) measures and DIF detection rates in the Mantel–Haenszel procedure when employing focal and reference groups with notably unbalanced sample sizes where the focal group has a fixed small sample which does not satisfy the minimum DIF sample size requirement specified by the testing programs, while the reference group sample size far exceeds the minimum requirement. Results showed equivalent or better results with such unbalanced but large samples than with some of the currently used minimum DIF sample size conditions. DIF investigation, therefore, does not necessarily need to cease when the focal group does not meet the minimum sample size requirement. Some analytic explanations and guidelines for DIF investigations with unbalanced sample sizes are also provided.

The Use of Multiple Imputation for Missing Data in Uniform DIF Analysis: Power and Type I Error Rates

Methods of uniform differential item functioning (DIF) detection have been extensively studied in the complete data case. However, less work has been done examining the performance of these methods when missing item responses are present. Research that has been done in this regard appears to indicate that treating missing item responses as incorrect can lead to inflated Type I error rates (false detection of DIF). The current study builds on this prior research by investigating the utility of multiple imputation methods for missing item responses, in conjunction with standard DIF detection techniques. Results of the study support the use of multiple imputation for dealing with missing item responses. The article concludes with a discussion of these results for multiple imputation in conjunction with other research findings supporting its use in the context of item parameter estimation with missing data.

Angoff's delta method revisited: Improving DIF detection under small samples

Most methods for detecting differential item functioning (DIF) are suitable when the sample sizes are sufficiently large to validate the null statistical distributions. There is no guarantee, however, that they will still perform adequately when there are few respondents in the focal group or in both the reference and the focal group. Angoff's delta plot is a potentially useful alternative for small-sample DIF investigation, but it suffers from an improper DIF flagging criterion. The purpose of this paper is to improve this classification rule under mild statistical assumptions. This improvement yields a modified delta plot with an adjusted DIF flagging criterion for small samples. A simulation study was conducted to compare the modified delta plot with both the classical delta plot approach and the Mantel-Haenszel method. It is concluded that the modified delta plot is consistently less conservative and more powerful than the usual delta plot, and is also less conservative and more powerful than the Mantel-Haenszel method as long as at least one group of respondents is small.

The Impact of Missing Data on the Detection of Nonuniform Differential Item Functioning

Missing information is a ubiquitous aspect of data analysis, including responses to items on cognitive and affective instruments. Although the broader statistical literature describes missing data methods, relatively little work has focused on this issue in the context of differential item functioning (DIF) detection. Such prior research has focused on uniform DIF and has identified a complicated relationship between the type and amount of missing data, along with the methods for dealing with it. The current simulation study was designed to extend this earlier work by focusing on the impact of missing data and methods for dealing with it on nonuniform DIF detection using popular tools, including logistic regression, crossing SIBTEST, and the item response theory likelihood ratio test. Results suggest that the relationships between the various factors manipulated here were complex with no one method emerging as optimal in all cases, although listwise deletion consistently produced results similar to those obtained with the complete data across simulated conditions. Implications of these results are also discussed.

Do Raven's Colored Progressive Matrices function in the same way in typical and clinical populations? Insights from the intellectual disability field

Standardized tests are used widely in comparative studies of clinical populations, either as dependent or control variables. Yet, one cannot always be sure that the test items measure the same constructs in the groups under study. In the present work, 460 participants with intellectual disability of undifferentiated etiology and 488 typical children were tested using Raven's Colored Progressive Matrices (RCPM). Data were analyzed using binomial logistic regression modeling designed to detect differential item functioning (DIF). Results showed that 12 items out of 36 function differentially between the two groups, but only 2 items exhibit at least moderate DIF. Thus, a very large majority of the items have identical discriminative power and difficulty levels across the two groups. It is concluded that RCPM can be used with confidence in studies comparing participants with and without intellectual disability. In addition, it is suggested that methods for investigating internal bias of tests used in cross-cultural, cross-linguistic or cross-gender comparisons should also be regularly employed in studies of clinical populations, particularly in the field of developmental disability, to show the absence of systematic measurement error (i.e. DIF) affecting item responses.

IMPACTO Y FUNCIONAMIENTO DIFERENCIAL DE LOS ÍTEMS RESPECTO AL GÉNERO EN EL TEST DE EFICACIA LECTORA (TECLE)

Este trabajo tiene como objetivo estudiar el posible funcionamiento diferencial de los ítems (FDI) que componen la Prueba de Eficacia Lectora (TECLE). Se utilizó el estadístico de Mantel-Haenszelt (MH) como procedimiento de detección del FDI para evaluar si los ítems del TECLE funcionan de forma distinta en grupos igualados en función del género. La muestra está conformada por 1159 alumnos de educación primaria privada con edades comprendidas entre 9 y 12 años (50,5% varones y 49,5% niñas). Los resultados del análisis del FDI encontraron que según el estadístico MH que el porcentaje de ítems con FDI es del 6,25%. Independientemente de que puedan existir diferencias reales en las habilidades de eficacia lectora entre niñas y niños en cuatro del los 64 ítems del TECLE, no es preciso realizar una baremación por sexo. Habría que considerar continuar con los estudios con jueces expertos en el tema para despejar la sospecha de FDI.

Differential Item Functioning Detection With Latent Classes

There is a long history of differential item functioning (DIF) detection methods for known, manifest grouping variables, such as sex or ethnicity. But if the experiences or cognitive processes leading to DIF are not perfectly correlated with the manifest groups, it would be more informative to uncover the latent groups underlying DIF. The use of item response theory (IRT) mixture models to detect latent groups and estimate the DIF caused by these latent groups has been explored/interpreted with real data sets, but the accuracy of model estimation has not been thoroughly explored. The purpose of this simulation research was to assess the accuracy of the recovery of classes, item parameters, and DIF effects in contexts where relatively small clusters of items showed DIF. Overall, the results from the study reveal that the use of IRT mixture models for latent DIF detection may be problematic. Class membership recovery was poor in all conditions tested. Discrimination parameters were estimated well for the invariant items, as well as for the DIF items when there was no group impact. But when there was group impact, discriminations for the DIF items were positively biased. When there was no group impact, DIF effect estimates tended to be positively biased. In general, having fewer items was associated with more biased estimates and larger standard errors.

An Examination of Power and Type I Errors for Two Differential Item Functioning Indices Using the Graded Response Model

This study examined two methods for detecting differential item functioning (DIF): Raju, van der Linden, and Fleer’s 1995 differential functioning of items and tests (DFIT) procedure and Thissen, Steinberg, and Wainer’s 1988 likelihood ratio test (LRT). The major research questions concerned which test provides the best balance of Type I errors and power and if the tests differ in terms of detecting different types of DIF. Monte Carlo simulations were conducted to address these questions. Equal and unequal sample size conditions were fully crossed with test lengths of 10 and 20 items. In addition, α and β parameters were manipulated in order to simulate DIF. Findings indicate that DFIT and LRT both had acceptable Type I error rates when sample sizes were equal but that DFIT produced too many Type I errors when sample sizes were unequal. Overall, the LRT exhibited greater power to detect both α and β parameter DIF than did DFIT. However, DFIT was more powerful than LRT when the last two β parameters had DIF as opposed to when the extreme β parameters had DIF.

Formulating the Rasch Differential Item Functioning Model Under the Marginal Maximum Likelihood Estimation Context and Its Comparison With Mantel–Haenszel Procedure in Short Test and Small Sample Conditions

This study elaborates the Rasch differential item functioning (DIF) model formulation under the marginal maximum likelihood estimation context. Also, the Rasch DIF model performance was examined and compared with the Mantel–Haenszel (MH) procedure in small sample and short test length conditions through simulations. The theoretically known relationship of the DIF estimators between the Rasch DIF model and the MH procedure was confirmed. In general, the MH method showed a conservative tendency for DIF detection rates compared with the Rasch DIF model approach. When there is DIF, the z test (when the standard error of the DIF estimator is estimated properly) and the likelihood ratio test in the Rasch DIF model approach showed higher DIF detection rates than the MH chi-square test for sample sizes of 100 to 300 per group and test lengths ranging from 4 to 39. In addition, this study discusses proposed Rasch DIF classification rules that accommodate statistical inference on the direction of DIF.

Cross-cultural adaptation and psychometric properties of the Brazilian-Portuguese version of the VSP-A (Vécu et Santé Perçue de l'Adolescent), a health-related quality of life (HRQoL) instrument for adolescents, in a healthy Brazilian population

BackgroundHealth-related quality of life (HRQoL) assessment, encompassing the adolescents' perceptions of their mental, physical, and social health and well-being is increasingly considered an important outcome to be used to identify population health needs and to provide targeted medical care. Although validated instruments are essential for accurately assessing HRQoL outcomes, there are few cross-culturally adapted tools for use in Brazil, and none designed exclusively for use among adolescents. The Vécu et Santé Perçue de l'Adolescent (VSP-A) is a generic, multidimensional self-reported instrument originally developed and validated in France that evaluates HRQoL of ill and healthy adolescents.PurposeTo cross-culturally adapt and validate the Brazilian-Portuguese version of the VSP-A, a generic HRQoL measure for adolescents originally developed in France.MethodsThe VSP-A was translated following a well-validated forward-backward process leading to the Brazilian version. The psychometric evaluation was conducted in a sample of 446 adolescents (14-18 years) attending 2 public high schools of São Gonçalo City. The adolescents self-reported the Brazilian VSP-A, the validated Psychosomatic Symptom Checklist and socio-demographic information. A retest evaluation was carried out on a sub-sample (n = 195) at a two-week interval.The internal construct validity was assessed through confirmatory factor analysis (CFA), multi-trait scaling analyses, Rasch analysis evaluating unidimensionality of each scale and Cronbach's alpha coefficients. The reproducibility was evaluated by intra-class correlation coefficients (ICC). Zumbo's ordinal logistic regression analysis was used to detect differential item functioning (DIF) between the Brazilian and the French items. External construct validity was investigated testing expected differences between groups using one-way analysis of variance (ANOVA), Mann-Whitney tests and the univariate general regression linear model.ResultsCFA showed an acceptable fit (RMSEA=0.05; CFI=0.93); 94% of scaling success was found for item-internal consistency and 98% for item discriminant validity. The items showed good fit to the Rasch model except 3 items with an INFIT at the upper threshold. Cronbach's Alpha ranged from 0.60 to 0.85. Test-retest reliability was moderate to good (ICC=0.55-0.82). DIF was evidenced in 4 out of 36 items. Expected patterns of differences were confirmed with significantly lower physical, psychological well being and vitality reported by symptomatic adolescents.ConclusionsAlthough DIF in few items and responsiveness must be further explored, the Brazilian version of VSP-A demonstrated an acceptable validity and reliability in adolescents attending school and might serve as a starting point for more specific clinical investigations.

Lordif: AnRPackage for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations

Logistic regression provides a flexible framework for detecting various types of differential item functioning (DIF). Previous efforts extended the framework by using item response theory (IRT) based trait scores, and by employing an iterative process using group-specific item parameters to account for DIF in the trait scores, analogous to purification approaches used in other DIF detection frameworks. The current investigation advances the technique by developing a computational platform integrating both statistical and IRT procedures into a single program. Furthermore, a Monte Carlo simulation approach was incorporated to derive empirical criteria for various DIF statistics and effect size measures. For purposes of illustration, the procedure was applied to data from a questionnaire of anxiety symptoms for detecting DIF associated with age from the Patient-Reported Outcomes Measurement Information System.

Differential Item Functioning Analysis of the Science and Mathematics Items in the University Entrance Examinations in Turkey

This study is aimed to detect differential item functioning (DIF) items across gender groups, analyze item content for the possible sources of DIF, and eventually investigate the effect of DIF items on the criterion-related validity of the test scores in the quantitative section of the university entrance examination (UEE) in Turkey. The reason for DIF items in the UEE basically comes from subject matter related factors, cognitive skills measured, and item format characteristics. It seems that higher order cognitive skills and figural or graphical representations used in item content are the two sources of DIF for favoring male students, whereas routine algorithmic calculations could produce DIF against males. Among the factors considered, cognitive skills assessed by items seem the most effective factor in producing gender DIF. However, DIF items do not create a threat to the criterion-related validity of the quantitative section of the UEE. Consideration of the DIF items and gender differences as presented here serves to underscore the need to consider gender differences in item selection on any measure that is used for a similar purpose.

RIM: A Random Item Mixture Model to Detect Differential Item Functioning

In this paper we present a new methodology for detecting differential item functioning (DIF). We introduce a DIF model, called the random item mixture (RIM), that is based on a Rasch model with random item difficulties (besides the common random person abilities). In addition, a mixture model is assumed for the item difficulties such that the items may belong to one of two classes: a DIF or a non‐DIF class. The crucial difference between the DIF class and the non‐DIF class is that the item difficulties in the DIF class may differ according to the observed person groups while they are equal across the person groups for the items from the non‐DIF class. Statistical inference for the RIM is carried out in a Bayesian framework. The performance of the RIM is evaluated using a simulation study in which it is compared with traditional procedures, like the likelihood ratio test, the Mantel‐Haenszel procedure and the standardized p ‐DIF procedure. In this comparison, the RIM performs better than the other methods. Finally, the usefulness of the model is also demonstrated on a real life data set.

Improvement in Detection of Differential Item Functioning Using a Mixture Item Response Theory Model

Usually, methods for detection of differential item functioning (DIF) compare the functioning of items across manifest groups. However, the manifest groups with respect to which the items function differentially may not necessarily coincide with the true source of the bias. It is expected that DIF detection under a model that includes a latent DIF variable is more sensitive to this source of bias. In a simulation study, it is shown that a mixture item response theory model, which includes a latent grouping variable, performs better in identifying DIF items than DIF detection methods using manifest variables only. The difference between manifest and latent DIF detection increases as the correlation between the manifest variable and the true source of the DIF becomes smaller. Different sample sizes, relative group sizes, and significance levels are studied. Finally, an empirical example demonstrates the detection of heterogeneity in a minority sample using a latent grouping variable. Manifest and latent DIF detection methods are applied to a Vocabulary test of the General Aptitude Test Battery (GATB).

Hierarchical Logistic Regression: Accounting for Multilevel Data in DIF Detection

The purpose of this study was to examine the performance of differential item functioning (DIF) assessment in the presence of a multilevel structure that often underlies data from large‐scale testing programs. Analyses were conducted using logistic regression (LR), a popular, flexible, and effective tool for DIF detection. Data were simulated using a hierarchical framework, such as might be seen when examinees are clustered in schools, for example. Both standard and hierarchical LR (accounting for multilevel data) approaches to DIF detection were employed. Results highlight the differences in DIF detection rates when the analytic strategy matches the data structure. Specifically, when the grouping variable was within clusters, LR and HLR performed similarly in terms of Type I error control and power. However, when the grouping variable was between clusters, LR failed to maintain the nominal Type I error rate of .05. HLR was able to maintain this rate. However, power for HLR tended to be low under many conditions in the between cluster variable case.

Explaining Crossing DIF in Polytomous Items Using Differential Step Functioning Effects

Crossing, or intersecting, differential item functioning (DIF) is a form of nonuniform DIF that exists when the sign of the between-group difference in expected item performance changes across the latent trait continuum. The presence of crossing DIF presents a problem for many statistics developed for evaluating DIF because positive and negative conditional between-group differences in expected item score can cancel to yield a negligible between-group difference when aggregated across the latent trait continuum. As a result, many DIF detection statistics are insensitive to DIF that is crossing in nature. Currently, little is known about why crossing DIF occurs in polytomous items. This study demonstrates that crossing DIF in polytomous items can be explained by the presence of differential step functioning (DSF) effects that vary in sign across the steps of the polytomous item, suggesting that crossing DIF is caused by the presence of one or more secondary factors that yield a relative advantage for different groups at different steps of the polytomous item. Using this result, this study describes how to use the pattern of the DSF effects to (a) target the search for secondary traits responsible for a crossing DIF effect, (b) identify situations where certain DIF statistics will lead to the conclusion of no DIF despite a substantial crossing DIF effect, and (c) generate realistic crossing DIF effects in simulation research.

Two Simple Approaches to Overcome a Problem With the Mantel-Haenszel Statistic: Comments on Wang, Bradlow, Wainer, and Muller (2008)

The Mantel-Haenszel (MH) procedure (Mantel and Haenszel) is a popular method for estimating and testing a common two-factor association parameter in a 2 × 2 × K table. Holland and Holland and Thayer described how to use the procedure to detect differential item functioning (DIF) for tests with dichotomously scored items. Wang, Bradlow, Wainer, and Muller showed that the MH procedure unexpectedly often found DIF where there was none for very easy items. They showed that their suggested Bayesian procedure for DIF detection did not have the same problem. We discuss a simpler solution of the problem—the use of the standardized difference in proportion correct (STD P-DIF) procedure in conjugation with the MH procedure—that has been used by DIF analysts since 1988. We also show, using results from real and simulated data, that it is possible to overcome the problem using the empirical Bayes (EB) procedure of Zwick, Thayer, and Lewis, which also is computationally simpler than the procedure suggested by Wang et al.

A general framework and an R package for the detection of dichotomous differential item functioning

Differential item functioning (DIF) is an important issue of interest in psychometrics and educational measurement. Several methods have been proposed in recent decades for identifying items that function differently between two or more groups of examinees. Starting from a framework for classifying DIF detection methods and from a comparative overview of the most traditional methods, an R package for nine methods, called difR, is presented. The commands and options are briefly described, and the package is illustrated through the analysis of a data set on verbal aggression.

Accounting for differential item functioning (DIF) in groups defined by educational attainment reduced the difference in rates of cognitive decline: Illustration of a novel DIF detection framework using modified mini-mental state examination (3MS) data from the Canadian study on health and aging (CSHA)

Accounting for differential item functioning (DIF) in groups defined by educational attainment reduced the difference in rates of cognitive decline: Illustration of a novel DIF detection framework using modified mini-mental state examination (3MS) data from the Canadian study on health and aging (CSHA)

THE VALUE OF THE STUDIED ITEM IN THE MATCHING CRITERION IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS

ABSTRACTThe nature of the matching criterion (usually the total score) in the study of differential item functioning (DIF) has been shown to impact the accuracy of different DIF detection procedures. One of the topics related to the nature of the matching criterion is whether the studied item should be included. Although many studies exist that suggest the studied item should always be included in the criterion, the validity of this statement for models other than the Rasch model has not been studied. This study evaluates the effect of including/excluding the studied item in the matching criterion for situations that mimic real testing situations where the assumptions of the Rasch model are violated. A simulation study was conducted where the effect of including/excluding the studied item in the matching criterion was studied relative to different magnitudes of DIF and different group ability distributions, for data that follow the two‐parameter logistic (2PL) item response theory (IRT) and multidimensional item response theory (MIRT) models. Results from the study show that including the studied item leads to less biased DIF estimates and more appropriate Type I error rate, especially when the group ability distributions are different. Systematic bias positioning DIF estimation in favor of the high ability group was consistently found across all simulated conditions when the studied item was excluded from the matching criterion.