Differential Item Functioning Studies Research Articles

Purification procedures used for the detection of gender DIF: Item bias in a foreign language test

Differential item functioning (DIF) detection was handled based on “Mantel-Haenszel (MH)”, “Simultaneous item bias test (SIBTEST)”, “Lord's chi-square”, “Raju's area” methods when item purification was performed or item purification was not performed using real data in current study. After detecting gender-related DIF, expert opinions were taken for bias study. It is important to conduct the gender bias research in the English test when purification is performed and when purification is not performed, as there were DIF studies, but there were not completely similar bias studies in the literature. The sample of the research consists of 7389 students who took the “Transition from Primary to Secondary Education Exam (TPSEE, referred to as “TEOG” in Turkey)” administered in April 2017. When gender-related DIF analysis was performed with the four methods, the results were found to differ partially. DIF analysis results differed in the different conditions item purification was performed or not. Detection of DIF was indicative of possible bias. In the second stage of the study, the opinions of seven experts were taken for item 11, for which DIF was detected at least at B level based on MH, SIBTEST. As a result of expert opinion, it was found that there was no item bias according to gender in any item in the English test. It is recommended that similar bias studies can be conducted for test developers to be aware of the features that may lead to item bias and to construct unbiased items.

Bayesian inference with spike-and-slab priors for differential item functioning detection in a multiple-group IRT tree model

Group differences have practical implications in analysing data from achievement tests or questionnaires. In the current study, we develop a model that accounts for between-group differences, differential item functioning (DIF), latent factors, and missing item response data simultaneously. Different from most of the present DIF studies where one has to iteratively select anchor items and detect DIF items, we achieve DIF detection and parameter estimation simultaneously by properly reparameterizing model parameters and applying some spike-and-slab priors (Ishwaran & Rao, Spike and slab variable selection: frequentist and Bayesian strategies. Ann Stat. 2005a;33:730–773; Ročková & George, The spike-and-slab LASSO. J Am Stat Assoc. 2018;113:431–444) in Bayesian estimation. Simulation studies are conducted to illustrate the validation of the proposed estimation procedure and the efficiency of DIF detection. The proposed method is further applied to a real dataset for illustration.

A Machine Learning Approach to Assess Differential Item Functioning in Psychometric Questionnaires Using the Elastic Net Regularized Ordinal Logistic Regression in Small Sample Size Groups.

Assessing differential item functioning (DIF) using the ordinal logistic regression (OLR) model highly depends on the asymptotic sampling distribution of the maximum likelihood (ML) estimators. The ML estimation method, which is often used to estimate the parameters of the OLR model for DIF detection, may be substantially biased with small samples. This study is aimed at proposing a new application of the elastic net regularized OLR model, as a special type of machine learning method, for assessing DIF between two groups with small samples. Accordingly, a simulation study was conducted to compare the powers and type I error rates of the regularized and nonregularized OLR models in detecting DIF under various conditions including moderate and severe magnitudes of DIF (DIF = 0.4 and 0.8), sample size (N), sample size ratio (R), scale length (I), and weighting parameter (w). The simulation results revealed that for I = 5 and regardless of R, the elastic net regularized OLR model with w = 0.1, as compared with the nonregularized OLR model, increased the power of detecting moderate uniform DIF (DIF = 0.4) approximately 35% and 21% for N = 100 and 150, respectively. Moreover, for I = 10 and severe uniform DIF (DIF = 0.8), the average power of the elastic net regularized OLR model with 0.03 ≤ w ≤ 0.06, as compared with the nonregularized OLR model, increased approximately 29.3% and 11.2% for N = 100 and 150, respectively. In these cases, the type I error rates of the regularized and nonregularized OLR models were below or close to the nominal level of 0.05. In general, this simulation study showed that the elastic net regularized OLR model outperformed the nonregularized OLR model especially in extremely small sample size groups. Furthermore, the present research provided a guideline and some recommendations for researchers who conduct DIF studies with small sample sizes.

A Revisit of Zumbo’s Third Generation DIF: How Are We Doing in Language Testing?

ABSTRACT The purpose of this study is to review the status of differential item functioning (DIF) research in language testing, particularly as it relates to the investigation of sources (or causes) of DIF, which is a defining characteristic of the third generation DIF. This review included 110 DIF studies of language tests dated from 1985 to 2019. We found that DIF researchers did not address sources of DIF more frequently in recent years than in earlier years. Nevertheless, DIF research in language testing has expanded with new DIF analysis procedures, more grouping variables, and more diversified methods for investigating sources of DIF. In addition, in the early years of DIF research, methods to identify sources of DIF relied heavily on content analysis. This review showed that while more sophisticated statistical procedures have been adopted in recent years to address sources of DIF, understanding sources of DIF still remains a challenging task. We also discuss the pros and cons of existing methods to detect sources of DIF and implications for future investigations.

Using Lasso and Adaptive Lasso to Identify DIF in Multidimensional 2PL Models

Differential item functioning (DIF) analysis refers to procedures that evaluate whether an item’s characteristic differs for different groups of persons after controlling for overall differences in performance. DIF is routinely evaluated as a screening step to ensure items behave the same across groups. Currently, the majority DIF studies focus predominately on unidimensional IRT models, although multidimensional IRT (MIRT) models provide a powerful tool for enriching the information gained in modern assessment. In this study, we explore regularization methods for DIF detection in MIRT models and compare their performance to the classic likelihood ratio test. Regularization methods have recently emerged as a new family of methods for DIF detection due to their advantages: (1) they bypass the tedious iterative purification procedure that is often needed in other methods for identifying anchor items, and (2) they can handle multiple covariates simultaneously. The specific regularization methods considered in the study are: lasso with expectation-maximization (EM), lasso with expectation-maximization-maximization (EMM) algorithm, and adaptive lasso with EM. Simulation results show that lasso EMM and adaptive lasso EM hold great promise when the sample size is large, and they both outperform lasso EM. A real data example from PROMIS depression and anxiety scales is presented in the end.

Differential item functioning of the Beck Anxiety Inventory in a rural, multi-ethnic cohort

BackgroundEvaluating measurement bias is vital to ensure equivalent assessment across diverse groups. One approach for evaluating test bias, differential item functioning (DIF), assesses item-level bias across specified groups by comparing item-level responses between groups that have the same overall score. Previous DIF studies of the Beck Anxiety Inventory (BAI) have only assessed bias across age, sex, and disease duration in monolingual samples. We expand this literature through DIF analysis of the BAI across age, sex, education, ethnicity, cognitive status, and test language. MethodsBAI data from a sample (n = 527, mean age=61.4 ± 12.7, mean education=10.9 ± 4.3, 69.3% female, 41.9% Hispanic/Latin American) from rural communities in West Texas, USA were analyzed. Item response theory (IRT) / logistic ordinal regression DIF was conducted across dichotomized demographic grouping factors. The Mann-Whitney U test and Hedge's g standardized mean differences were calculated before and after adjusting for the impact of DIF. ResultsSignificant DIF was demonstrated in 10/21 items. An adverse impact of DIF was not identified when demographics were assessed individually. Adverse DIF was identified for only one participant (1/527, 0.2%) when all demographics were aggregated. LimitationsThese results might not be generalizable to a sample with broader racial representation, more severe cognitive impairment, and higher levels of anxiety. ConclusionsMinimal item-level bias was identified across demographic factors considered. These results support prior evidence that the BAI is valid for assessing anxiety across age and sex while contributing new evidence of its clinical relevance across education, ethnicity, cognitive status, and English/Spanish test language.

Gender-Related Differential Item Functioning of 2015 Wassce Core Mathematics Results in Southern Ghana Using Logistic Regression Procedure

Differential item functioning (DIF) occurs when individuals of the same ability level from separate groups have a different probability of answering an item correctly. This study was conducted in two parts: in the first part a real 2015 West African Senior Secondary Certificate Examination (WASSCE) core mathematics test data were analyzed for uniform and nonuniform DIF using binary logistic regression (LR) procedure and in the second part, content analysis of items identified as DIF were classified under the levels of the cognitive domain by experts. Three research questions were formulated for the study. A sample of 4,285 male and 3,712 female candidates were selected from a population of 15,258 candidates who sat for the examination in 2015 from 20 selected schools in Southern Ghana. The instrument for the study was the 50 multiple-choice core mathematics items. The findings showed that there was 43 significant gender differential item functioning items of which 9 were uniform and 34 non-uniform. Also, the content analysis revealed that items that favoured males were mainly number and numeration, algebraic processes, probability and statistics and mensuration whiles plane geometry and coordinate geometry revealed DIF in favour of females. It was concluded that test items used were not free from gender DIF. It was recommended that DIF studies should be conducted by test developers in order to be review or exclude DIF items to enhance fairness in assessment.

Differential Item Functioning: Beyond validity evidence based on internal structure.

In the latest release of the Standards for Educational and Psychological Testing, Differential Item Functioning (DIF) is considered as validity evidence based on internal structure. However, there are no indications of how to design a DIF study as a validation study. In this paper, we propose relating DIF to all sources of validity evidence, and provide a general conceptual framework for transforming “typical” DIF studies into validation studies. We perform a comprehensive review of the literature and make theoretical and practical proposals. The article provides arguments in favour of addressing DIF detection and interpretation as validation studies, and suggestions for conducting DIF validation studies. The combination of quantitative and qualitative data within a mixed methods research perspective, along with planning DIF studies as validation studies, can help improve the validity of test score interpretations.

Differential Item Functioning Detection Across Two Methods of Defining Group Comparisons: Pairwise and Composite Group Comparisons.

This study compares two methods of defining groups for the detection of differential item functioning (DIF): (a) pairwise comparisons and (b) composite group comparisons. We aim to emphasize and empirically support the notion that the choice of pairwise versus composite group definitions in DIF is a reflection of how one defines fairness in DIF studies. In this study, a simulation was conducted based on data from a 60-item ACT Mathematics test (ACT; Hanson & Béguin). The unsigned area measure method (Raju) was used as the DIF detection method. An application to operational data was also completed in the study, as well as a comparison of observed Type I error rates and false discovery rates across the two methods of defining groups. Results indicate that the amount of flagged DIF or interpretations about DIF in all conditions were not the same across the two methods, and there may be some benefits to using composite group approaches. The results are discussed in connection to differing definitions of fairness. Recommendations for practice are made.

An investigation of DIF mechanisms in the context of differential testlet effects

This study investigated differential item functioning (DIF) mechanisms in the context of differential testlet effects across subgroups. Specifically, we investigated DIF manifestations when the stochastic ordering assumption on the nuisance dimension in a testlet does not hold. DIF hypotheses were formulated analytically using a parametric marginal item response function approach and compared with empirical DIF results from a unidimensional item response theory approach. The comparisons were made in terms of type of DIF (uniform or non-uniform) and direction (whether the focal or reference group was advantaged). In general, the DIF hypotheses were supported by the empirical results, showing the usefulness of the parametric approach in explaining DIF mechanisms. Both analytical predictions of DIF and the empirical results provide insights into conditions where a particular type of DIF becomes dominant in a specific DIF direction, which is useful for the study of DIF causes.

Gender DIF in Reading and Mathematics Tests With Mixed Item Formats

This was a study of differential item functioning (DIF) for grades 4, 7, and 10 reading and mathematics items from state criterion-referenced tests. The tests were composed of multiple-choice and constructed-response items. Gender DIF was investigated using POLYSIBTEST and a Rasch procedure. The Rasch procedure flagged more items for DIF than did the simultaneous item bias procedure—particularly multiple-choice items. For both reading and mathematics tests, multiple-choice items generally favored males while constructed-response items generally favored females. Content analyses showed that flagged reading items typically measured text interpretations or implied meanings; males tended to benefit from items that asked them to identify reasonable interpretations and analyses of informational text. Most items that favored females asked students to make their own interpretations and analyses, of both literary and informational text, supported by text-based evidence. Content analysis of mathematics items showed that items favoring males measured geometry, probability, and algebra. Mathematics items favoring females measured statistical interpretations, multistep problem solving, and mathematical reasoning.

Issues Affecting Item Response Theory Fit in Language Assessment: A Study of Differential Item Functioning in the Iranian National University Entrance Exam

This study aimed at examining the issues affecting the use of IRT models in investigating differential item functioning in high stakes testing. It specifically focused on the Iranian National University Entrance Exam (INUEE) Special English Subtest. A sample of 200,000 participants was randomly selected from the candidates taking part in the INUEE 2003 and 2004 respectively. The data collected in six domains of vocabulary, grammar, word order, language function, cloze test and reading comprehension were analyzed to evaluate the applicability of item response theory (IRT; Embretson & Reise, 2000), including the use of IRT for assessing differential item functioning (DIF; Zumbo, 2007). Substantial model-data misfit was observed in calibrations using PARSCALE and BILOG MG software (Scientific Software International, 2004). Additional analysis through Xcalibre and Iteman 4 (Assessment Systems Corporation, 2010) suggested that item response theory, including IRT-based DIF analysis, is not applicable when the test administered is noticeably beyond the participants’ level of capability, when the test is speeded, or if students are penalized for their wrong answers.

RECOMMENDATIONS FOR CONDUCTING DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSES FOR STUDENTS WITH DISABILITIES BASED ON PREVIOUS DIF STUDIES

ABSTRACTThe purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and DIF results. Based on descriptive and graphical summaries of previous DIF studies, we make recommendations for future studies of DIF for students with disabilities.

Differential Item Functioning Analysis of an EFL Vocabulary Test

Differential item functioning (DIF) has been studied as a statistical method of finding item bias, and DIF studies have been conducted on tests of...

The Effects of Small Sample Size on Identifying Polytomous DIF Using the Liu–Agresti Estimator of the Cumulative Common Odds Ratio

This study is an evaluation of the behavior of the Liu—Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu—Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no simulation study focusing on small samples has analyzed this estimator regarding effect size, Type I error, and power rates. The article begins with a description of this estimator in the context of polytomous DIF. Then it presents the methods and results for a simulation study in which three factors are manipulated: between-group difference in ability distribution, form of DIF introduced into the polytomous item, and sample size. The results of this study indicate that for samples smaller than 200, very little power was observed for statistical significance testing; however, Type I error rates were close to nominal levels, and the recovery of the log odds ratio as an effect size was relatively unaffected by sample size for high discriminating items. Implications for practice are discussed.

THE VALUE OF THE STUDIED ITEM IN THE MATCHING CRITERION IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS

ABSTRACTThe nature of the matching criterion (usually the total score) in the study of differential item functioning (DIF) has been shown to impact the accuracy of different DIF detection procedures. One of the topics related to the nature of the matching criterion is whether the studied item should be included. Although many studies exist that suggest the studied item should always be included in the criterion, the validity of this statement for models other than the Rasch model has not been studied. This study evaluates the effect of including/excluding the studied item in the matching criterion for situations that mimic real testing situations where the assumptions of the Rasch model are violated. A simulation study was conducted where the effect of including/excluding the studied item in the matching criterion was studied relative to different magnitudes of DIF and different group ability distributions, for data that follow the two‐parameter logistic (2PL) item response theory (IRT) and multidimensional item response theory (MIRT) models. Results from the study show that including the studied item leads to less biased DIF estimates and more appropriate Type I error rate, especially when the group ability distributions are different. Systematic bias positioning DIF estimation in favor of the high ability group was consistently found across all simulated conditions when the studied item was excluded from the matching criterion.

A bibliometric study of differential item functioning

This study presents a bibliometric analysis of scientific output in the area of Differential Item Functioning (DIF), the aim being to offer an overview of research activity in this field and characterise its most important aspects and their evolution over the last quarter of the 20th century, thus providing data regarding the basis on which this activity is being developed at the beginning of the 21st century. The analysis makes use of the Web of Science database, the search being restricted to articles published between 1975 and 2000 and which contain the terms ‘differential item functioning’, ‘DIF’ or ‘item bias’. The various analyses focus on the presentation of publication frequencies and percentages, as well as on the application of Bradford’s law of scattering and Lotka’s law.

DIF Detection and Interpretation in Large-Scale Science Assessments: Informing Item Writing Practices

Differential item functioning (DIF) analyses are a routine part of the development of large-scale assessments. Less common are studies to understand the potential sources of DIF. The goals of this study were (a) to identify gender DIF in a large-scale science assessment and (b) to look for trends in the DIF and non-DIF items due to content, cognitive demands, item type, item text, and visual-spatial or reference factors. To facilitate the analyses, DIF studies were conducted at 3 grade levels and for 2 randomly equivalent forms of the science assessment at each grade level (administered in different years). The DIF procedure itself was a variant of the "standardization procedure" of Dorans and Kulick (1986) and was applied to very large sets of data (6 sets of data, each involving 60,000 students). It has the advantages of being easy to understand and to explain to practitioners. Several findings emerged from the study that would be useful to pass on to test development committees. For example, when there was DIF in science items, MC items tended to favor male examinees and OR items tended to favor female examinees. Compiling DIF information across multiple grades and years increases the likelihood that important trends in the data will be identified and that item writing practices will be informed by more than anecdotal reports about DIF.

A SIBTEST Approach to Testing DIF Hypotheses Using Experimentally Designed Test Items

This paper considers a modification of the DIF procedure SIBTEST for investigating the causes of differential item functioning (DIF). One way in which factors believed to be responsible for DIF can be investigated is by systematically manipulating them across multiple versions of an item using a randomized DIF study (Schmitt, Holland, & Dorans, 1993). In this paper: it is shown that the additivity of the index used for testing DIF in SIBTEST motivates a new extension of the method for statistically testing the effects of DIF factors. Because an important consideration is whether or not a studied DIF factor is consistent in its effects across items, a methodology for testing item x factor interactions is also presented. Using data from the mathematical sections of the Scholastic Assessment Test (SAT), the effects of two potential DIF factors—item format (multiple‐choice versus open‐ended) and problem type (abstract versus concrete)—are investigated for gender Results suggest a small but statistically significant and consistent effect of item format (favoring males for multiple‐choice items) across items, and a larger but less consistent effect due to problem type.

Top-down purification of Tobacyk’s Revised Paranormal Belief Scale

Using mostly undergraduate student data ( N =560), factor analysis confirmed the existence of seven factors in Tobacyk’s Revised Paranormal Belief Scale ( RPBS ). However, this replication proved to be of dubious value since (1) the factor structure was significantly affected by age and gender; (2) the items in Tobacyk’s original factors were significantly non-additive; (3) the items showed pervasive differential item functioning ( DIF ) related to respondents’ ages, gender, or both; and (4) simulations indicated that the factor structure are the result of DIF . The removal of DIF via “top-down purification” yielded two correlated clusters of items dubbed “ New Age Philosophy ” and “ Traditional Paranormal Beliefs ”. Both clusters obey an Andrich rating-scale model with person reliabilities of 0.90 and 0.74, respectively. Principal component analysis confirmed that these two clusters are unidimensional and without significant age or gender DIF ( p >0.10). While the Traditional Paranormal Beliefs cluster continued to show a significant main effect of age, neither cluster showed a significant gender effect, thereby calling into question some traditional findings regarding paranormal beliefs. We strongly suspect that the dearth of DIF studies indicates that age and gender biases due to DIF have largely gone unnoticed in the extant personality and assessment literature.