Differential Item Functioning Test Research Articles

Enhancing Precision in Predicting Magnitude of Differential Item Functioning: An M-DIF Pretrained Model Approach

Despite numerous studies on the magnitude of differential item functioning (DIF), different DIF detection methods often define effect sizes inconsistently and fail to adequately account for testing conditions. To address these limitations, this study introduces the unified M-DIF model, which defines the magnitude of DIF as the difference in item difficulty parameters between reference and focal groups. The M-DIF model can incorporate various DIF detection methods and test conditions to form a quantitative model. The pretrained approach was employed to leverage a sufficiently representative large sample as the training set and ensure the model’s generalizability. Once the pretrained model is constructed, it can be directly applied to new data. Specifically, a training dataset comprising 144 combinations of test conditions and 144,000 potential DIF items, each equipped with 29 statistical metrics, was used. We adopt the XGBoost method for modeling. Results show that, based on root mean square error (RMSE) and BIAS metrics, the M-DIF model outperforms the baseline model in both validation sets: under consistent and inconsistent test conditions. Across all 360 combinations of test conditions (144 consistent and 216 inconsistent with the training set), the M-DIF model demonstrates lower RMSE in 357 cases (99.2%), illustrating its robustness. Finally, we provided an empirical example to showcase the practical feasibility of implementing the M-DIF model.

Do conservatives really have an advantage in mental health? Anexamination of measurement invariance.

Many studies have found that conservatives show an advantage in mental health and happiness and various causes of this have been debated (e.g., religiousness, ideology, or genetics). However, not much attention has been given to examining whether this advantage is psychometrically real, or whether it is due to test bias. We analyzed data from two large Finnish surveys of adults (Ns = 848 and 4,978) from Lahtinen (2024), that measured general anxiety and depression symptoms, as well as a new wokeness scale. Using differential item functioning tests, we found no evidence for measurement bias in these scales. The correlation between index scores of wokeness and mental health (internalizing) was -0.36, which increased to -0.41 when measurement error was removed. The association between wokeness and anxiety (r = -0.33, adjusted r = 0.37) was stronger than wokeness and depression (r = 0.20, adjusted r = 0.22).

Extending the Cluster Approach to Differential Item Functioning in Polytomous Items

To objectively compare groups on any latent trait using tests, the absence of differential item functioning (DIF) is crucial. While the importance of DIF has been well-established in research, the question of how to identify DIF-free items is still largely open. The fact that item difficulty is not identified from observations may explain this. Recently, DIF tests utilizing the differences between item difficulties across groups, which are identified, were proposed for the Rasch and 2-parameter logistic models. The current paper aims to extend these approaches to the polytomous case using the partial credit model. Performance of the new approach is assessed using a simulation study, and practical recommendations are made.

Rasch analysis of the self efficacy (SE-12) questionnaire measuring clinical communication skills

ObjectiveThe aim of this study was to examine the construct validity and reliability of the Self Efficacy (SE-12) questionnaire using Rasch analysis. MethodsThe SE-12 was administered to Danish health care professionals prior to their participation in a communication skills training program. Analysis of fit to the Rasch model, ordering of response categories, dimensionality-testing, test for differential item functioning, test for local dependency, and calculation of reliability were used to evaluate the psychometric characteristics of the SE-12. ResultsIn this study, 1057 respondents were analyzed. Overall, the SE-12 demonstrated an acceptable fit to the Rasch model. Response categories were appropriately ordered for all twelve items. However, items 6 (structure) and item 8 (empathy) demonstrated differential item functioning, with men being more likely to affirm the first and women the last item. Local dependency was observed between five item groups but adjusting for these improved fit indices significantly. The SE-12 exhibited high reliability with PSI ranging from 0.92 to 0.94. A transformation table converting ordinal scores to interval scores is provided. ConclusionThe SE-12 demonstrates good construct validity and excellent reliability. Minor issues regarding local dependency and differential functioning require attention.Innovation: A 5-item version could be explored without compromising validity and reliability.

Using Regularization to Identify Measurement Bias Across Multiple Background Characteristics: A Penalized Expectation–Maximization Algorithm

Testing for differential item functioning (DIF) has undergone rapid statistical developments recently. Moderated nonlinear factor analysis (MNLFA) allows for simultaneous testing of DIF among multiple categorical and continuous covariates (e.g., sex, age, ethnicity, etc.), and regularization has shown promising results for identifying DIF among many covariates. However, computationally inefficient estimation methods have hampered practical use of the regularized MNFLA method. We develop a penalized expectation–maximization (EM) algorithm with soft- and firm-thresholding to more efficiently estimate regularized MNLFA parameters. Simulation and empirical results show that, compared to previous implementations of regularized MNFLA, the penalized EM algorithm is faster, more flexible, and more statistically principled. This method also yields similar recovery of DIF relative to previous implementations, suggesting that regularized DIF detection remains a preferred approach over traditional methods of identifying DIF.

Can life satisfaction be measured fairly for different groups of South African first-year university students? Testing differential item functioning and invariance of The Satisfaction With Life Scale

Student well-being has gradually become a topic of interest in higher education, and the accurate, valid, and reliable measure of well-being constructs is crucial in the South African context. This study examined item bias and configural, metric and scalar invariance of the Satisfaction With Life Scale (SWLS) for South African first-year university students. A cross-sectional design was used. A sample of 780 first-year South African university students was included. Confirmatory factor analysis, differential item functioning (DIF) measurement invariance, and internal consistency were tested. A one-factor structure was confirmed. Item 1 of the SWLS was particularly problematic concerning bias (uniform and non-uniform bias). Measurement invariance was established; however, Item 1 was again problematic, resulting in only partial metric and scalar invariance. The scale was reliable (α ≥ 0.70). This study contributes to the limited research on the specific psychometric properties of the SWLS in a diverse Higher Education setting. The results could assist with valid and reliable measurement when developing interventions to enhance student well-being.

Psychometric Properties and Refinement of the ATT-IPV Scale to Measure Attitudes about Intimate Partner Violence among Married Adolescents and Their Husbands in Niger.

(1) Background: This study sought to assess the appropriateness of a five-item scale to measure attitudes towards IPV (ATT-IPV) among married adolescent girls and their husbands in Niger, a population in which this scale has not yet been tested. (2) Methods: Using data collected from 1100 pairs of married adolescents, aged 13-19 years old, and their husbands across 48 villages in rural Niger, we performed classical test theory reliability and exploratory factor analysis, followed by item response theory (IRT) analyses and testing differential item functioning (DIF) by gender. (3) Results: The ATT-IPV scale was found to be internally consistent (alpha = 0.8) and unidimensional in this population, with all items loading onto one factor. We found differential item functioning of the following item: "In your opinion, is a husband justified in hitting or beating his wife in the following situations: If she burns his food?" by gender, suggesting that in order to have a scale that performs similarly in men and women, that item should be removed. (4) Conclusions: The ATT-IPV scale is useful as a measure of attitudes towards IPV among married adolescents and their husbands in Niger. However, it may need to be updated to reflect additional forms of violence and to eliminate gender-differential responses in order to be a more effective measure.

Assessing the Validity of the Baby Pediatric Symptom Checklist Using a Nationally Representative Household Survey.

The Baby Pediatric Symptom Checklist (BPSC) is a screening tool developed for detecting behavioral or emotional concerns among parents of children younger than 18 months. Nationally representative survey data have not yet been used to assess the validity of the BPSC, nor to evaluate its appropriateness for use among children between 18 and 23 months old. The current study assesses the validity of the BPSC using data from the National Health Interview Survey (NHIS). Data from the 2019 NHIS were used to evaluate the 12-item BPSC screening tool among a nationally representative sample of children 2 to 23 months. Confirmatory factor analysis (CFA) and differential item functioning (DIF) were used to assess construct and predictive validity and test how response items differed by selected sociodemographics. Quantile regression was used to calculate 50th, 70th, and 90th percentiles for age-based normative curves of the previously established domains of irritability, inflexibility, and difficulty with routines. A 3-factor CFA produced comparable results to the original study. Tests of DIF did not reveal any significant effects for the child's sex, race and Hispanic origin, household urbanization level, number of children in family, or respondent type (mother, father, other). In addition, DIF was not found between children aged 2 to 17 months and 18 to 23 months. Age-based normative data were calculated for each subscale. The use of the BPSC in a nationally representative survey produced findings comparable to those of the original-validation study. The NHIS can be used to track BPSC scores over time at the population-level.

Comparing the construct validity of the Patient Health Questionnaire (PHQ-9) and the Major Depression Inventory (MDI) using Rasch analysis

ObjectiveThe Patient Health Questionnaire (PHQ-9) and the Major Depression Inventory (MDI) are psychometric instruments frequently used for depression diagnosis and monitoring in primary care settings. The goal of this study was to assess and compare the construct validity of the PHQ-9 and the MDI using Rasch analysis. MethodsThe PHQ-9 and the MDI were distributed in random ordering (first or last) to a waiting room sample of Danish primary care patients. Analysis of fit to the Rasch model, ordering of response categories, dimensionality-testing, test for differential item functioning, test for local dependency, and calculation of reliability indices were used to evaluate the psychometric characteristics of the instruments. ResultsA sample of 281 persons (18–87 years, 72 % females) responded to both instruments. Reliability was acceptable for PHQ-9 (PSI 0.80) and excellent for MDI (PSI 0.88). Both instruments demonstrated fit to the Rasch model after rescoring of all items. Dimensionality testing supports that both instruments are unidimensional. Indication of local dependency was observed for PHQ-9 item 2&6 and MDI item 1&4. No indication of differential item functioning regarding age, gender or ordering of instruments was found. ConclusionThe PHQ-9 and the MDI both demonstrate measurement problems. Modifying individual items and reducing the number of response categories in both instruments could potentially improve their measurement capabilities. Study findings do not provide a conclusive indication of which instrument has superior operating characteristics.

Chronic Airways Assessment Test: psychometric properties in patients with asthma and/or COPD

BackgroundNo short patient-reported outcome (PRO) instruments assess overall health status across different obstructive lung diseases. Thus, the wording of the introduction to the Chronic Obstructive Pulmonary Disease (COPD) Assessment Test (CAT) was modified to permit use in asthma and/or COPD. This tool is called the Chronic Airways Assessment Test (CAAT).MethodsThe psychometric properties of the CAAT were evaluated using baseline data from the NOVELTY study (NCT02760329) in patients with physician-assigned asthma, asthma + COPD or COPD. Analyses included exploratory/confirmatory factor analyses, differential item functioning and analysis of construct validity. Responses to the CAAT and CAT were compared in patients with asthma + COPD and those with COPD.ResultsCAAT items were internally consistent (Cronbach’s alpha: > 0.7) within each diagnostic group (n = 510). Models for structural and measurement invariance were strong. Tests of differential item functioning showed small differences between asthma and COPD in individual items, but these were not consistent in direction and had minimal overall impact on the total score. The CAAT and CAT were highly consistent when assessed in all NOVELTY patients who completed both (N = 277, Pearson’s correlation coefficient: 0.90). Like the CAT itself, CAAT scores correlated moderately (0.4–0.7) to strongly (> 0.7) with other PRO measures and weakly (< 0.4) with spirometry measures.ConclusionsCAAT scores appear to reflect the same health impairment across asthma and COPD, making the CAAT an appropriate PRO instrument for patients with asthma and/or COPD. Its brevity makes it suitable for use in clinical studies and routine clinical practice.Trial registration: NCT02760329.

Multi-Group Generalizations of SIBTEST and Crossing-SIBTEST

ABSTRACT This article presents generalizations of SIBTEST and crossing-SIBTEST statistics for differential item functioning (DIF) investigations involving more than two groups. After reviewing the original two-group setup for these statistics, a set of multigroup generalizations that support contrast matrices for joint tests of DIF are presented. To investigate the Type I error and power behavior of these generalizations, a Monte Carlo simulation study was then explored. Results indicated that the proposed generalizations are reasonably effective at recovering their respective population parameter definitions, maintain optimal Type I error control, have suitable power to detect uniform and non-uniform DIF, and in shorter tests are competitive with the generalized logistic regression and generalized Mantel–Haenszel tests for DIF.

Evidence of item bias in a national flourishing measure for autistic youth.

Flourishing is a positive health indicator that aligns with strengths-based perspectives and measures within autism research. Flourishing indicators were recently included in the National Survey of Children's Health (NSCH) and have been used to evidence disparities in flourishing experienced by autistic children compared to non-autistic peers. Yet, little has been done to examine the utility of standard flourishing items for this population. This study examined the NSCH caregiver-reported flourishing items for measurement item bias. A cross-sectional, representative sample of autistic and non-autistic US children aged 6-17 years (n= 41,691) was drawn from the 2018-2019 NSCH public dataset. A confirmatory factor analysis using a multiple indicators and multiple causes model (MIMIC-CFA) was conducted to (1) test for differential item functioning (DIF; i.e., measurement bias); and (2) estimate latent mean group differences after controlling for DIF. Findings supported a 3-factor (social competence, school motivation, and behavioral control), 10-item model structure consistent with past literature, yet measurement bias was evident for 6 of the 10 items. Persistent group differences, after accounting for DIF and covariates, indicates that caregivers of autistic children perceive their children are experiencing meaningfully lower flourishing outcomes compared to caregivers of non-autistic children. However, evidence of measurement bias for items related to the social competence dimension calls into question the applicability of this measure for autistic children. Further interpretation of group differences and use of this measure should be approached with caution.

Generalized Mantel-Haenszel Estimators for Simultaneous Differential Item Functioning Tests.

The Mantel-Haenszel estimator is one of the most popular techniques for measuring differential item functioning (DIF). A generalization of this estimator is applied to the context of DIF to compare items by taking the covariance of odds ratio estimators between dependent items into account. Unlike the Item Response Theory, the method does not rely on the local item independence assumption which is likely to be violated when one item provides clues about the answer of another item. Furthermore, we use these (co)variance estimators to construct a hypothesis test to assess DIF for multiple items simultaneously. A simulation study is presented to assess the performance of several tests. Finally, the use of these DIF tests is illustrated via application to two real data sets.

Power Analysis for the Wald, LR, Score, and Gradient Tests in a Marginal Maximum Likelihood Framework: Applications in IRT

The Wald, likelihood ratio, score, and the recently proposed gradient statistics can be used to assess a broad range of hypotheses in item response theory models, for instance, to check the overall model fit or to detect differential item functioning. We introduce new methods for power analysis and sample size planning that can be applied when marginal maximum likelihood estimation is used. This allows the application to a variety of IRT models, which are commonly used in practice, e.g., in large-scale educational assessments. An analytical method utilizes the asymptotic distributions of the statistics under alternative hypotheses. We also provide a sampling-based approach for applications where the analytical approach is computationally infeasible. This can be the case with 20 or more items, since the computational load increases exponentially with the number of items. We performed extensive simulation studies in three practically relevant settings, i.e., testing a Rasch model against a 2PL model, testing for differential item functioning, and testing a partial credit model against a generalized partial credit model. The observed distributions of the test statistics and the power of the tests agreed well with the predictions by the proposed methods in sufficiently large samples. We provide an openly accessible R package that implements the methods for user-supplied hypotheses.

Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression

Differential item functioning (DIF) occurs when the probability of endorsing an item differs across groups for individuals with the same latent trait level. The presence of DIF items may jeopardize the validity of an instrument; therefore, it is crucial to identify DIF items in routine operations of educational assessment. While DIF detection procedures based on item response theory (IRT) have been widely used, a majority of IRT-based DIF tests assume predefined anchor (i.e., DIF-free) items. Not only is this assumption strong, but violations to it may also lead to erroneous inferences, for example, an inflated Type I error rate. We propose a general framework to define the effect sizes of DIF without a priori knowledge of anchor items. In particular, we quantify DIF by item-specific residuals from a regression model fitted to the true item parameters in respective groups. Moreover, the null distribution of the proposed test statistic using robust estimator can be derived analytically or approximated numerically even when there is a mix of DIF and non-DIF items, which yields asymptotically justified statistical inference. The Type I error rate and the power performance of the proposed procedure are evaluated and compared with the conventional likelihood-ratio DIF tests in a Monte Carlo experiment. Our simulation study has shown promising results in controlling Type I error rate and power of detecting DIF items. Even when there is a mix of DIF and non-DIF items, the true and false alarm rate can be well controlled when a robust regression estimator is used.

Uirt: A command for unidimensional IRT modeling

In this article, I introduce the uirt command, which allows one to estimate parameters of a variety of unidimensional item response theory models (two-parameter logistic model, three-parameter logistic model, graded response model, partial credit model, and generalized partial credit model). uirt has extended item-fit analysis capabilities, features multigroup modeling, allows testing for differential item functioning, and provides tools for generating plausible values with a latent regression conditioning model. I provide examples to illustrate cases where uirt can be especially useful in conducting analyses within the item response theory approach.

Analysis of Children’s Online Reading Behavior Oriented for Family Education

Aiming at the problem of supply-demand matching of online reading, an analysis method of children’s online reading behavior oriented for family education has been put forward. The data-based classification method is constructed to classify the sample population by statistical methods, and the traditional index classification is carried out by using K-medoids clustering and logistic regression analysis. The matching degree of population classification is discussed through comparison. R language and Mplus are used to analyze the data for the objective classification of the sample data set. Based on the reading response behavior of children’s online reading users, a differential item functioning (DIF) test of socioeconomic status is carried out. At the same time, the population is divided by traditional economic classification indicators to carry out a DIF test and explore the differences in the reading ability of different classification groups. By comparing the results of the two grouping methods, the main family socioeconomic status factors affecting reading performance are explored and targeted countermeasures are put forward. The experimental results show that when analyzing children’s online reading behavior, using machine learning algorithms such as cluster analysis, logistic regression analysis, and so on can get consistent results and then using the DIF test to explore the responses of category groups can effectively distinguish group differences.

Analysis of Race and Sex Bias in the Autism Diagnostic Observation Schedule (ADOS-2)

There are long-standing disparities in the prevalence of autism spectrum disorder (ASD) across race and sex. Surprisingly, few studies have examined whether these disparities arise partially out of systematic biases in the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2), the reference standard measure of ASD. To examine differential item functioning (DIF) of ADOS-2 items across sex and race. This is a cross-sectional study of children who were evaluated for ASD between 2014 and 2020 at a specialty outpatient clinic located in the Mid-Atlantic region of the US. Data were analyzed from July 2021 to February 2022. Child race (Black/African American vs White) and sex (female vs male). Item-level biases across ADOS-2 harmonized algorithm items, including social affect (SA; 10 items) and repetitive/restricted behaviors (RRBs; 4 items), were evaluated across 3 modules. Measurement bias was identified by examining DIF and differential test functioning (DTF), within a graded response, item response theory framework. Statistical significance was determined by a likelihood ratio χ2 test, and a series of metrics was used to examine the magnitude of DIF and DTF. A total of 6269 children (mean [SD] age, 6.77 [3.27] years; 1619 Black/African American [25.9%], 3151 White [50.3%], and 4970 male [79.4%]), were included in this study. Overall, 16 of 140 ADOS-2 diagnostic items (11%) had a significant DIF. For race, 8 items had a significant DIF, 6 of which involved SA. No single item showed DIF consistently across all modules. Most items with DIF had greater difficulty and poorer discrimination in Black/African American children compared with White children. For sex, 5 items showed significant DIF. DIF was split across SA and RRB. However, hand mannerisms evidenced DIF across all 5 algorithms, with generally greater difficulty. The magnitude of DIF was only moderate to large for 2 items: hand mannerisms (among female children) and repetitive interests (among Black/African American children). The overall estimated effect of DIF on total DTF was not large. These findings suggest that the ADOS-2 does not have widespread systematic measurement bias across race or sex. However, the findings raise some concerns around underdetection that warrant further research.

A comparison of methods to address item non-response when testing for differential item functioning in multidimensional patient-reported outcome measures.

Item non-response (i.e., missing data) may mask the detection of differential item functioning (DIF) in patient-reported outcome measures or result in biased DIF estimates. Non-response can be challenging to address in ordinal data. We investigated an unsupervised machine-learning method for ordinal item-level imputation and compared it with commonly-used item non-response methods when testing for DIF. Computer simulation and real-world data were used to assess several item non-response methods using the item response theory likelihood ratio test for DIF. The methods included: (a) list-wise deletion (LD), (b) half-mean imputation (HMI), (c) full information maximum likelihood (FIML), and (d) non-negative matrix factorization (NNMF), which adopts a machine-learning approach to impute missing values. Control of Type I error rates were evaluated using a liberal robustness criterion for α = 0.05 (i.e., 0.025-0.075). Statistical power was assessed with and without adoption of an item non-response method; differences > 10% were considered substantial. Type I error rates for detecting DIF using LD, FIML and NNMF methods were controlled within the bounds of the robustness criterion for > 95% of simulation conditions, although the NNMF occasionally resulted in inflated rates. The HMI method always resulted in inflated error rates with 50% missing data. Differences in power to detect moderate DIF effects for LD, FIML and NNMF methods were substantial with 50% missing data and otherwise insubstantial. The NNMF method demonstrated comparable performance to commonly-used non-response methods. This computationally-efficient method represents a promising approach to address item-level non-response when testing for DIF.

The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods

BackgroundItem response theory (IRT) methods for addressing differential item functioning (DIF) can detect group differences in responses to individual items (e.g., bias). IRT and DIF-detection methods have been used increasingly often to identify bias in cognitive test performance by characteristics (DIF grouping variables) such as hearing impairment, race, and educational attainment. Previous analyses have not considered the effect of missing data on inferences, although levels of missing cognitive data can be substantial in epidemiologic studies.MethodsWe used data from Visit 6 (2016–2017) of the Atherosclerosis Risk in Communities Neurocognitive Study (N = 3,580) to explicate the effect of artificially imposed missing data patterns and imputation on DIF detection.ResultsWhen missing data was imposed among individuals in a specific DIF group but was unrelated to cognitive test performance, there was no systematic error. However, when missing data was related to cognitive test performance and DIF group membership, there was systematic error in DIF detection. Given this missing data pattern, the median DIF detection error associated with 10%, 30%, and 50% missingness was -0.03, -0.08, and -0.14 standard deviation (SD) units without imputation, but this decreased to -0.02, -0.04, and -0.08 SD units with multiple imputation.ConclusionsIncorrect inferences in DIF testing have downstream consequences for the use of cognitive tests in research. It is therefore crucial to consider the effect and reasons behind missing data when evaluating bias in cognitive testing.