Item-level Differential Item Functioning Research Articles

Measuring Personality Systems in Older Adults: Age-Neutrality of the Dutch BIS/BAS Questionnaire.

With a rapidly aging population, age-neutrality of psychological instruments is becoming ever more important for accurate assessment of older adults. This study aims to test the age-neutrality of the Dutch Behavioral Inhibition System/Behavioral Approach System (BIS/BAS) scales by means of Differential Item Functioning (DIF) and Differential Test Functioning (DTF) analyses. DIF and DTF analyses were conducted using an odds ratio approach. Potential DIF was examined for the two primary scales and three BAS subscales in 390 Dutch-speaking participants divided over three age groups. Overall, the BIS-BAS scales lacked age-neutrality when comparing older adults to young adults: eight out of 20 items (ie, 40%) displayed DIF according to the adjusted Bonferroni corrected cut-offs, which exceeds the threshold of 25% for large DIF. Thus, for 40% of the test items, items were differently endorsed by young and older adults with the same position on the construct measured by the item. Therefore, the implications of the item-level DIF on the scale level were investigated across age groups. DTF analyses revealed large DTF for all BIS and BAS scales according to the adjusted Bonferroni corrected cut-offs. DIF found in items on the BIS scale and BAS-Drive and BAS-Fun Seeking scale can probably be explained by a difference in degree of expression across age groups. A solution could be the development of age-specific norms. DIF on BAS-Reward Responsiveness subscale could be partly due to measuring a different construct across age groups. Rephrasing or replacing such items with DIF may increase age-neutrality of the BIS/BAS Scales.

Test fairness: Examining differential functioning of the reading comprehension section of the GSEEE in China

This study investigated differential item functioning (DIF), differential bundle functioning (DBF), and differential test functioning (DTF) across gender of the reading comprehension section of the Graduate School Entrance English Exam in China. The datasets included 10,000 test-takers’ item-level responses to 6 five-item testlets. Both DIF and DBF were examined by using poly-simultaneous item bias test and item-response-theory-likelihood-ratio test, and DTF was investigated with multi-group confirmatory factor analyses (MG-CFA). The results indicated that although none of the 30 items exhibited statistically and practically significant DIF across gender at the item level, 2 testlets were consistently identified as having significant DBF at the testlet level by the two procedures. Nonetheless, DBF does not manifest itself at the overall test score level to produce DTF based on MG-CFA. This suggests that the relationship between item-level DIF and test-level DTF is a complicated issue with the mediating effect of testlets in testlet-based language assessment.

Looking at DIF From a New Perspective: A Structure-Based Approach Acknowledging Inherent Indefinability.

Differential item functioning (DIF), although highly relevant for psychometric assessment in various fields of psychology, is mathematically not well-defined. Especially the impact, the difference between the means of the person parameters in the focal and the reference group, is a parameter that is not identified without further assumptions. Common DIF detection methods necessarily impose such assumptions, however, in most cases the specific constraints remain quite vague and are implicit in the mathematical algorithms. As an alternative, a structure-based approach is proposed that is independent of the impact and, therefore, not affected by the identification problem. The approach allows to (a) reveal all DIF-relevant information from the data, (b) define indices that quantify the amount of DIF in a test as a whole, and (c) perform item-level DIF analyses. For the last application, it is unavoidable to assume additional identification constraints. However, compared to other existing methods (which also rely on such constraints), the approach suggested here provides advantages as it is not only easier to perform but also much more transparent concerning its underlying assumptions.

Testing whether patients with diabetes and healthy people perceive the meaning of the items in the Persian version of the SF-36 questionnaire similarly: a differential item functioning analysis

It has been rarely studied whether observed disparity in health-related quality-of-life (HRQoL) scores between patients with diabetes and healthy individuals is due to differential item functioning (DIF) or a true difference in the underlying construct. This study aimed to examine DIF in the SF-36 questionnaire and its effect on comparing HRQoL scores between patients with diabetes and healthy people. The sample consisted of 230 patients with type 2 diabetes and 642 healthy individuals who filled out the Persian version of the SF-36 questionnaire. To detect DIF across patients with diabetes and healthy individuals, multiple-group multiple-indicator multiple-causes model was used. In addition, item calibration strategy was used to determine whether the effect of item-level DIF was transferred to the scale level. Nine out of thirty-six (25%) items were detected as DIF, of which one item (11%) was flagged as uniform and eight items (89%) as non-uniform DIF. Most of the DIF items were detected in the mental health component which includes vitality, perceived mental health and social functioning subscales rather than in physical health component. Moreover, nonsignificant latent mean differences for general health perception and social functioning subscales became significant after DIF calibration. The findings of the present study show that patients with diabetes and healthy individuals perceived some items in the SF-36 questionnaire differently. More importantly, in some subscales, the effect of item-level DIF was transferred to the scale level. Consequently, considerable caution should be taken in comparing HRQoL scores between patients with diabetes and healthy individuals.

A Comparison of Paper and Computer Administered Strengths and Difficulties Questionnaire

The Strengths and Difficulties Questionnaire (SDQ) is one of the most widely used measures of young people’s mental health difficulties in research and clinical decision-making. Although the SDQ is available in both paper and computer survey formats, cross-format equivalences have yet to be established. The current study aimed to assess the measure’s equivalence across paper- and computer-based survey formats in a community-based school setting. The study examined self-reported measures completed by a matched sample of 11–14 year olds in secondary schools in England (589 completed paper version; 589 online version). Analyses demonstrate that the factor structure, although did not vary by survey format, resulted in poorly fitting models limiting the use of model based invariance testing. Results indicate that the measure does not operate similarly across different formats, with scale-level mean differences observed for the hyperactivity scale, which also affects the total difficulties score, with higher scores seen in the paper version. Responses to the impact supplement were also influenced by survey format, with higher impact in specific domains disclosed on the computer-based measure. Item-level differential item functioning was observed for four items in the measure; two from the prosocial scale where the DIF is large enough to affect the scale (DTF, ν2 = 0.14). The inconsistency across survey formats highlights the need for more assessment of influences of different survey formats on young people, their perceived privacy and their mental health disclosures via different media. The findings also highlight the potential confounding effect of format when different methods of data collection are used, with a potentially substantive impact on cross-sample comparisons and within child clinical review.

Age-Neutrality of the Trait Facets Proposed for Personality Disorders in DSM-5: A DIFAS Analysis of the PID-5

An age-neutral measurement system is one of the basic conditions for an accurate personality assessment across the lifespan, both longitudinally and cross-sectionally. In this study the age-neutrality of the Personality Inventory for DSM-5 PID-5; Krueger et al. (Psychological Medicine 42, 1879–1890, 2012) was investigated. Potential Differential Item Functioning (DIF) was examined for the 25 trait facets in older versus younger adults. Overall, 33 items displayed large DIF, according to the adjusted Bonferroni corrected cutoffs (Mantel Chi-square, Liu-Agresti Cumulative Common Log-Odds Ratio [L-A LOR], and Cox’s Noncentrality Parameter Estimator [Cox’s B]). In a next step, the implications of the item level DIF across age groups was investigated on scale (i.e., facet) level. These Differential Test Functioning (DTF) analyses revealed large DTF for four of the 25 PID-5 facets (i.e., Withdrawal, Attention Seeking, Rigid Perfectionism and Unusual Beliefs). Current initial results show that most PID-5 traits are measured equally well across age, however, further research is needed to further refine this instrument and make it entirely age-neutral.

Measurement Equivalence in ADL and IADL Difficulty Across International Surveys of Aging: Findings From the HRS, SHARE, and ELSA

To examine the measurement equivalence of items on disability across three international surveys of aging. Data for persons aged 65 and older were drawn from the Health and Retirement Survey (HRS, n = 10,905), English Longitudinal Study of Aging (ELSA, n = 5,437), and Survey of Health, Ageing and Retirement in Europe (SHARE, n = 13,408). Differential item functioning (DIF) was assessed using item response theory (IRT) methods for activities of daily living (ADL) and instrumental activities of daily living (IADL) items. HRS and SHARE exhibited measurement equivalence, but 6 of 11 items in ELSA demonstrated meaningful DIF. At the scale level, this item-level DIF affected scores reflecting greater disability. IRT methods also spread out score distributions and shifted scores higher (toward greater disability). Results for mean disability differences by demographic characteristics, using original and DIF-adjusted scores, were the same overall but differed for some subgroup comparisons involving ELSA. Testing and adjusting for DIF is one means of minimizing measurement error in cross-national survey comparisons. IRT methods were used to evaluate potential measurement bias in disability comparisons across three international surveys of aging. The analysis also suggested DIF was mitigated for scales including both ADL and IADL and that summary indexes (counts of limitations) likely underestimate mean disability in these international populations.

Modeling DIF in Complex Response Data Using Test Design Strategies

This study introduces an approach for modeling multidimensional response data with construct-relevant group and domain factors. The item level parameter estimation process is extended to incorporate the refined effects of test dimension and group factors. Differences in item performances over groups are evaluated, distinguishing two levels of differential item functioning (DIF): a domain level and an item level. An illustration is presented using a Dutch spelling proficiency scale administered to two subgroups. DIF is modeled by the interaction between group and item domain (domain level DIF) and by the interaction between groups and items within each domain (item level DIF). A set of item response theory models was estimated using an adaptation of the logistic regression approach. The model with domain specific item-by-group interactions or DIF performed better than the other models neglecting domain or group differences. The method appears to be promising in that explicit domain factors can be implemented into model estimation procedure to better understand why items favor a specific language group over another.

Examining the relationship between differential item functioning and differential test functioning

The present study utilized both the IRT-LR (item response theory likelihood ratio) and a series of CFA (confirmatory factor analysis) multi-sample analyses to systematically examine the relationships between DIF (differential item functioning) and DTF (differential test functioning) with a random sample of 15 000 Korean examinees. Specifically, DIF was detected using the IRT-LR method and the cumulative effect of DIF on DTF was gauged by the multi-sample analysis technique offered by the LISREL 8.5 program. The results of the current study indicate that item level DIF, once detected, may be carried to the test level bias regardless of the DIF directions, thereby showing mixed evidence to the previous findings reported in the literature. This suggests that the relation of DIF to DTF seems to be much more complex than that reported in the literature and, accordingly, more empirical studies are needed to bridge the gap in the literature about DIF-DTF relationships.

Does item-level DIF manifest itself in scale-level analyses? Implications for translating language tests

Based on the observation that scale-level methods are sometimes exclusively used to investigate measurement invariance for test translation, this article describes the results of a simulation study investigating whether item-level differential item functioning (DIF) manifests itself in scale-level analyses such as single and multi-group factor analyses and per group coefficient alpha. The simulation factors were two levels of DIF (moderate and large) and four levels of percentage of items with DIF (ranging from approximately 3-41% of the items). The results indicate that item-level DIF did not manifest itself in the scale-level results. Clearly, then, translation efforts in language testing should ensure measurement equivalence by investigatingitem-level translation DIF, and it may be misleading to give consideration only to the scale-level methods results as evidence of translation equivalence.

Invariance on the NEO PI-R Neuroticism Scale

We explored between gender invariance on the NEO PI-R Neuroticism scale. Sörbom's (1974;1982) mean and covariance structural analysis was used to evaluate weak and strong forms of factorial invariance for the six facet scales when treated as indicators of a common neuroticism factor. Item response theory procedures were used to evaluate differential item functioning (DIF) within each of the six facets. The structural modeling results indicated that the Neuroticism scale is weakly invariant (i.e., factor loadings were equivalent for men and women), but the hypothesis of strong invariance was not supported. The within-facet DIF analysis was difficult to interpret. Although several items displayed significant DIF, it was difficult to associate DIF with specific aspects of item content. Also, even when substantial item-level DIF was identified on the Anxiety facet, its effects when considered at the facet level appeared inconsequential. This latter finding illustrates that item-level DIF does not necessarily make a difference when a scale is considered as a whole.

GRAPHICAL IRT‐BASED DIF ANALYSES

ABSTRACTThe detection of differential item functioning (DIF) has become an important psychometric research topic in recent years. A number of item response theory (IRT) methods for solving this problem have been suggested. A common approach is to calculate some function of the area between item response curves estimated from the subpopulations of interest. While these methods relay overall item level DIF information, they do not indicate the location and magnitude of DIF along the ability continuum. In order to provide these important details, this paper presents a method for producing simultaneous confidence bands for the difference between item response curves. After these bands have been plotted, the size and regions of DIF are easily identified. Implementation considerations and illustrative examples are also given.