Large Differential Item Functioning Research Articles

Cognitive diagnostic assessment of reading comprehension for high- stakes tests: Using GDINA model

Cognitive diagnostic assessment (CDA) is used to study cognitive and educational psychology, and designed to diagnose the underlying abilities of test takers in comprehension language skills such as reading comprehension. Through applying CDA, a test has undergone accurate studies to remove biased test items which yield great impact on individuals, educational systems and societies. In this case, psychometric statistical analyses were applied, Differential Attribute Functioning (DAF)) was also used to detect the probability of the mastery of attributes among test takers, and Differential Item Functioning (DIF) was estimated to show item performance among different candidates in terms of gender, their GPAs in BA, and MA degrees. The randomly selected participants of this study were 7,420 females and males sitting for the nationwide PhD admission test to pursue their education in Applied Linguistics. Moreover, a Q-matrix was developed, data were fed into R studio software, and the Generalized Deterministic Inputs, Noisy “and” Gate (GDINA) model was run. The results of the study flagged large DIF in gender group in 2019; and in gender, BA, and MA groups in 2020. In sum, this study is an attempt to raise the awareness of test developers to shed light on the critical discursive sources of inequity and bias. The implication of this study can provide pedagogically useful diagnostic information for test designers and teachers since a proficiency test needs to be valid, reliable, and fair in the context of high-stakes tests so that it would lead to positive changes.

Measuring Personality Systems in Older Adults: Age-Neutrality of the Dutch BIS/BAS Questionnaire.

With a rapidly aging population, age-neutrality of psychological instruments is becoming ever more important for accurate assessment of older adults. This study aims to test the age-neutrality of the Dutch Behavioral Inhibition System/Behavioral Approach System (BIS/BAS) scales by means of Differential Item Functioning (DIF) and Differential Test Functioning (DTF) analyses. DIF and DTF analyses were conducted using an odds ratio approach. Potential DIF was examined for the two primary scales and three BAS subscales in 390 Dutch-speaking participants divided over three age groups. Overall, the BIS-BAS scales lacked age-neutrality when comparing older adults to young adults: eight out of 20 items (ie, 40%) displayed DIF according to the adjusted Bonferroni corrected cut-offs, which exceeds the threshold of 25% for large DIF. Thus, for 40% of the test items, items were differently endorsed by young and older adults with the same position on the construct measured by the item. Therefore, the implications of the item-level DIF on the scale level were investigated across age groups. DTF analyses revealed large DTF for all BIS and BAS scales according to the adjusted Bonferroni corrected cut-offs. DIF found in items on the BIS scale and BAS-Drive and BAS-Fun Seeking scale can probably be explained by a difference in degree of expression across age groups. A solution could be the development of age-specific norms. DIF on BAS-Reward Responsiveness subscale could be partly due to measuring a different construct across age groups. Rephrasing or replacing such items with DIF may increase age-neutrality of the BIS/BAS Scales.

Assessing Indonesian student inductive reasoning: Rasch analysis

Inductive reasoning is an ability related to student academic achievement and is embedded in 21st-century competency frameworks. The purpose of this study was to evaluate Indonesian students’ inductive reasoning, validate an adapted inductive reasoning test for Indonesia, and classify the difficulty of items and students’ reasoning ability. The participants were 856 students in Grades 10, 11, and 12 in senior high schools as well as undergraduate students in higher educational institutions in West Kalimantan province, Indonesia. Four different tasks were employed to assess student inductive reasoning. The data collection comprised an onlinetest through the eDia assessment platform and a traditional paper-based test. The results from the Rasch analysis demonstrated that the adapted inductive reasoning test met the validity and reliability criteria based on Rasch parameters. Differential item functioning (DIF) analysis revealed that only one of the 40 items is in the moderate to large DIF category based on the test method. The results further revealed that students performed better in solving figural items than numeric items. Furthermore, older students in higher grades had higher logit measures than younger students, however, no significant differences were found between students in Grades 10 and 11. No significant differences were found between females and males. The difficulty of items and students’ abilities were also classified to understand the evaluation of students’ inductive reasoning.

Implementing a Standardized Effect Size in the POLYSIBTEST Procedure.

A study was conducted to implement the use of a standardized effect size and corresponding classification guidelines for polytomous data with the POLYSIBTEST procedure and compare those guidelines with prior recommendations. Two simulation studies were included. The first identifies new unstandardized test heuristics for classifying moderate and large differential item functioning (DIF) for polytomous response data with three to seven response options. These are provided for researchers studying polytomous data using POLYSIBTEST software that has been published previously. The second simulation study provides one pair of standardized effect size heuristics that can be employed with items having any number of response options and compares true-positive and false-positive rates for the standardized effect size proposed by Weese with one proposed by Zwick et al. and two unstandardized classification procedures (Gierl; Golia). All four procedures retained false-positive rates generally below the level of significance at both moderate and large DIF levels. However, Weese's standardized effect size was not affected by sample size and provided slightly higher true-positive rates than the Zwick et al. and Golia's recommendations, while flagging substantially fewer items that might be characterized as having negligible DIF when compared with Gierl's suggested criterion. The proposed effect size allows for easier use and interpretation by practitioners as it can be applied to items with any number of response options and is interpreted as a difference in standard deviation units.

Investigating the Distractors to Explain DIF Effects Across Gender in Large-Scale Tests With Non-Linear Logistic Regression Models

The purpose of this study is to examine the distractors of items that exhibit differential item functioning (DIF) across gender to explain the possible sources of DIF in the context of large-scale tests. To this end, two non-linear logistic regression (NLR) models-based DIF methods (three parameters, 3PL-NLR and four-parameter, 4PL-NLR) were first used to detect DIF items, and the Mantel-Haenszel Delta (MH-Delta) DIF method was used to calculate the DIF effect size for each DIF item. Then, the multinomial log-linear regression (MLR) model and 2-PL nested logit model (2PL-NLM) were applied to items exhibiting DIF with moderate and large DIF effect sizes. The ultimate goals are (a) to examine behaviors of distractors across gender and (b) to investigate if distractors have any impact on DIF effects. DIF results of the Art Section of the General Aptitude Test (GAT-ART) based on both 3PL-NLR and 4PL-NLR methods indicate that only 10 DIF items had moderate to large DIF effects sizes. According to MLR differential distractor functioning (DDF) results, all items exhibited DDF across gender except for one item. An interesting finding of this study is that DIF items related to the verbal analogy and context analysis were in favor of female students, while all DIF items related to the reading comprehension subdomain were in favor of male students, which may signal the existence of content specific DIF or true ability difference across gender. DDF results show that distractors have a significant effect on DIF results. Therefore, DDF analysis is suggested along with DIF analysis since it signals the possible causes of DIF.

Detecting Differential Item Functioning Using Cognitive Diagnosis Models: Applications of the Wald Test and Likelihood Ratio Test in a University Entrance Examination

ABSTRACT The present study aims to examine gender differential item functioning (DIF) in the reading comprehension section of a high stakes test using cognitive diagnosis models. Based on the multiple-group generalized deterministic, noisy “and” gate (MG G-DINA) model, the Wald test and likelihood ratio test are used to detect DIF. The flagged items are further inspected to find the attributes they measure, and the probabilities of correct response are checked across latent profiles to gain insights into the potential reasons for the occurrence of DIF. In addition, attribute and latent class prevalence are examined across males and females. The three items displaying large DIF involve three attributes, namely Vocabulary, Main Idea, and Details. The results indicate that females have lower probabilities of correct response across all latent profiles, and fewer females have mastered all the attributes. Moreover, the findings show that the same attribute mastery profiles are prevalent across genders. Finally, the results of the DIF analysis are used to select models that could replace the complex MG G-DINA without significant loss of information.

Partial Measurement Invariance: Extending and Evaluating the Cluster Approach for Identifying Anchor Items.

When measurement invariance does not hold, researchers aim for partial measurement invariance by identifying anchor items that are assumed to be measurement invariant. In this paper, we build on Bechger and Maris’s approach for identification of anchor items. Instead of identifying differential item functioning (DIF)-free items, they propose to identify different sets of items that are invariant in item parameters within the same item set. We extend their approach by an additional step in order to allow for identification of homogeneously functioning item sets. We evaluate the performance of the extended cluster approach under various conditions and compare its performance to that of previous approaches, that are the equal-mean difficulty (EMD) approach and the iterative forward approach. We show that the EMD and the iterative forward approaches perform well in conditions with balanced DIF or when DIF is small. In conditions with large and unbalanced DIF, they fail to recover the true group mean differences. With appropriate threshold settings, the cluster approach identified a cluster that resulted in unbiased mean difference estimates in all conditions. Compared to previous approaches, the cluster approach allows for a variety of different assumptions as well as for depicting the uncertainty in the results that stem from the choice of the assumption. Using a real data set, we illustrate how the assumptions of the previous approaches may be incorporated in the cluster approach and how the chosen assumption impacts the results.

Gender Differences in Mathematics and Science Achievement of Eighth Grade Students in Ukraine on TIMSS 2011

The purpose of the paper is in-depth analysis of gender differences in Ukrainian student achievement in mathematics and science on TIMSS 2011 results because the average scores similarity hides some significant differences between the achievements of boys and girls. Boys significantly outperform girls in some cognitive and content domains, while the benefit of girls never was significant. The procedure of Differential Item Functioning (DIF) on the basis of the Mantel-Haenszel method was used to test the fairness of assessment. The results show that the set of TIMSS 2011 items does not have any gender bias on Ukrainian students since there are insignificant number of items with large DIF, among which some favouring boys, and some favouring girls. Each of gender groups is equally represented among both stronger and weaker participants. Any significant relationship between the DIF size and the difficulty or type of the item is not detected. But the items in number, physics and earth science, and math problems on applying function in favour of boys. This is likely caused by the fact that in Ukrainian society still there is attitude to such fields of study as masculine.

A Differential Item Functioning (DIF) Study of the Infection Control Standard Isolation Precaution Instrument Across Gender and Major Among Healthcare Workers

Background: Measuring healthcare workers’ (HCWs) knowledge, attitudes, and practices (KAPs) regarding isolation precaution is essential for infection control which needs a valid and reliable instrument. Objectives: This study aimed to assess differential item functioning (DIF) across gender and major for the knowledge and practice items of the questionnaire, previously designed in Shiraz, Iran. Methods: This cross-sectional survey was conducted on 1070 participants (males/females: 306/764; medical students/nurses: 466/624). The study instrument had three subscales with nine questions for each KAP subscale. The Mantel-Haenszel (MH) statistic was used. The DIF and differential test functioning (DTF) analyses were also performed in this study. Results: There were very similar DIF outcomes for the knowledge and practice subscales, with one or two items indicating moderate DIF but comparable total scores across genders. Across majors, several items showed large DIF for both subscales. It was found that large DTF affects major for both subscales. Conclusions: Our findings indicated large DIF and DTF levels of the questionnaire among medical students and nursing groups. More attention should be paid when developing the items. This study shows the importance of paying attention to valid evidence for instruments developed within the field of healthcare.

Examining provision and sufficiency of testing accommodations for English learners

Accommodations are commonly provided to address language-related barriers students may experience during testing. Research on the validity of scores from accommodated test administrations remains somewhat inconclusive. The current study investigated item response patterns to understand whether accommodations, as used in practice among English learners (ELs) in the United States, allow for comparable measurement between ELs and non-ELs. Results indicated that although significant differences are evident in overall test scores for ELs and non-ELs, only minimal measurement concerns were evident. Very few items displayed moderate or large differential item functioning (DIF); no tests showed small, medium, or large differential test functioning. The current study adds to existing literature on measurement comparability and accommodation research on ELs; implications for practice are provided.

Detection of Sex-Related Differential Item Functioning in Raven’s Standard Progressive Matrices Test Using the Mantel-Haenszel Method

This study aimed to determine the presence of sex-related differential item functioning (DIF) in Raven’s Standard Progressive Matrices (SPM) Test. The research design was a comparative study, where boys formed the focal group and girls formed the reference group. The software used was SPSS-v26, and binary data of focal and reference groups were analyzed using jMetrik software to detect DIF according to the Mantel-Haenszel method. A total of 1032 students (49.6% boys and 50.4% girls), 570 from intermediate school and 462 from secondary school, were selected from 24 schools using a stratified random-sampling procedure. The statistical analyses showed that the Raven’s Standard Progressive Matrices (SPM) Test was one-dimensional. The results showed that, of five moderate DIF items, four items favored the focal group (boys), and one item favored the reference group (girls). The results also showed two large DIF items; the direction of the one item favored the reference group (girls), and the direction of the second item favor the focal group (boys). The findings showed that there were several unbiased items, but some were clearly biased against female performance. According to these findings, we recommend reanalyzing the data using methods depending on the item response theory, such as the logistic regression, simultaneous item bias test (SIB) or IRT-likelihood ratio (IRT-LR) methods to confirm the results seen here. Keywords : Differential Item Functioning (DIF), Bias Item, Reference Group, Focal Group DOI: 10.7176/JEP/11-29-10 Publication date: October 31 st 2020

Differential Item Functioning Effect Size From the Multigroup Confirmatory Factor Analysis for a Meta-Analysis: A Simulation Study.

This study presents a new approach to synthesizing differential item functioning (DIF) effect size: First, using correlation matrices from each study, we perform a multigroup confirmatory factor analysis (MGCFA) that examines measurement invariance of a test item between two subgroups (i.e., focal and reference groups). Then we synthesize, across the studies, the differences in the estimated factor loadings between the two subgroups, resulting in a meta-analytic summary of the MGCFA effect sizes (MGCFA-ES). The performance of this new approach was examined using a Monte Carlo simulation, where we created 108 conditions by four factors: (1) three levels of item difficulty, (2) four magnitudes of DIF, (3) three levels of sample size, and (4) three types of correlation matrix (tetrachoric, adjusted Pearson, and Pearson). Results indicate that when MGCFA is fitted to tetrachoric correlation matrices, the meta-analytic summary of the MGCFA-ES performed best in terms of bias and mean square error values, 95% confidence interval coverages, empirical standard errors, Type I error rates, and statistical power; and reasonably well with adjusted Pearson correlation matrices. In addition, when tetrachoric correlation matrices are used, a meta-analytic summary of the MGCFA-ES performed well, particularly, under the condition that a high difficulty item with a large DIF was administered to a large sample size. Our result offers an option for synthesizing the magnitude of DIF on a flagged item across studies in practice.

Social deprivation among socio-economic contrasted french areas: Using item response theory analysis to assess differential item functioning of the EPICES questionnaire in stroke patients.

BackgroundMultiple approaches have been proposed to measure low socio-economic status. In France the concept of precariousness, akin to social deprivation, was developed and is widely used. EPICES is a short questionnaire that was developed to measure this concept. This study aimed to evaluate Differential Item Functioning (DIF) in the EPICES questionnaire between contrasted areas: mainland France, French West Indies (FWI) and French Guiana (FG).MethodsThe population was taken from the INDIA study, which aimed to evaluate the impact of social inequalities on stroke characteristics and prognosis. Eligible people were patients referred to neurology or emergency departments for a suspicion of stroke. We assessed the DIF using hybrid ordinal logistic regression method, derived from item response theory.ResultsWe analysed 1 553 stroke patients, including 768 from FWI (49.5%), 289 from FG (18.6%) and 496 from mainland (31.9%). We identified five items with a moderate to large DIF in area comparisons: “meeting with a social worker”, “complementary health insurance”, “home-owning”, “financial difficulties” and “sport activities”. Correlation between EPICES score and the latent variable was strong (r = 0.84).ConclusionThis is the first attempt to assess the DIF of the EPICES score between different French populations. We found several items with DIF, which can be explained by individual interpretation or local context. However, the DIFs did not lead to a large difference between the latent variable and the EPICES score, which indicates that it can be used to assess precariousness and social deprivation between contrasted areas.

Detecting Differential Item Functioning and Differential Step Functioning Due to Differences that Should Matter

This study illustrates the use of differential item functioning (DIF) and differential step functioning (DSF) analyses to detect differences in item difficulty that are related to experiences of examinees, such as their teachers’ instructional practices, that are relevant to the knowledge, skill, or ability the test is intended to measure. This analysis is in contrast to the typical use of DIF or DSF to detect differences related to characteristics of examinees, such as gender, language, or cultural knowledge, that should be irrelevant. Using data from two forms of Ontario’s Grade 9 Assessment of Mathematics, analyses were performed comparing groups of students defined by their teachers’ instructional practices. All constructed-response items were tested for DIF using the Mantel Chi-Square, standardized Liu Agresti cumulative common log-odds ratio, and standardized Cox’s noncentrality parameter. Items exhibiting moderate to large DIF were subsequently tested for DSF. In contrast to typical DIF or DSF analyses, which inform item development, these analyses have the potential to inform instructional practice.

Successful use of the Unified Dyskinesia Rating Scale regardless of PD- or dyskinesia-duration

ObjectiveWe assessed differential item functioning (DIF) in the Unified Dyskinesia Rating Scale (UDysRS) to evaluate bias risk from the duration of Parkinson's Disease (PD) and duration of dyskinesia. BackgroundAssessing DIF is a core validation step for rating scales. If DIF is present for an item, interpretation must consider influences from the tested covariates. DIF can be uniform or non-uniform, depending on the consistency of influence from the given covariate across all levels of dyskinesia. MethodsUsing a large UDysRS database (N = 2313), uniform and non-uniform DIF related to the duration of PD and/duration of dyskinesia were tested. Unidimensionality of UDysRS was first confirmed using confirmatory factor analysis. DIF analysis was conducted using two independent latent models. DIF in an item was confirmed if both methods independently identified DIF at a significance level using Bonferroni correction. McFadden pseudo R^2 measured clinical relevancy of DIF magnitude (negligible, moderate, and large) for items identified with DIF, and items with DIF were considered clinically relevant if they exceeded a negligible designation. ResultsMost items did not show uniform or non-uniform DIF based on PD and dyskinesia duration in isolation or in combination. For all items where DIF was identified, the magnitude statistic was in the negligible range (McFadden pseudo R^2 < 0.035) and the combined impact of multiple identified DIF items on UDysRS likewise did not exceed the negligible designation. ConclusionThe absence of clinically relevant DIF suggests that the UDysRS can be applied across all patients regardless of their PD- or dyskinesia-duration.

The Adaptation and Psychometric Examination of a Social-Emotional Developmental Screening Tool in Taiwan

ABSTRACTResearch Findings: The Ages & Stages Questionnaires: Social-Emotional, Second Edition (ASQ:SE-2) was translated and adapted into Traditional Chinese in Taiwan. A sample of 1,455 children, ranging from 42 months 0 days to 53 months 30 days old, reflecting the population sizes of different regions in Taiwan, completed the 48-month ASQ:SE-2. Data were analyzed by item response theory modeling. A multidimensional Rasch Partial Credit Model was chosen for data analysis. Differential item functioning (DIF) was used to explore the difference between the Traditional Chinese ASQ:SE-2 and the English ASQ:SE-2 (N = 3,005) administered in the U.S. Results indicated that (a) item fit statistics was between 0.88 – 1.26 (M = 1.00, SD = 0.10), (b) difficulty was between -0.79 – 3.19 (M = 2.06, SD = 0.84), (c) reliability was 0.79 for all items, 0.75/0.74 for Emotion/Sociality dimension, and (d) six out of 35 items (17.1%) with moderate to large DIF. Practice or Policy: This research provided psychometric evidence for using the ASQ:SE-2-TC with a Taiwanese population. The promising psychometric findings encourage the further investment on validating ASQ:SE-2-TC. The cultural explanations can inform test developers to become aware of the potential influence from social values, parenting style, or childrearing practices.

Item response modeling: a psychometric assessment of the children\u2019s fruit, vegetable, water, and physical activity self-efficacy scales among Chinese children

BackgroundThis study aimed to evaluate the psychometric properties of four self-efficacy scales (i.e., self-efficacy for fruit (FSE), vegetable (VSE), and water (WSE) intakes, and physical activity (PASE)) and to investigate their differences in item functioning across sex, age, and body weight status groups using item response modeling (IRM) and differential item functioning (DIF).MethodsFour self-efficacy scales were administrated to 763 Hong Kong Chinese children (55.2% boys) aged 8-13 years. Classical test theory (CTT) was used to examine the reliability and factorial validity of scales. IRM was conducted and DIF analyses were performed to assess the characteristics of item parameter estimates on the basis of children’s sex, age and body weight status.ResultsAll self-efficacy scales demonstrated adequate to excellent internal consistency reliability (Cronbach’s α: 0.79-0.91). One FSE misfit item and one PASE misfit item were detected. Small DIF were found for all the scale items across children’s age groups. Items with medium to large DIF were detected in different sex and body weight status groups, which will require modification. A Wright map revealed that items covered the range of the distribution of participants’ self-efficacy for each scale except VSE.ConclusionsSeveral self-efficacy scales’ items functioned differently by children’s sex and body weight status. Additional research is required to modify the four self-efficacy scales to minimize these moderating influences for application.

Gambling-Related Cognition Scale (GRCS): Are skills-based games at a disadvantage?

The Gambling-Related Cognition Scale (GRCS; Raylu & Oei, 2004) was developed to evaluate gambling-related cognitive distortions for all types of gamblers, regardless of their gambling activities (poker, slot machine, etc.). It is therefore imperative to ascertain the validity of its interpretation across different types of gamblers; however, some skills-related items endorsed by players could be interpreted as a cognitive distortion despite the fact that they play skills-related games. Using an intergroup (168 poker players and 73 video lottery terminal [VLT] players) differential item functioning (DIF) analysis, this study examined the possible manifestation of item biases associated with the GRCS. DIF was analyzed with ordinal logistic regressions (OLRs) and Ramsay's (1991) nonparametric kernel smoothing approach with TestGraf. Results show that half of the items display at least moderate DIF between groups and, depending on the type of analysis used, 3 to 7 items displayed large DIF. The 5 items with the most DIF were more significantly endorsed by poker players (uniform DIF) and were all related to skills, knowledge, learning, or probabilities. Poker players' interpretations of some skills-related items may lead to an overestimation of their cognitive distortions due to their total score increased by measurement artifact. Findings indicate that the current structure of the GRCS contains potential biases to be considered when poker players are surveyed. The present study conveys new and important information on bias issues to ponder carefully before using and interpreting the GRCS and other similar wide-range instruments with poker players. (PsycINFO Database Record

A Differential Item Functioning Analysis of the EQ-5D in Cancer.

To determine whether differential item functioning (DIF) was present in the EuroQol five-dimensional questionnaire (EQ-5D) used in cancer (non-small cell lung cancer and prostate cancer). The Partial Credit Model was applied to the three-level version of the EQ-5D with data obtained from four randomized controlled trials in prostate cancer and non-small cell lung cancer completed at baseline before treatment (N = 2213). DIF was assessed across cancer type (two levels), sex (two levels), and age group (three levels) using Mantel-Haenszel chi-square statistics and evaluated against the Educational Testing Service classification rules. The presence of DIF was determined in 14 of 25 (56%) potential DIF contrasts in all the EQ-5D domains. Although mostly the DIF was categorized as either negligible (3 of 25 [12%]) or medium (7 of 25 [28%]), large DIF was observed in 4 of the 25 contrasts (16%). The mobility domain, in particular, showed consistently large DIF across cancer type, sex, and age. Given the use of the instrument in health status assessments across conditions and interventions, these results may have significant implications for the EQ-5D in health economic evaluations. Further research is warranted to determine whether these results hold for other cancers.

Differential Performance on National Exams

The purpose of this study is to evaluate two methodological perspectives of test fairness using a national Secondary School Certificate (SSC) examinations. SSC is a suit of multi-subject national qualification tests at Grade 10 level in South Asian countries, such as Bangladesh, India, and Pakistan. Because it is a high-stakes test, the fairness of SSC tests is a major concern among public and educational policy planners. This study is a first attempt to investigate test fairness of the national SSC examination of Pakistan using two independent differential item functioning (DIF) and differential bundle functioning (DBF) procedures. The SSC was evaluated for possible gender bias using multiple-choice tests in three core subjects, namely, English, Mathematics, and Physics. The study was conducted in two phases using explanatory item response model (EIRM) and Simultaneous Item Bias Test (SIBTEST). In Phase 1, test items were studied for DIF, and items with severe DIF were flagged in each subject. In Phase 2, the item bundles were analyzed for DBF. Three items were detected with large DIF, one for each subject, and one item bundle was detected with a negligible DBF. Taken together, the results demonstrate that there is no major threat to the validity of the interpretation of examinees’ test scores on the SSC examination. The outcome from this study provided evidence for test fairness, which will enhance test development practices at the national examination authorities.