Differential Item Functioning Items Research Articles

Differential Item Functioning in Students Rating of Teaching Effectiveness Surveys in Higher Education According to Academic Disciplines: Data from a Saudi University

This study explored academic discipline as a source of differential item functioning(DIF) in students’ rating of teaching quality and effectiveness at higher education institutions. Data utilized in this study was collected by Imam Abudalruman Bin Faisal University - KSA. The total number of surveys analyzed for the purpose of this study is 36459 from three colleges: Education, Health, and Engineering. Using Extended Rasch model (Rating Scale Model), the results show that the instrument contains four DIF items. The content of these four items confirm the possibility of considering discipline as a source of DIF items in students evaluation of teaching in higher education. Moreover, the results of the current study show that removing DIF items from the instrument increases construct validity.

Differential Item Functioning in Students Rating of Teaching Effectiveness Surveys in Higher Education According to Academic Disciplines: Data from a Saudi University

This study explored academic discipline as a source of differential item functioning(DIF) in students’ rating of teaching quality and effectiveness at higher education institutions. Data utilized in this study was collected by Imam Abudalruman Bin Faisal University - KSA. The total number of surveys analyzed for the purpose of this study is 36459 from three colleges: Education, Health, and Engineering. Using Extended Rasch model (Rating Scale Model), the results show that the instrument contains four DIF items. The content of these four items confirm the possibility of considering discipline as a source of DIF items in students evaluation of teaching in higher education. Moreover, the results of the current study show that removing DIF items from the instrument increases construct validity.

An Exploratory Analysis of Differential Item Functioning and Its Possible Sources in a Higher Education Admissions Context

ABSTRACTWe analyzed a pool of items from an admissions test for differential item functioning (DIF) for groups based on age, socioeconomic status, citizenship, or English language status using Mantel-Haenszel and item response theory. DIF items were systematically examined to identify its possible sources by item type, content, and wording. DIF was primarily found in the citizenship group. As suggested by expert reviewers, possible sources of DIF in the direction of U.S. citizens was often in Quantitative Reasoning in items containing figures, charts, tables depicting real-world (as opposed to abstract) contexts. DIF items in the direction of non-U.S. citizens included “mathematical” items containing few words. DIF for the Verbal Reasoning items included geocultural references and proper names that may be differentially familiar for non-U.S. citizens. This study is responsive to foundational changes in the fairness section of the Standards for Educational and Psychological Testing, which now consider additional groups in sensitivity analyses, given the increasing demographic diversity in test-taker populations.

Recent advances in analysis of differential item functioning in health research using the Rasch model

BackgroundRasch analysis with a focus on Differential Item Functioning (DIF) is increasingly used for examination of psychometric properties of health outcome measures. To take account of DIF in order to retain precision of measurement, split of DIF-items into separate sample specific items has become a frequently used technique. The purpose of the paper is to present and summarise recent advances of analysis of DIF in a unified methodology. In particular, the paper focuses on the use of analysis of variance (ANOVA) as a method to simultaneously detect uniform and non-uniform DIF, the need to distinguish between real and artificial DIF and the trade-off between reliability and validity. An illustrative example from health research is used to demonstrate how DIF, in this case between genders, can be identified, quantified and under specific circumstances accounted for using the Rasch model.MethodsRasch analyses of DIF were conducted of a composite measure of psychosomatic problems using Swedish data from the Health Behaviour in School-aged Children study for grade 9 students collected during the 1985–2014 time periods.ResultsThe procedures demonstrate how DIF can be identified efficiently by ANOVA of residuals, and how the magnitude of DIF can be quantified and potentially accounted for by resolving items according to identifiable groups and using principles of test equating on the resolved items. The results of the analysis also show that the real DIF in some items does affect person measurement estimates.ConclusionsFirstly, in order to distinguish between real and artificial DIF, the items showing DIF initially should not be resolved simultaneously but sequentially. Secondly, while resolving instead of deleting a DIF item may retain reliability, both options may affect the content validity negatively. Resolving items with DIF is not justified if the source of the DIF is relevant for the content of the variable; then resolving DIF may deteriorate the validity of the instrument. Generally, decisions on resolving items to deal with DIF should also rely on external information.

Analysis of Item-Level Bias in the Bayley-III Language Subscales: The Validity and Utility of Standardized Language Assessment in a Multilingual Setting.

The purpose of this study was to improve standardized language assessments among bilingual toddlers by investigating and removing the effects of bias due to unfamiliarity with cultural norms or a distributed language system. The Expressive and Receptive Bayley-III language scales were adapted for use in a multilingual country (Singapore). Differential item functioning (DIF) was applied to data from 459 two-year-olds without atypical language development. This involved investigating if the probability of success on each item varied according to language exposure while holding latent language ability, gender, and socioeconomic status constant. Associations with language, behavioral, and emotional problems were also examined. Five of 16 items showed DIF, 1 of which may be attributed to cultural bias and another to a distributed language system. The remaining 3 items favored toddlers with higher bilingual exposure. Removal of DIF items reduced associations between language scales and emotional and language problems, but improved the validity of the expressive scale from poor to good. Our findings indicate the importance of considering cultural and distributed language bias in standardized language assessments. We discuss possible mechanisms influencing performance on items favoring bilingual exposure, including the potential role of inhibitory processing.

Differential item functioning by language on the PROMIS® physical functioning items for children and adolescents.

To assess the equivalence of self-reports of physical functioning between pediatric respondents to the English- and Spanish-language patient-reported outcomes measurement information system (PROMIS®) physical functioning item banks. The PROMIS pediatric physical functioning item banks include 29 upper extremity items and 23 mobility items. A sample of 5091 children and adolescents (mean age=12years, range 8-17; 49% male) completed the English-language version of the items. A sample of 605 children and adolescents (mean age=12years, range 8-17; 55% male; 96% Hispanic) completed the Spanish-language version of the items. We found language (English versus Spanish) differential item functioning (DIF) for 4 upper extremity items and 7 mobility items. Product-moment correlations between estimated upper extremity and mobility scores using the English versus the equated Spanish item parameters for Spanish-language respondents were 0.98 and 0.99, respectively. After excluding cases with significant person misfit, we found DIF for the same 4 upper extremity items that had DIF in the full sample and for 12 mobility items (including the same 7 mobility items that had DIF in the full sample). The identification of DIF items between English- and Spanish-language respondents was affected slightly by excluding respondents displaying person misfit. The results of this study provide support for measurement equivalence of self-reports of physical functioning by children and adolescents who completed the English- and Spanish-language surveys. Future analyses are needed to replicate the results of this study in other samples.

Improving the Assessment of Differential Item Functioning in Large-Scale Programs With Dual-Scale Purification of Rasch Models: The PISA Example.

By design, large-scale educational testing programs often have a large proportion of missing data. Since the effect of missing data on differential item functioning (DIF) assessment has been investigated in recent years and it has been found that Type I error rates tend to be inflated, it is of great importance to adapt existing DIF assessment methods to the inflation. The DIF-free-then-DIF (DFTD) strategy, which originally involved one single-scale purification procedure to identify DIF-free items, has been extended to involve another scale purification procedure for the DIF assessment in this study, and this new method is called the dual-scale purification (DSP) procedure. The performance of the DSP procedure in assessing DIF in large-scale programs, such as Program for International Student Assessment (PISA), was compared with the DFTD strategy through a series of simulation studies. Results showed the superiority of the DSP procedure over the DFTD strategy when tests consisted of many DIF items and when data were missing by design as in large-scale programs. Moreover, an empirical study of the PISA 2009 Taiwan sample was provided to show the implications of the DSP procedure. The applications as well as further studies of DSP procedure are also discussed.

Investigating the Effects of Differential Item Functioning on Proficiency Classification.

This study provides a comprehensive evaluation of the effects of differential item functioning (DIF) on proficiency classification. Using Monte Carlo simulation, item- and test-level DIF magnitudes were varied systematically to investigate their impact on proficiency classification at multiple decision points. Findings from this study clearly show that the presence of DIF affects proficiency classification not by lowering the overall correct classification rates but by affecting classification error rates differently for reference and focal group members. The study also reveals that multiple items with low levels of DIF can be particularly problematic. They can do similar damage to proficiency classification as high-level DIF items with the same cumulative magnitudes but are much harder to detect with current DIF and differential bundle functioning (DBF) techniques. Finally, how DIF affects proficiency classification errors at multiple cut scores is fully described and discussed.

Modified Logistic Regression Approaches to Eliminating the Impact of Response Styles on DIF Detection in Likert-Type Scales.

Extreme response styles (ERS) is prevalent in Likert- or rating-type data but previous research has not well-addressed their impact on differential item functioning (DIF) assessments. This study aimed to fill in the knowledge gap and examined their influence on the performances of logistic regression (LR) approaches in DIF detections, including the ordinal logistic regression (OLR) and the logistic discriminant functional analysis (LDFA). Results indicated that both the standard OLR and LDFA yielded severely inflated false positive rates as the magnitude of the differences in ERS increased between two groups. This study proposed a class of modified LR approaches to eliminating the ERS effect on DIF assessment. These proposed modifications showed satisfactory control of false positive rates when no DIF items existed and yielded a better control of false positive rates and more accurate true positive rates under DIF conditions than the conventional LR approaches did. In conclusion, the proposed modifications are recommended in survey research when there are multiple group or cultural groups.

Differential item functioning analysis of items on the Geriatric Depression Scale-15 based on the presence or absence of cognitive impairment

ABSTRACTObjective: The Geriatric Depression Scale-15 (GDS-15) is a screener for depressive symptoms in older adults. The present study aims to investigate the differential item functioning (DIF) of the GDS-15 items to determine whether or not they are biased by the presence of cognitive impairment.Method: Data from 215 older patients were used to examine the GDS-15. Individuals were categorized as cognitively impaired if they scored below the 10th percentile on the Mattis Dementia Rating Scale II. To evaluate DIF, configural invariance, metric invariance, scalar invariance, residual invariance, and factor variance were evaluated. Additional analyses were conducted to know the role identified DIF items play in the screening process.Results: Most levels of invariance indicated that items operated equivalently across groups (p > 0.05). However, analysis of scalar invariance indicated worse model fit (p = 0.001), such that the threshold for Item 13 differed between the groups. Freeing this threshold resulted in scalar invariance (p = 0.12).Conclusions: Because partial measurement invariance was achieved suggesting that the tool as a whole functions similarly for older adults with and without cognitive impairment, professionals can be confident that the GDS-15 screens for depression as well in individuals with cognitive impairment as those without.

Checking Equity: Why Differential Item Functioning Analysis Should Be a Routine Part of Developing Conceptual Assessments.

We provide a tutorial on differential item functioning (DIF) analysis, an analytic method useful for identifying potentially biased items in assessments. After explaining a number of methodological approaches, we test for gender bias in two scenarios that demonstrate why DIF analysis is crucial for developing assessments, particularly because simply comparing two groups' total scores can lead to incorrect conclusions about test fairness. First, a significant difference between groups on total scores can exist even when items are not biased, as we illustrate with data collected during the validation of the Homeostasis Concept Inventory. Second, item bias can exist even when the two groups have exactly the same distribution of total scores, as we illustrate with a simulated data set. We also present a brief overview of how DIF analysis has been used in the biology education literature to illustrate the way DIF items need to be reevaluated by content experts to determine whether they should be revised or removed from the assessment. Finally, we conclude by arguing that DIF analysis should be used routinely to evaluate items in developing conceptual assessments. These steps will ensure more equitable-and therefore more valid-scores from conceptual assessments.

The Effect of Background Variables on Gender DIF

The purpose of this study is to investigate the presence of differential item functioning (DIF) between gender groups in PISA 2015 science items in nine selected countries. Moreover, the effect of socioeconomic status, reading ability, science self-efficacy and science motivation on the presence of gender-related DIF are examined, respectively. One cluster from computer-based assessment (CBA) is taken into consideration. The countries are selected among the ones that implemented CBA, on the basis of their rank in science achievement. Multiple Indicator Multiple Causes method (MIMIC) is used in our analysis. DIF analysis in the MIMIC involves fit comparisons of both full and reduced models to determine if the items can measure the latent trait equally among the specified groups. The MIMIC analysis is conducted in two steps. First, the items is being tested for showing DIF among gender groups. Then the socioeconomic status, the reading ability, the science self-efficacy and the motivation are added to the model to test gender-related DIF items and their effects, respectively. According to results of the study, gender-related DIF appeared in all of the selected countries vary between two and six items. In four of the countries none of the selected variables significantly affect the presence of gender-related DIF. Instead, in the remaining countries the number of gender-related DIF items is reduced by adding different combinations of the selected variables to the model. The effect of variables which reduce the number of gender-related DIF items will discussed within each country.

For Which Boys and Which Girls Are Reading Assessment Items Biased Against? Detection of Differential Item Functioning in Heterogeneous Gender Populations

ABSTRACTIn gender differential item functioning (DIF) research it is assumed that all members of a gender group have similar item response patterns and therefore generalizations from group level to subgroup and individual levels can be made accurately. However DIF items do not necessarily disadvantage every member of a gender group to the same degree, indicating existence of heterogeneity of response patterns within gender groups. In this article the impact of heterogeneity within gender groups on DIF investigations was investigated. Specifically, it was examined whether DIF results varied when comparing males versus females, gender × socioeconomic status subgroups and latent classes of gender. DIF analyses were conducted on reading achievement data from the Canadian sample of the Programme of International Student Assessment 2009. Results indicated considerable heterogeneity within males and females and DIF results were found to vary when heterogeneity was taken into account versus when it was not.

Impact of Differential-Item-Functioning on the Personal Epistemological Beliefs for Senior High School Students

Abstract. The purpose of this study is to investigate whether the expression of items used to examine personal epistemological beliefs could affect the probabilities of response for compared groups and research outcomes. Differential-item-functioning (DIF) analysis on school types and location of school according to the Rasch model was performed in this study. Nonacademically inclined school students (n = 212) and academically inclined school students (n = 197) were selected to complete the questionnaire. The questionnaire consisted of three dimensions of beliefs about knowing and two dimensions of beliefs about learning. The results of the DIF analysis suggested that the items in the dimensions about knowing favor academically inclined students and students from schools located in the northern areas of Taiwan. The items in the learning dimensions favor nonacademically inclined students and students from schools located in the southern areas of Taiwan. The group comparisons were different between the scale that included the DIF items and the scale that excluded the DIF items. The discussion addresses the value of the precise detections for inappropriate items and the effect of the academic achievement on completing the survey.

Factor Structure of Sizing Me Up, a Self-Reported Weight-Related Quality of Life Instrument, in Community Children across Weight Status.

To further understand the psychometric properties of the Sizing Me Up-a newly developed quality of life (QoL) instrument related to weight status. We extended the target population to a community sample, including obese, overweight, normal-weight, and underweight children. Based on the data of 497 students in third to sixth grades, we used the following approaches: analysis of variance to detect the QoL scores among groups; confirmatory factor analysis (CFA) to examine the original structure; and Rasch analysis to test the misfit items, disordered response descriptors, and differential item functioning (DIF) items. Obese children had the lowest QoL total score; overweight children had lower QoL total score than normal-weight children; and underweight children had lower QoL in social avoidance than normal-weight children (all p < 0.01). CFA suggested that the factor structure is acceptable. Rasch corroborated that there were no misfit items; however, 10 items showed disordered thresholds in response descriptors and 6 items displayed significant DIF. Sizing Me Up can be applied to underweight children in the community and it can yield valid and reliable scores. Future studies are needed to explore the issues of disordered response descriptors and DIF items. Additional revisions for the Sizing Me Up may be warranted.

Correction of Differentially Functioning Items: Basis for Maintaining and Enhancing Test Validity and Reliability

This study looked into differentially functioning items in a Chemistry Achievement Test. It also&lt;br /&gt;examined the effect of eliminating differentially functioning items on the content and concurrent validity,&lt;br /&gt;and internal consistency reliability of the test. Test scores of two hundred junior high school students&lt;br /&gt;matched on school type were subjected to Differential Item Functioning (DIF) analysis. One hundred&lt;br /&gt;students came from a public school, while the other 100 were private school examinees. The&lt;br /&gt;descriptive-comparative research design utilizing differential item functioning analysis and validity and&lt;br /&gt;reliability analysis was employed. The Chi-Square, Distractor Response Analysis, Logistic Regression,&lt;br /&gt;and the Mantel-Haenszel Statistic were the methods used in the DIF analysis. A six-point scale ranging&lt;br /&gt;from inadequate to adequate was used to assess the content validity of the test. Pearson r was used in&lt;br /&gt;the concurrent validity analysis. The KR-20 formula was used for estimating the internal consistency&lt;br /&gt;reliability of the test. The findings revealed the presence of differentially functioning items between the&lt;br /&gt;public and private school examinees. The DIF methods differed in the number of differentially&lt;br /&gt;functioning items identified. However, there was a high degree of correspondence between the Logistic&lt;br /&gt;Regression and Mantel-Haenszel Statistic. After the elimination of the differentially functioning items,&lt;br /&gt;the content and the concurrent validity, and the internal consistency reliability differed per DIF method&lt;br /&gt;used. The content validity of the test differed ranging from slightly adequate to moderately adequate in&lt;br /&gt;the number of items retained. The concurrent validity of the test also differed but all were positive and&lt;br /&gt;indicate moderate relationship between the examinees’ test scores and their GPA in Science III.&lt;br /&gt;Likewise, the internal consistency reliability of the test differed. The more differentially functioning&lt;br /&gt;items eliminated, the lesser was the content and concurrent validity, and internal consistency reliability&lt;br /&gt;of the test becomes. Elimination of differentially functioning items diminishes content and concurrent&lt;br /&gt;validity, and internal consistency reliability, but could be use as basis in enhancing content, concurrent&lt;br /&gt;as well as internal consistency reliability by replacing eliminated DIF items.

Using Hospital Anxiety and Depression Scale (HADS) on patients with epilepsy: Confirmatory factor analysis and Rasch models

PurposeThe problems of mood disorders are critical in people with epilepsy. Therefore, there is a need to validate a useful tool for the population. The Hospital Anxiety and Depression Scale (HADS) has been used on the population, and showed that it is a satisfactory screening tool. However, more evidence on its construct validity is needed. MethodA total of 1041 people with epilepsy were recruited in this study, and each completed the HADS. Confirmatory factor analysis (CFA) and Rasch analysis were used to understand the construct validity of the HADS. In addition, internal consistency was tested using Cronbachs’ α, person separation reliability, and item separation reliability. Ordering of the response descriptors and the differential item functioning (DIF) were examined using the Rasch models. ResultsThe HADS showed that 55.3% of our participants had anxiety; 56.0% had depression based on its cutoffs. CFA and Rasch analyses both showed the satisfactory construct validity of the HADS; the internal consistency was also acceptable (α=0.82 in anxiety and 0.79 in depression; person separation reliability=0.82 in anxiety and 0.73 in depression; item separation reliability=0.98 in anxiety and 0.91 in depression). The difficulties of the four-point Likert scale used in the HADS were monotonically increased, which indicates no disordering response categories. No DIF items across male and female patients and across types of epilepsy were displayed in the HADS. ConclusionsThe HADS has promising psychometric properties on construct validity in people with epilepsy. Moreover, the additive item score is supported for calculating the cutoff.

Language‐related differential item functioning between English and German PROMIS Depression items is negligible

To investigate differential item functioning (DIF) of PROMIS Depression items between US and German samples we compared data from the US PROMIS calibration sample (n=780), a German general population survey (n=2,500) and a German clinical sample (n=621). DIF was assessed in an ordinal logistic regression framework, with 0.02 as criterion for R2 -change and 0.096 for Raju's non-compensatory DIF. Item parameters were initially fixed to the PROMIS Depression metric; we used plausible values to account for uncertainty in depression estimates. Only four items showed DIF. Accounting for DIF led to negligible effects for the full item bank as well as a post hoc simulated computer-adaptive test (< 0.1 point on the PROMIS metric [mean=50, standard deviation =10]), while the effect on the short forms was small (< 1 point). The mean depression severity (43.6) in the German general population sample was considerably lower compared to the US reference value of 50. Overall, we found little evidence for language DIF between US and German samples, which could be addressed by either replacing the DIF items by items not showing DIF or by scoring the short form in German samples with the corrected item parameters reported.

Effects of Differential Item Functioning on Examinees' Test Performance and Reliability of Test

Simulations were conducted to examine the effect of differential item functioning (DIF) on measurement consequences such as total scores, item response theory (IRT) ability estimates, and test reliability in terms of the ratio of true-score variance to observed-score variance and the standard error of estimation for the IRT ability parameter. The objective was to provide bounds of the likely DIF effects on these measurement consequences. Five factors were manipulated: test length, percentage of DIF items per form, item type, sample size, and level of group ability difference. Results indicate that the greatest DIF effect was less than 2 points on the 0 to 60 total score scale and about 0.15 on the IRT ability scale. DIF had a limited effect on the ratio of true-score variance to observed-score variance, but its influence on the standard error of estimation for the IRT ability parameter was evident for certain ability values.

Testing whether patients with diabetes and healthy people perceive the meaning of the items in the Persian version of the SF-36 questionnaire similarly: a differential item functioning analysis

It has been rarely studied whether observed disparity in health-related quality-of-life (HRQoL) scores between patients with diabetes and healthy individuals is due to differential item functioning (DIF) or a true difference in the underlying construct. This study aimed to examine DIF in the SF-36 questionnaire and its effect on comparing HRQoL scores between patients with diabetes and healthy people. The sample consisted of 230 patients with type 2 diabetes and 642 healthy individuals who filled out the Persian version of the SF-36 questionnaire. To detect DIF across patients with diabetes and healthy individuals, multiple-group multiple-indicator multiple-causes model was used. In addition, item calibration strategy was used to determine whether the effect of item-level DIF was transferred to the scale level. Nine out of thirty-six (25%) items were detected as DIF, of which one item (11%) was flagged as uniform and eight items (89%) as non-uniform DIF. Most of the DIF items were detected in the mental health component which includes vitality, perceived mental health and social functioning subscales rather than in physical health component. Moreover, nonsignificant latent mean differences for general health perception and social functioning subscales became significant after DIF calibration. The findings of the present study show that patients with diabetes and healthy individuals perceived some items in the SF-36 questionnaire differently. More importantly, in some subscales, the effect of item-level DIF was transferred to the scale level. Consequently, considerable caution should be taken in comparing HRQoL scores between patients with diabetes and healthy individuals.