Sources Of Differential Item Functioning Research Articles

Examining Sources of Gender DIF in Mathematics Assessments Using a Confirmatory Multidimensional Model Approach

This study contributes to understanding sources of gender differential item functioning (DIF) on mathematics tests. This study focused on identifying sources of DIF and differential bundle functioning for boys and girls on the British Columbia Principles of Mathematics Exam (Grade 12) using a confirmatory SIBTEST approach based on a multidimensional model. Problem solving as a content area was confirmed as a source of gender DIF in favor of boys when the item is presented in the form of a story problem or when the problems are noncontext specific. Patterns and relations content areas produced a mixture of confirmed sources of DIF, with some subtopics favoring the girls and some favoring the boys. In contrast to what might be expected given the findings of previous gender DIF research, this study did not find geometry to be a source of gender DIF. All of the higher cognitive level items favored boys. High levels of DIF were detected in favor of girls on the bundle of computation items in which no equations were provided in the question.

Using Multiple-Variable Matching to Identify Cultural Sources of Differential Item Functioning

Identifying the sources of differential item functioning (DIF) in international assessments is very challenging, because such sources are often nebulous and intertwined. Even though researchers frequently focus on test translation and content area, few actually go beyond these factors to investigate other cultural sources of DIF. This article introduces the multiple-variable matching method using logistic regression analysis to identify sources of DIF. A case study demonstrates how this methodology identified Extra Lesson Hours After School (ELHAS) as a potential source of DIF between Taiwan and the United States in the Third International Mathematics and Science Study (TIMSS) 1999. DIF is not a fixed character of any test item, nor is a cultural factor an inherent source of DIF. The legitimacy of a source of DIF relies on the specific context and purpose for the cross-country comparison.

Gender Differences in Physical Self-Concept: A Multidimensional Differential Item Functioning Analysis

Typically, group differences are analyzed at the subdomain or test level using composite scores. This can mask the effect of individual items across groups. For example, two items from the Physical Self-Description Questionnaire (PSDQ) are worded in terms of internal (“I am good looking”) and external (“Nobody thinks I’m good looking”) frames of reference. Any difference between the group means using a composite score including these two items is interpreted globally, thus obscuring individual item meaning. A better approach to detecting group differences is to work at the item level using differential item functioning (DIF) methods. One such method is multidimensional DIF, which identifies any secondary unintended latent dimensions at the item level that may lead to between-group score differences. The results suggest that 28 of the PSDQ’s 70 items (40%) are biased in favor of either men (17%) or women (23%) and that a primary source of DIF relates to item composition (negative wording) and internal or external frames of reference.

Linguistic sources of item bias for second generation immigrants in Dutch tests

This article reports the first results of a long-term research project focusing on the detection and possible linguistic causes of differential item functioning (DIF) for second generation immigrant students in the Final Test of Primary Education in the Netherlands. The main aim of the project is to provide test constructors with information which is as detailed as possible about sources of DIF in order to help them avoid item bias in future forms of the test. The project was carried out in three steps. First, DIF items were identified using two statistical procedures: IRT and Mantel-Haenszel. The second stage was an investigation of sources of DIF, beginning with an inventory of subject matter elements which can cause difficulty for immigrant students, according to previous educational research. This was followed by a careful evaluation of DIF items by the researchers and by external experts, and a think-aloud experiment with students. At the third stage, judges decided whether each DIF item was biased and thus reduced the construct validity of the test. The article concentrates on the classification of linguistic sources of DIF and a consideration of whether they work to the advantage or disadvantage of immigrant students.

Onderzoek Naar Oorzaken van 'Differential Item Functioning' in de Eindtoets Basisonderwijs

Previous research has shown that the Dutch 'Final Test of Primary Education' contains a number of unintentionally and therefore unwanted, difficult test items, leading to Differential Item Functioning (DIF) for immigrant minority students whose parents' dominant language is Turkish or Arab/Berber. Two statistical procedures were used to identify DIF-items in the Final Test of 1997. Subsequently, five experiments were conducted to detect causes of DIF, revealing a number of hypotheses concerning possible linguistic, cultural, and textual sources. These hypotheses were used to manipulate original DIF-items into intentionally DIF-free items. The article discusses three possible sources of DIF: (1) the use of fixed (misleading) answer-options and (2) of misleading illustrations (both in the disadvantage of the minority students), and (3) the fact that questions concerning past tense often lead to DIF (in their advantage).

DIF Detection and Interpretation in Large-Scale Science Assessments: Informing Item Writing Practices

Differential item functioning (DIF) analyses are a routine part of the development of large-scale assessments. Less common are studies to understand the potential sources of DIF. The goals of this study were (a) to identify gender DIF in a large-scale science assessment and (b) to look for trends in the DIF and non-DIF items due to content, cognitive demands, item type, item text, and visual-spatial or reference factors. To facilitate the analyses, DIF studies were conducted at 3 grade levels and for 2 randomly equivalent forms of the science assessment at each grade level (administered in different years). The DIF procedure itself was a variant of the "standardization procedure" of Dorans and Kulick (1986) and was applied to very large sets of data (6 sets of data, each involving 60,000 students). It has the advantages of being easy to understand and to explain to practitioners. Several findings emerged from the study that would be useful to pass on to test development committees. For example, when there was DIF in science items, MC items tended to favor male examinees and OR items tended to favor female examinees. Compiling DIF information across multiple grades and years increases the likelihood that important trends in the data will be identified and that item writing practices will be informed by more than anecdotal reports about DIF.

Revising Translated Differential Item Functioning Items as a Tool for Improving Cross-Lingual Assessment

When a test is translated from a source language into a target language, the 2 tests are generally not psychometrically equivalent. Differential item functioning (DIF) analysis can facilitate an understanding of the differences between the tests in the 2 languages and can help translators to revise translated items to produce non-DIF (or lower DIF) items. Based on the reasons found for DIF in Allalouf, Hambleton, and Sireci (1999; differences in word difficulty, item format, content, and cultural relevance), a panel of translators and researchers revised the target-language DIF items detected in that study. The revised items were then readministered. Results showed that the revisions succeeded in reducing DIF considerably. For cross-validation purposes, the original translated items were readministered and the study results were generally validated. An attempt was made to determine which sources of DIF and which item types could be revised most effectively. The revision process is expensive, so considerations of cost-effectiveness should be taken into account. Revisions to reduce DIF in translated items may follow the steps used in this study.

Disentangling Sources of Differential Item Functioning in Multilanguage Assessments

This article describes and discusses strategies used in disentangling sources of differential item functioning (DIF) in multilanguage assessments where multiple factors are expected to be causing DIF. Three strategies are used for identifying adaptation and curricular differences as sources of DIF: (a) judgmental reviews by multiple bilingual translators of all items, (b) cross-validation of DIF in multiple groups, and (c) examination of the distribution of DIF items by topic. Twenty-seven percent of the mathematics DIF items and 37% of the science DIF items were interpreted to be due to adaptation-related differences based on judgmental reviews. Most of these interpretations were also supported by the cross-validation analyses. Clustering of DIF items by topic provided curricular differences as interpretation for DIF only for small portions of the DIF items, approximately 23% of the mathematics DIF items and 13% of the science DIF items.

Disentangling Sources of Differential Item Functioning in Multilanguage Assessments

Disentangling Sources of Differential Item Functioning in Multilanguage Assessments

Analysis of Differential Item Functioning (DIF) Using Hierarchical Logistic Regression Models

Over the past 25 years a range of parametric and nonparametric methods have been developed for analyzing Differential Item Functioning (DIF). These procedures are typically performed for each item individually or for small numbers of related items. Because the analytic procedures focus on individual items, it has been difficult to pool information across items to identify potential sources of DIF analytically. In this article, we outline an approach to DIF analysis using hierarchical logistic regression that makes it possible to combine results of logistic regression analyses across items to identify consistent sources of DIF, to quantify the proportion of explained variation in DIF coefficients, and to compare the predictive accuracy of alternate explanations for DIF. The approach can also be used to improve the accuracy of DIF estimates for individual items by applying empirical Bayes techniques, with DIF-related item characteristics serving as collateral information. To illustrate the hierarchical logistic regression procedure, we use a large data set derived from recent computer-based administrations of Step 2, the clinical science component of the United States Medical Licensing Examination (USMLE®). Results of a small Monte Carlo study of the accuracy of the DIF estimates are also reported.

Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis

Increasingly, tests are being translated and adapted into different languages. Differential item functioning (DIF) analyses are often used to identify non‐equivalent items across language groups. However, few studies have focused on understanding why some translated items produce DIF. The purpose of the current study is to identify sources of differential item and bundle functioning on translated achievement tests using substantive and statistical analyses. A substantive analysis of existing DIF items was conducted by an 11‐member committee of testing specialists. In their review, four sources of translation DIF were identified. Two certified translators used these four sources to categorize a new set of DIF items from Grade 6 and 9 Mathematics and Social Studies Achievement Tests. Each item was associated with a specific source of translation DIF and each item was anticipated to favor a specific group of examinees. Then, a statistical analysis was conducted on the items in each category using SIBTEST. The translators sorted the mathematics DIF items into three sources, and they correctly predicted the group that would be favored for seven of the eight items or bundles of items across two grade levels. The translators sorted the social studies DIF items into four sources, and they correctly predicted the group that would be favored for eight of the 13 items or bundles of items across two grade levels. The majority of items in mathematics and social studies were associated with differences in the words, expressions, or sentence structure of items that are not inherent to the language and/or culture. By combining substantive and statistical DIF analyses, researchers can study the sources of DIF and create a body of confirmed DIF hypotheses that may be used to develop guidelines and test construction principles for reducing DIF on translated tests.

Identifying the Causes of DIF in Translated Verbal Items

Translated tests are being used increasingly for assessing the knowledge and skills of individuals who speak different languages. There is little research exploring why translated items sometimes function differently across languages. If the sources of differential item functioning (DIF) across languages could be predicted, it could have important implications on test development, scoring and equating. This study focuses on two questions: “Is DIF related to item type?”, “What are the causes of DIF?” The data were taken from the Israeli Psychometric Entrance Test in Hebrew (source) and Russian (translated). The results indicated that 34% of the items functioned differentially across languages. The analogy items were the most problematic with 65% showing DIF, mostly in favor of the Russian‐speaking examinees. The sentence completion items were also a problem (45% D1F). The main reasons for DIF were changes in word difficulty, changes in item format, differences in cultural relevance, and changes in content.

REVISING SAT®‐VERBAL ITEMS TO ELIMINATE DIFFERENTIAL ITEM FUNCTIONING

ABSTRACTBased on initial SAT‐Verbal pretest data and/or hypotheses advanced in the research literature, the authors selected 7 sentence completions and 16 analogies with extreme levels of differential item functioning (DIF) and then systematically revised and readministered the items in an attempt to reduce or eliminate DIF. Several diverse conclusions can be drawn from the data. First, because of the apparent success in reducing extreme levels of DIF in SAT‐Verbal items, the authors recommend that such efforts be continued. Second, the particular terminology used in stems and keys (rather than the underlying reasoning skill being measured) seems to be a recurring source of DIF in SAT‐Verbal items. Third, larger sample sizes, particularly for minority focal groups, would help to stabilize the DIF categories used by Educational Testing Service (ETS) test developers. Fourth, because the ETS delta metric is unbounded at the extremes, the use of both the Standardization (p‐metric) and Mantel‐Haenszel (delta‐metric) methodologies is recommended for classifying the level of DIF for very easy and very difficult items. Finally, the paper concludes with a suggestion for further research concerning the possible relationship between DIF and predictive validity.

EVALUATING HYPOTHESES ABOUT DIFFERENTIAL ITEM FUNCTIONING1,2

Studies evaluating hypotheses about sources of differential item functioning (DIF) are classified into two categories: observational studies evaluating operational items and randomized DIF studies evaluating specially constructed items. For observational studies, advice is given for item classification, sample selection, the matching criterion, and the choice of DIF techniques, as well as how to summarize, synthesize and translate DIF data into DIF hypotheses. In randomized DIF studies of specially constructed items, specific hypotheses, often generated from observational studies, are evaluated under rigorous conditions. Advice for these studies focuses on the importance of carefully constructed items to assess DIF hypotheses. In addition, randomized DDF studies are cast within a causal inference framework, which provides a justification for the use of standardization analyses or logistic regression analysis to estimate effect sizes. Two studies that have components spanning the observational and controlled domains are summarized for illustrative purposes. Standardization analyses are used for both studies. Special logistic regression analyses of an item from one of these studies are provided to illustrate a new approach in the assessment of DIF hypotheses using specially constructed items.