Differential Item Functioning Detection Research Articles

Evaluating Type I Error and Power Rates Using an Effect Size Measure With the Logistic Regression Procedure for DIF Detection

The logistic regression (LR) procedure for differential item functioning (DIF) detection is a model-based approach designed to identify both uniform and nonuniform DIF. However, this procedure tends to produce inflated Type I errors. This outcome is problematic because it can result in the inefficient use of testing resources, and it may interfere with the study of the underlying causes of DIF. Recently, an effect size measure was developed for the LR DIF procedure and a classification method was proposed. However, the effect size measure and classification method have not been systematically investigated. In this study, we developed a new classification method based on those established for the Simultaneous Item Bias Test. A simulation study also was conducted to determine if the effect size measure affects the Type I error and power rates for the LR DIF procedure across sample sizes, ability distributions, and percentage of DIF items included on a test. The results indicate that the inclusion of the effect size measure can substantially reduce Type I error rates when large sample sizes are used, although there is also a reduction in power.

Item Invariance in Four Subtests of the Universal Nonverbal Intelligence Test (UNIT) across Groups of Deaf and Hearing Children

The Universal Nonverbal Intelligence Test (UNIT) is a new, individually administered, nonverbal intelligence test that is administered to children ages 5 through 17, regardless of hearing status, via gesture. Methods to detect differential item functioning (DIF), including the Mantel-Haenszel and Item Response Theory (IRT) likelihood ratio procedures, were used to determine whether UNIT items in four subtests (Symbolic Memory, Spatial Memory, Analogic Reasoning, and Object Memory) function differently for profoundly deaf and hearing children from the national standardization sample. No items were found to exhibit DIF, indicating that the probability of a correct response does not seem to be affected by hearing status. Further studies of the psychometric properties of the UNIT when administered to deaf children are encouraged, as well as investigations of DIF in other tests that are widely used for assessing deaf children.

Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures.

Cognitive screening tests and items have been found to perform differently across groups that differ in terms of education, ethnicity and race. Despite the profound implications that such bias holds for studies in the epidemiology of dementia, little research has been conducted in this area. Using the methods of modern psychometric theory (in addition to those of classical test theory), we examined the performance of the Attention subscale of the Mattis Dementia Rating Scale. Several item response theory models, including the two- and three-parameter dichotomous response logistic model, as well as a polytomous response model were compared. (Log-likelihood ratio tests showed that the three-parameter model was not an improvement over the two-parameter model.) Data were collected as part of the ten-study National Institute on Aging Collaborative investigation of special dementia care in institutional settings. The subscale KR-20 estimate for this sample was 0.92. IRT model-based reliability estimates, provided at several points along the latent attribute, ranged from 0.65 to 0.97; the measure was least precise at the less disabled tail of the distribution. Most items performed in similar fashion across education groups; the item characteristic curves were almost identical, indicating little or no differential item functioning (DIF). However, four items were problematic. One item (digit span backwards) demonstrated a large error term in the confirmatory factor analysis; item-fit chi-square statistics developed using BIMAIN confirm this result for the IRT models. Further, the discrimination parameter for that item was low for all education subgroups. Generally, persons with the highest education had a greater probability of passing the item for most levels of theta. Model-based tests of DIF using MULTILOG identified three other items with significant, albeit small, DIF. One item, for example, showed non-uniform DIF in that at the impaired tail of the latent distribution, persons with higher education had a higher probability of correctly responding to the item than did lower education groups, but at less impaired levels, they had a lower probability of a correct response than did lower education groups. Another method of detection identified this item as having DIF (unsigned area statistic=3.05, p<0.01, and 2.96, p<0.01). On average, across the entire score range, the lower education group's probability of answering the item correctly was 0.11 higher than the higher education group's probability. A cross-validation with larger subgroups confirmed the overall result of little DIF for this measure. The methods used for detecting differential item functioning (which may, in turn, be indicative of bias) were applied to a neuropsychological subtest. These methods have been used previously to examine bias in screening measures across education and ethnic and racial subgroups. In addition to the important epidemiological applications of ensuring that screening measures and neuropsychological tests used in diagnoses are free of bias so that more culture-fair classifications will result, these methods are also useful for the examination of site differences in large multi-site clinical trials. It is recommended that these methods receive wider attention in the medical statistical literature.

Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures

Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures

Detection of Differential Item Functioning on the Kirton Adaption-Innovation Inventory Using Multiple-Group Mean and Covariance Structure Analyses

This article demonstrates how Sörbom's (1974) mean and covariance structure analysis (MACS) model can be used to detect uniform and non-uniform differential item functioning (DIF) on polytomous ordered response items that are assumed to approximate a continuous scale. Using item responses from 773 civil service employees, the MACS model was applied to three cognitive style scales on the widely-used Kirton Adaption-Innovation Inventory (KAI, Kirton, 1976; 1977). As predicted, DIF across managerial and staff support occupational groups were detected on each of the scales. As expected, there was no evidence of DIF across gender groups and across two groups formed through a random split on any of the scales. Substantive implications for the use of the KAI as well as methodological implications and future research direction for detection of DIF are discussed.

A Comparison of χ2, RFA and IRT Based Procedures in the Detection of DIF

Bias research began at the end of the 1960s anddeveloped rapidly in the following decades for obvioussocial and political reasons, and due to the importantimpact that this issue has on the field ofpsychological and educational measurement. Since then,several methods have been proposed for the study anddetection of item bias or differential itemfunctioning (DIF). This paper presents a simulationstudy comparing the potential of some of these methodsfor detecting DIF: two IRT-based techniques (area measures), three χ2-based procedures (MantelHaenszel, Logit Model and Logistic Regression) and theRestricted Factor Analysis method. The results showthat the technique that appears to do the best job isthe Mantel Haenszel statistic. Moreover, all detectiontechniques tend to overidentify DIF items, that is,some of the items labeled with DIF may in fact bewithout DIF. This tendency is slightly reversed in theLogistic Regression procedure.

A Comparison of Logistic Regression and Analysis of Variance Differential Item Functioning Detection Methods

Differential item functioning (DIF) detection rates were compared between logistic regression and analysis of variance for dichotomously scored items. These two DIF methods were compared using simulated binary item response data sets of varying test length (20, 40, and 60 items), sample size (200, 400, and 600 examinees), discrimination type (fixed and varying), and relative underlying ability (equal and unequal) between groups under conditions of uniform DIF, nonuniform DIF, combination DIF, and false positive errors. These test conditions were replicated 100 times. For both DIF detection methods, a test length of 20 items was sufficient for satisfactory DIF detection with detection rate increasing as sample size increased. With the exception of uniform DIF, the logistic regression method had higher mean detection rates than the analysis of variance method. Because the type of DIF present in real data is rarely known, the logistic regression method is recommended for most practical applications.

An Investigation of the Power of the Likelihood Ratio Goodness‐of‐Fit Statistic in Detecting Differential Item Functioning

The purpose of this study was to investigate the power and Type I error rate of the likelihood ratio goodness‐of‐fit (LR) statistic in detecting differential item functioning (DIF) under Samejima's (1969, 1972) graded response model. A multiple‐replication Monte Carlo study was utilized in which DIF was modeled in simulated data sets which were then calibrated with MULTILOG (Thissen, 1991) using hierarchically nested item response models. In addition, the power and Type I error rate of the Mantel (1963) approach for detecting DIF in ordered response categories were investigated using the same simulated data, for comparative purposes. The power of both the Mantel and LR procedures was affected by sample size, as expected. The LR procedure lacked the power to consistently detect DIF when it existed in reference/focal groups with sample sizes as small as 500/500. The Mantel procedure maintained control of its Type I error rate and was more powerful than the LR procedure when the comparison group ability distributions were identical and there was a constant DIF pattern. On the other hand, the Mantel procedure lost control of its Type I error rate, whereas the LR procedure did not, when the comparison groups differed in mean ability; and the LR procedure demonstrated a profound power advantage over the Mantel procedure under conditions of balanced DIF in which the comparison group ability distributions were identical. The choice and subsequent use of any procedure requires a thorough understanding of the power and Type I error rates of the procedure under varying conditions of DIF pattern, comparison group ability distributions.–or as a surrogate, observed score distributions–and item characteristics.

Stepwise Analysis of Differential Item Functioning Based on Multiple‐Group Partial Credit Model

Bock, Muraki, and Pfeiffenberger (1988) proposed a dichotomous item response theory (IRT) model for the detection of differential item functioning (DIF), and they estimated the IRT parameters and the means and standard deviations of the multiple latent trait distributions. This IRT DIF detection method is extended to the partial credit model (Masters, 1982; Muraki, 1993) and presented as one of the multiple‐group IRT models. Uniform and non‐uniform DIF items and heterogeneous latent trait distributions were used to generate polytomous responses of multiple groups. The DIF method was applied to this simulated data using a stepwise procedure. The standardized DIF measures for slope and item location parameters successfully detected the non‐uniform and uniform DIF items as well as recovered the means and standard deviations of the latent trait distributions.This stepwise DIF analysis based on the multiple‐group partial credit model was then applied to the National Assessment of Educational Progress (NAEP) writing trend data.

Detection of Differential Item Functioning Under the Graded Response Model With the Likelihood Ratio Test

Type I error rates of the likelihood ratio test for the detection of differential item functioning (DIF) were investigated using monte carlo simulations. The graded response model with five ordered categories was used to generate datasets of a 30-item test for samples of 300 and 1,000 simulated examinees. All DIF comparisons were simulated by randomly pairing two groups of examinees. Three different sample size combinations of reference and focal groups were simulated under two ability matching conditions. For each of the six combinations of sample sizes by ability matching conditions, 100 replications of DIF detection comparisons were simulated. Item parameter estimates and likelihood values were obtained by marginal maximum likelihood estimation using the computer program MULTILOG. Irpe I eryor rates of the likelihood ratio test statistics for all six combinations of sample sizes and ability matching conditions were within theoretically expected values at each of the nominal alpha levels considered.

Using Statistical Procedures to Identify Differentially Functioning Test Items

This module is intended to prepare the reader to use statistical proce­ dures to detect differentially functioning test items. 7b provide back­ ground, differential item functioning (DIF) is distinguished from item and test bias, and the importance of DIF screening within the overall test development process is discussed. The Mantel-Haenszel statistic, logistic regression, SIBTES'r, the Standardization procedure, and various IRT-based approaches are presented. For each of these proce­ dures, the theoretical framework is presented, the relative strengths and weaknesses of the method are highlighted, and guidance is pro­ vided for interpretation of the resulting statistical indices. Numerous technical decisions are required in order for the practitioner to appro­ priately implement these procedures. These decisions are discussed in some detail, as are the policy decisions necessary to implement an op­ erational DIF detection program. The module also includes an anno­ tated bibliography and a self-test.

Identification of Nonuniform Differential Item Functioning: A Comparison of Mantel-Haenszel and Item Response Theory Analysis Procedures

Although practical and one of the most popular methods, the Mantel-Haenszel (MH) procedure was not initially designated for detecting nonuniform differential item functioning (DIF). To improve detection rates for items showing nonuniform DIF, a modification of the standard MH procedure has been proposed by breaking the full sample at approximately the middle of the test score distribution and then rerunning the MH procedure on each subsample separately. The purpose of the present study was to compare this MH variation to a residual procedure based on item response theory analysis to test its capacity for the detection of nonuniform DIF. Different sets of examinee responses were generated to simulate both uniform and nonuniform DIF. Results from this study suggest that although the MH procedure may be effective for detecting nonuniform DIF, and although this capacity may be increased with a simple modification to the MH standard procedure, the residual analysis procedure achieves a higher detection rate with both uniform and nonuniform DIF.

Estimating the Importance of Differential Item Functioning

Several methods have been proposed to detect differential item functioning (DIF), an item response pattern in which members of different demographic groups have different conditional probabilities of answering a test item correctly, given the same level of ability. In this article, the mixture index of fit, proposed by Rudas, Clogg, and Lindsay (1994) , is used to estimate the fraction of the population for which DIF occurs, and this approach is compared to the Mantel-Haenszel ( Mantel &amp; Haenszel, 1959 ) test of DIF developed by Holland (1985 ; see Holland &amp; Thayer, 1988) . The proposed estimation procedure, which is noniterative, can provide information about which portions of the item response data appear to be contributing to DIF.

Identifying Cultural Differences in Items and Traits

The authors investigated the cross-cultural measurement equivalence of items in the English-language version of the NEO Personality Inventory, a measure of the five-factor personality model, in a different cultural context in which English is a language of instruction. The item responses of 432 Filipino and 610 American college students were analyzed. Three methods for detecting differential item functioning (DIF)-two based on item response theory-plus the Mantel-Haenszel method showed fairly good agreement in the detection of DIF items. Nearly 40% of the 180 items showed DIF. Several significant cultural mean differences that were found with the original raw scores were no longer significant when the authors analyzed "purified" scales obtained by deleting DIF items.

Logistic Regression and Its Use in Detecting Differential Item Functioning in Polytomous Items

A computer simulation study was conducted to determine the feasibility of using logistic regression procedures to detect differential item functioning (DIF) in polytomous items. One item in a simulated test of 25 items contained DIF; parameters' for that item were varied to create three conditions of nonuniform DIF and one of uniform DIF. Item scores were generated using a generalized partial credit model, and the data were recoded into multiple dichotomies in order to use logistic regression procedures. Results indicate that logistic regression is powerful in detecting most forms of DIF; however, it required large amounts of data manipulation, and interpretation of the results was sometimes difficult. Some logistic regression procedures may be useful in the post hoc analysis of DlF for polytomous items.

Simulation Studies of the Effects of Small Sample Size and Studied Item Parameters on SIBTEST and Mantel‐Haenszel Type I Error Performance

Two simulation studies investigated Type I error performance of two statistical procedures for detecting differential item functioning (DIF): SIBTEST and Mantel‐Haenszel (MH). Because MH and SIBTEST are based on asymptotic distributions requiring “large” numbers of examinees, the first study examined Type 1 error for small sample sizes. No significant Type I error inflation occurred for either procedure. Because MH has the potential for Type I error inflation for non‐Rasch models, the second study used a markedly non‐Rasch test and systematically varied the shape and location of the studied item. When differences in distribution across examinee group of the measured ability were present, both procedures displayed inflated Type 1 error for certain items; MH displayed the greater inflation. Also, both procedures displayed statistically biased estimation of the zero DIF for certain items, though SIBTEST displayed much less than MH. When no latent distributional differences were present, both procedures performed satisfactorily under all conditions.

WISC-III Verbal Item Invariance across Samples of Deaf and Hearing Children of Similar Measured Ability

Invariance of item difficulty estimates on the Wechsler Intelligence Scale for Children-Third Edition (WISC-III; Wechsler, 1991) Verbal scale was investigated across samples of deaf children and younger hearing children of similar ability. The samples consisted of 110 deaf children who use sign language and subsamples of younger hearing children from the WISC-III standardization sample. The samples were of similar ability as measured by total raw scores and Rasch mean logit ability estimates. Items were calibrated using a Rasch Model. Methods for detecting differential item functioning (DIF), including standardized differences and anchor item fit statistics, were used to examine invariance. Numerous items were found to function differently across groups, indicating that the item level performance of the deaf children was not characteristic of younger hearing children. Thus, Verbal subtest scores should not be used as an indication of delayed verbal ability, and the subtest scores of deaf and hearing children should not be compared.

A Comparison of Alternative Matching Strategies for DIF Detection in Tests That Are Multidimensional

Most currently accepted approaches for identifying differentially functioning test items compare performance across groups after first matching examinees on the ability of interest. The typical basis for this matching is the total test score. Previous research indicates that when the test is not approximately unidimensional, matching using the total test score may result in an inflated Type I error rate. This study compares the results of differential item functioning (DIF) analysis with matching based on the total test score, matching based on subtest scores, or multivariate matching using multiple subtest scores. Analysis of both actual and simulated data indicate that for the dimensionally complex test examined in this study, using the total test score as the matching criterion is inappropriate. The results suggest that matching on multiple subtest scores simultaneously may be superior to using either the total test score or individual relevant subtest scores.

An Investigation of the Likelihood Ratio Test For Detection of Differential Item Functioning

Type I error rates for the likelihood ratio test for de tecting differential item functioning (DIF) were investi gated using monte carlo simulations. Two- and three-parameter item response theory (IRT) models were used to generate 100 datasets of a 50-item test for samples of 250 and 1,000 simulated examinees for each IRT model. Item parameters were estimated by marginal maximum likelihood for three IRT models: the three-parameter model, the three-parameter model with a fixed guessing parameter, and the two-parameter model. All DIF comparisons were simulated by ran domly pairing two samples from each sample size and IRT model condition so that, for each sample size and IRT model condition, there were 50 pairs of reference and focal groups. Type I error rates for the two-param eter model were within theoretically expected values at each of the α levels considered. Type I error rates for the three-parameter and three-parameter model with a fixed guessing parameter, however, were different from the theoretically expected values at the α levels considered.

ESTIMATING THE IMPORTANCE OF DIFFERENTIAL ITEM FUNCTIONING

Several methods have been proposed to detect differential item functioning (DIF), an item response pattern in which members of different demographic groups have different conditional probabilities of answering a test item correctly, given the same level of ability. In this article, the mixture index of fit, proposed by Rudas, Clogg, and Lindsay (1994), is used to estimate the fraction of the population for which DIF occurs, and this approach is compared to the Mantel-Haenszel (Mantel & Haenszel, 1959) test of DIF developed by Holland (1985; see Holland & Thayer, 1988). The proposed estimation procedure, which is noniterative, can provide information about which portions of the item response data appear to be contributing to DIF.