Mantel-Haenszel Differential Item Functioning Research Articles

Robustness of Weighted Differential Item Functioning (DIF) Analysis: The Case of Mantel–Haenszel DIF Statistics

Two families of analysis methods can be used for differential item functioning (DIF) analysis. One family is DIF analysis based on observed scores, such as the Mantel–Haenszel (MH) and the standardized proportion‐correct metric for DIF procedures; the other is analysis based on latent ability, in which the statistic is a measure of departure from measurement invariance (DMI) for two studied groups. Previous research has shown, that DIF and DMI do not necessarily agree with each other. In practice, many operational testing programs use the MH DIF procedure to flag potential DIF items. Recently, weighted DIF statistics has been proposed, where weighted sum scores are used as the matching variable and the weights are the item discrimination parameters. It has been shown theoretically and analytically that, given the item parameters, weighted DIF statistics can close the gap between DIF and DMI. The current study investigates the robustness of using weighted DIF statistics empirically through simulations when item parameters have to be estimated from data.

Statistical Equivalence Testing Approaches for Mantel–Haenszel DIF Analysis

The null hypothesis test used in differential item functioning (DIF) detection tests for a subgroup difference in item-level performance—if the null hypothesis of “no DIF” is rejected, the item is flagged for DIF. Conversely, an item is kept in the test form if there is insufficient evidence of DIF. We present frequentist and empirical Bayes approaches for implementing statistical equivalence testing for DIF using the Mantel–Haenszel (MH) DIF statistic. With these approaches, rejection of the null hypothesis of “DIF” allows the conclusion of statistical equivalence, a more stringent criterion for keeping items. In other words, the roles of the null and alternative hypotheses are interchanged in order to have positive evidence that the DIF of an item is small. A simulation study compares the equivalence testing approaches to the traditional MH DIF detection method with the Educational Testing Service classification system. We illustrate the methods with item response data from the 2012 Programme for International Student Assessment.

Type I Error Inflation in DIF Identification With Mantel–Haenszel

It is known that sum score-based methods for the identification of differential item functioning (DIF), such as the Mantel–Haenszel (MH) approach, can be affected by Type I error inflation in the absence of any DIF effect. This may happen when the items differ in discrimination and when there is item impact. On the other hand, outlier DIF methods have been developed that are robust against this Type I error inflation, although they are still based on the MH DIF statistic. The present article gives an explanation for why the common MH method is indeed vulnerable to the inflation effect whereas the outlier DIF versions are not. In a simulation study, we were able to produce the Type I error inflation by inducing item impact and item differences in discrimination. At the same time and in parallel with the Type I error inflation, the dispersion of the DIF statistic across items was increased. As expected, the outlier DIF methods did not seem sensitive to impact and differences in item discrimination.

Investigating item and construct bias in an English verbal analogies scale

For this study the researcher was interested in investigating the scalar equivalence of the adapted English version of the verbal analogies (VA) subscale of the Woodcock Muñoz Language Survey (WMLS) across English first-language speakers and Xhosa first-language speakers. This was achieved by utilising differential item functioning (DIF) and construct bias statistical techniques. The Mantel-Haenszel DIF detection method was employed to detect DIF, while construct equivalence was examined by means of exploratory factor analysis (EFA) utilising an a priori two-factor structure. The Tucker's phi coefficient was used to assess the congruence of the construct across the two language groups. The sample consisted of 192 English and 193 Xhosa first-language learners, who were selected from ex-Model C and previously disadvantaged schools in the Port Elizabeth and Grahamstown region. The main findings of this study indicated that the adapted English version of the VA scale displayed DIF items across the two language groups. Moreover, construct equivalence could only be established for one factor across the two language groups, as the second factor displayed non-negligible incongruities even after the removal of DIF items.

Accuracy of DIF Estimates and Power in Unbalanced Designs Using the Mantel–Haenszel DIF Detection Procedure

This study examined how much improvement was attainable with respect to accuracy of differential item functioning (DIF) measures and DIF detection rates in the Mantel–Haenszel procedure when employing focal and reference groups with notably unbalanced sample sizes where the focal group has a fixed small sample which does not satisfy the minimum DIF sample size requirement specified by the testing programs, while the reference group sample size far exceeds the minimum requirement. Results showed equivalent or better results with such unbalanced but large samples than with some of the currently used minimum DIF sample size conditions. DIF investigation, therefore, does not necessarily need to cease when the focal group does not meet the minimum sample size requirement. Some analytic explanations and guidelines for DIF investigations with unbalanced sample sizes are also provided.

The development of an ICF-oriented, adaptive physician assessment instrument of mobility, self-care, and domestic life

The objective of this study was development and psychometric testing of an adaptive, International Classification of Functioning, Disability, and Health (ICF)-oriented questionnaire to be processed by the rehabilitation physician that aids in assessing mobility, self-care, and domestic life (Moses-Physician). The intent is to develop a physician version, analogous in content to the existing patient questionnaire 'Moses-Patient'. The 58 items of Moses-Patient were converted to an external assessment format without altering the content. The data were compiled for 549 patients with musculoskeletal diseases, 212 patients with cardiac diseases, and 259 neurology rehabilitation patients. Analyses were carried out on the basis of the one-parameter item response theory (Rasch model). Effect sizes and the reliable change index were calculated to test responsiveness. Differential item functioning (DIF) was tested using DIF contrasts, equivalent to Mantel-Haenszel DIF sizes. After the item response theory analysis, 47 of 58 items remained, distributed over 12 scales. The scales are more homogeneous in content than in the patient version because of the omission of 11 items and thus do not cover the ICF categories as broadly. Model fit indices (infit and outfit mean square statistics) were in an acceptable range for all items. Cronbach's alpha was between 0.73 and 0.95. Moreover, there is clear evidence of unidimensionality and sensitivity to change of the scales of Moses-Physician. The item parameters of Moses-Physician are invariant with respect to sex and age for all scales. However, there are clear differences regarding disease groups. The Moses-Physician questionnaire is an adaptive, Rasch-scaled assessment instrument that, to a great extent, covers the contents of the ICF chapters 'mobility', 'self-care', and 'domestic life'.

DIF DETECTION WITH SMALL SAMPLES: APPLYING SMOOTHING TECHNIQUES TO FREQUENCY DISTRIBUTIONS IN THE MANTEL‐HAENSZEL PROCEDURE

ABSTRACTAll differential item functioning (DIF) methods require at least a moderate sample size for effective DIF detection. Samples that are less than 200 pose a challenge for DIF analysis. Smoothing can improve upon the estimation of the population distribution by preserving major features of an observed frequency distribution while eliminating the noise brought about by irregular data points. This study applied smoothing techniques to frequency distributions and investigated the impact of smoothed data on the Mantel‐Haenszel (MH) DIF detection in small samples. Eight sample‐size combinations were randomly drawn from a real data set to make the study realistic and were replicated 80 times to produce stable results. The population DIF results were used as the criteria to evaluate sample estimates using root‐mean square difference (RMSD), bias analysis, and Type II error rate. Loglinear smoothing was found to provide slight to moderate improvements in MH DIF estimation with small samples.

A Generalized Formula for the Mantel-Haenszel Differential Item Functioning Parameter

The present study derives a general formula for the population parameter being estimated by the Mantel-Haenszel (MH) differential item functioning (DIF) statistic. Because the formula is general, it is appropriate for either uniform DIF (defined as a difference in item response theory item difficulty values) or nonuniform DIF; and it can be used regardless of the form of the item response function. In the case of uniform DIF modeled with two-parameter-logistic response functions, the parameter is well known to be linearly related to the difference in item difficulty between the focal and reference groups. Even though this relationship is known to not strictly hold true in the case of three-parameter-logistic (3PL) uniform DIE the degree of the departure from this relationship has not been known and has been generally believed to be small By evaluating the MH DIF parameter, we show that for items of medium or high difficulty, the parameter is much smaller in absolute value than expected based on the difference in item difficulty between the two groups. These results shed new light on results from previous simulation studies that showed the MH DIF statistic has a tendency to shrink toward zero with increasing difficulty level when used with 3PL data.

SEX DIFFERENCES IN THE PERFORMANCE OF HIGH‐SCORING EXAMINEES ON THE SAT®‐M

ABSTRACTPerformance of high‐scoring males and females on the mathematics section of three forms of the College Board's Scholastic Aptitude Test (SAT‐M) was examined to determine how item content, solution strategy, and speededness differentially affect performance. The mathematical and verbal sections of the SAT were also compared for similarities in the performance patterns of high scorers.Items on three forms of the SAT‐M were classified on the basis of content and potential solution strategies. Male and female performance in the top 5 percent was examined using the Mantel‐Haenszel Differential Item Functioning (MH‐DIF) procedure. A main effect was found for item‐solution strategy, but not for item content. Males tended to outperform females on items requiring logical estimation or insight even when subjects were matched on years of mathematics course taking and average mathematics grades.Conventional measures indicated that the test was not differentially speeded. However, omit rates indicated that females omitted a greater proportion of items requiring estimation. Different patterns by gender were found on the mathematical and verbal sections of the test.