Item Response Theory Likelihood Ratio Research Articles

Investigating Differential Item Functioning of an International Mathematics Competition Items across Gender Groups

Mathematical problem-solving competitions have existed for over a century. Scholars report the gender gap in these competitions. As a result, it is necessary to determine whether any score difference between gender groups is attributable to a genuine difference or is the result of the exam itself. Thus, the current study specifically examined bias in one of the well-known mathematics competitions: the Kangaroo Mathematics competition. Determining the fairness of Kangaroo mathematics competition items across gender groups is crucial for creating accurate comparisons and avoiding unintended construct irrelevant bias. To examine the bias, Differential Item Functioning (DIF) analyses were conducted using Logistic Regression, Mantel-Haenszel, and Item Response Theory Likelihood Ratio Test DIF detection methods. After a series of investigations, out of 336 items, it was concluded that these mathematics items were free of DIF and bias across the gender groups. Further implications were discussed in detail regarding the validity and bias.

Psychometric Properties of the WHO-5 Well-Being Index among Nurses during the COVID-19 Pandemic: A Cross-Sectional Study in Three Countries.

Nurses’ well-being has been increasingly recognised due to the ongoing pandemic. However, no validation scales measuring nurses’ well-being currently exist. Thus, we aimed to validate the WHO-5 Well-Being Index (WHO-5) in a sample of nurses. A cross-sectional multinational study was conducted, and a total of 678 nurses who worked during the COVID-19 pandemic in Spain (36.9%), Chile (40.0%) and Norway (23.1%) participated in this study. The nurses completed the WHO-5, the Patient Health Questionnaire-2 (PHQ-2), the Generalized Anxiety Disorder-2 (GAD-2) and three questions about the quality of life (QoL). The WHO-5 demonstrated good reliability and validity in the three countries. Cronbach’s alphas ranged from 0.81 to 0.90. High correlations were found between the WHO-5 and the psychological well-being dimension of QoL, and negative correlations between the WHO-5 and PHQ-2. The unidimensional scale structure was confirmed in all the countries, explaining more than 68% of the variance. The item response theory likelihood ratio model did not show discernible differences in the WHO-5 across the countries. To conclude, the WHO-5 is a psychometrically sound scale for measuring nurses’ well-being during a pandemic. The scale showed strong construct validity for cross-cultural comparisons; however, more research is required with larger sample sizes.

A comparison of methods to address item non-response when testing for differential item functioning in multidimensional patient-reported outcome measures.

Item non-response (i.e., missing data) may mask the detection of differential item functioning (DIF) in patient-reported outcome measures or result in biased DIF estimates. Non-response can be challenging to address in ordinal data. We investigated an unsupervised machine-learning method for ordinal item-level imputation and compared it with commonly-used item non-response methods when testing for DIF. Computer simulation and real-world data were used to assess several item non-response methods using the item response theory likelihood ratio test for DIF. The methods included: (a) list-wise deletion (LD), (b) half-mean imputation (HMI), (c) full information maximum likelihood (FIML), and (d) non-negative matrix factorization (NNMF), which adopts a machine-learning approach to impute missing values. Control of Type I error rates were evaluated using a liberal robustness criterion for α = 0.05 (i.e., 0.025-0.075). Statistical power was assessed with and without adoption of an item non-response method; differences > 10% were considered substantial. Type I error rates for detecting DIF using LD, FIML and NNMF methods were controlled within the bounds of the robustness criterion for > 95% of simulation conditions, although the NNMF occasionally resulted in inflated rates. The HMI method always resulted in inflated error rates with 50% missing data. Differences in power to detect moderate DIF effects for LD, FIML and NNMF methods were substantial with 50% missing data and otherwise insubstantial. The NNMF method demonstrated comparable performance to commonly-used non-response methods. This computationally-efficient method represents a promising approach to address item-level non-response when testing for DIF.

Detecting Differential Item Functioning Using SIBTEST, MH, LR and IRT Methods

In this study, differential item functioning (DIF) and differential bundle functioning (DBF) analyses of the Academic Staff and Postgraduate Education Entrance Examination Quantitative Ability Tests were carried out. Mantel-Haenszel, logistic regression, SIBTEST, Item Response Theory-Likelihood Ratio and BILOG-MG DIF Algorithm methods were used for DIF analyses. SIBTEST was the method used for DBF analyses. Data sets for the study came from an earlier application of the examination. Gender DIF analyses showed that eleven items showed DIF. Four of the items favored male applicants, where seven of them favored female applicants. In order to investigate the sources of DIF, we consulted experts. In general, the items which could be solved using routine algorithmic operations and which are presented in the algebraic, abstract format showed DIF in favor of females. The “real-life” word problems favored males. According to DBF analyses, the operations item group favored females and the word problems item group favored males.

Performances Based on Ability Estimation of the Methods of Detecting Differential Item Functioning: A Simulation Study

The aim of the study is to examine differential item functioning (DIF) detection methods—the simultaneous item bias test (SIBTEST), Item Response Theory likelihood ratio (IRT-LR), Lord chi square (χ2), and Raju area measures—based on ability estimates when purifying items with DIF from the test, considering conditions of ratio of the items with DIF, effect size of DIF, and type of DIF. This study is a simulation study and 50 replications were conducted for each condition. In order to compare DIF detection methods, error (RMSD) and coefficient of concordance (Pearson’s correlation coefficient) were calculated according to estimated and initial abilities for the reference group. As a result of the study, the lowest error and the highest concordance were seen in the case of 10% uniform DIF in the test and the method of IRT-LR, considering all other conditions. Moreover, for the method of SIBTEST and IRT-LR in all conditions, it was found that the error obtained by purifying items with C level DIF is lower than the error obtained by purifying items with both B and C level DIF. Similarly, for the method of SIBTEST and IRT-LR in all conditions, it was seen that the concordance coefficient found by purifying C level DIF is higher than the coefficient by purifying items with both B and C level DIF.

Use of Information Criteria in the Study of Group Differences in Trace Lines.

A brief review of various information criteria is presented for the detection of differential item functioning (DIF) under item response theory (IRT). An illustration of using information criteria for model selection as well as results with simulated data are presented and contrasted with the IRT likelihood ratio (LR) DIF detection method. Use of information criteria for general IRT model selection is discussed.

Detecting Multidimensional Differential Item Functioning with the Multiple Indicators Multiple Causes Model, the Item Response Theory Likelihood Ratio Test, and Logistic Regression

Differential item functioning (DIF) is typically evaluated in educational assessments with a simple structure in which items are associated with a single latent trait. This study aims to extend the investigation of DIF for multidimensional assessments with a nonsimple structure in which items can be associated with two or more latent traits. A simulation study was conducted with the multidimensional extensions of the item response theory likelihood ratio (IRT-LR) test, the multiple indicators multiple causes (MIMIC) model, and logistic regression for detecting uniform and nonuniform DIF in multidimensional assessments. The results indicated that the IRT-LR test outperformed the MIMIC and logistic regression approaches in detecting nonuniform DIF. When detecting uniform DIF, the MIMIC and logistic regression approaches appeared to perform better than the IRT-LR test in short tests, while the performances of all three approaches were very similar in longer tests. Type I error rates for logistic regression were severely inflated compared to the other two approaches. The IRT-LR test appears to be a more balanced and powerful method than the MIMIC and logistic regression approaches in detecting DIF in multidimensional assessments with a nonsimple structure.

METHODS AND SOFTWARE FOR RESEARCH OF THE VALIDITY OF TEST RESULTS FOR THE TESTED GROUPS WITH CERTAIN INDIVIDUAL CHARACTERISTICS

The necessary condition for the presence of biased assessment by some test is differential item functioning in different groups of test takers. The ideas of some statistical methods for detecting Differential Item Functioning are described in the given article. They were developed in the framework of the main approaches to modeling test results: using contingency tables, regression models, multidimensional models and models of Item Response Theory. The Mantel-Haenszel procedure, logistic regression method, SIBTEST and Item Response Theory Likelihood Ratio Test are considered. The characteristics of each method and conditions of their application are specified. Overview of existing free software tools implementing these methods is carried out. Comparisons of these methods are conducted on the example of real data. Also notes that it is appropriate to use several methods simultaneously to reduce the risk of false conclusions.

Comparing Performances (Type I error and Power) of IRT Likelihood Ratio SIBTEST and Mantel-Haenszel Methods in the Determination of Differential Item Functioning

This simulation study compared the performances (Type I error and power) of Mantel-Haenszel (MH), SIBTEST, and item response theory-likelihood ratio (IRT-LR) methods under certain conditions. Manipulated factors were sample size, ability differences between groups, test length, the percentage of differential item functioning (DIF), and underlying model used to generate data. Results suggest that SIBTEST had the highest Type I error in the detection of uniform DIF, but MH had the highest power under all conditions. In addition, the percentage of DIF and the underlying model appear to have influenced the Type I error rate of IRT-LR. Ability differences between groups, test length, the percentage of DIF, model, and the interactions between ability differences*percentage of DIF, ability differences*test length, test length*percentage of DIF, test length*model affected the SIBTEST methods' Type I error rate. In the MH procedure, effective factors for Type I error rate were: sample size, test length, the percentage of DIF, ability differences*percentage of DIF, ability differences*model, and ability differences*percentage of DIF*model. No factors were effective on the power of SIBTEST and MH, but the underlying model had a significant effect on the IRT-LR power rate.

Causes of gender DIF on an EFL language test: A multiple-data analysis over nine years

This study tracked gender differential item functioning (DIF) on the English subtest of the Korean College Scholastic Aptitude Test (KCSAT) over a nine-year period across three data points, using both the Mantel-Haenszel (MH) and item response theory likelihood ratio (IRT-LR) procedures. Further, the study identified two factors (i.e. reading strategy and perceived interest) that explained a portion of the variance in the magnitude of gender DIF via a series of multiple linear regression analyses. The results indicated (1) an interaction between item type and gender DIF and (2) a significant relationship between gender differences in the examinee’s perceived interest in test items and the magnitude of gender DIF. The study discusses the results based on previous DIF research and presents pedagogical implications and some avenues for further research.

Determination of a Differential Item Functioning Procedure Using the Hierarchical Generalized Linear Model

The aim of this research is to compare the result of the differential item functioning (DIF) determining with hierarchical generalized linear model (HGLM) technique and the results of the DIF determining with logistic regression (LR) and item response theory–likelihood ratio (IRT-LR) techniques on the test items. For this reason, first in this research, it is determined whether the students encounter DIF with HGLM, LR, and IRT-LR techniques according to socioeconomic status (SES), in the Turkish, Social Sciences, and Science subtest items of the Secondary School Institutions Examination. When inspecting the correlations among the techniques in terms of determining the items having DIF, it was discovered that there was significant correlation between the results of IRT-LR and LR techniques in all subtests; merely in Science subtest, the results of the correlation between HGLM and IRT-LR techniques were found significant. DIF applications can be made on test items with other DIF analysis techniques that were not taken to the scope of this research. The analysis results, which were determined by using the DIF techniques in different sample sizes, can be compared.

The Impact of Missing Data on the Detection of Nonuniform Differential Item Functioning

Missing information is a ubiquitous aspect of data analysis, including responses to items on cognitive and affective instruments. Although the broader statistical literature describes missing data methods, relatively little work has focused on this issue in the context of differential item functioning (DIF) detection. Such prior research has focused on uniform DIF and has identified a complicated relationship between the type and amount of missing data, along with the methods for dealing with it. The current simulation study was designed to extend this earlier work by focusing on the impact of missing data and methods for dealing with it on nonuniform DIF detection using popular tools, including logistic regression, crossing SIBTEST, and the item response theory likelihood ratio test. Results suggest that the relationships between the various factors manipulated here were complex with no one method emerging as optimal in all cases, although listwise deletion consistently produced results similar to those obtained with the complete data across simulated conditions. Implications of these results are also discussed.

Item Parameter Invariance of the Kaufman Adolescent and Adult Intelligence Test Across Male and Female Samples

The Kaufman Adolescent and Adult Intelligence Test (KAIT™) is an individually administered test of intelligence for individuals ranging in age from 11 to 85+ years. The item response theory—likelihood ratio procedure, based on the two-parameter logistic model, was used to detect differential item functioning (DIF) in the KAIT across males and females in the standardization sample. Root mean squared differences and item parameter differences were used to indicate the magnitude of DIF and identify which group the item parameter favored. A z test of proportion differences was conducted to determine if the number of parameters exhibiting gender DIF exceeded the number expected by chance, estimated by randomly dividing the sample in half and repeating the analyses. Of the 176 item parameters examined, 42 (24%) lacked invariance, with most items reporting uniform DIF. Implications for test score interpretation and future research are discussed.

Judgmental and Statistical DIF Analyses of the PISA-2003 Mathematics Literacy Items

Comparisons of human characteristics across different language groups and cultures become more important in today's educational assessment practices as evidenced by the increasing interest in international comparative studies. Within this context, the fairness of the results across different language and cultural groups draws the attention of researchers. In the present study, the utility of hypothesis generated about differential item functioning (DIF) will be studied using the Program for International Student Assessment (PISA) 2003 data set. Items that were flagged as DIF through Restricted Factor Analysis (RFA), Mantel-Haenzsel (MH), and Item Response Theory Likelihood Ratio Analysis (IRT-LR) were simultaneously considered for content-wise evaluation. A group of judges reported that (1) cognitive skills measured by the items, (2) translation errors, and (3) use of quantitative words are three important sources of error that might be causing DIF in the PISA 2003 study. Results indicated that systematic a-priory analyses of the item content may help anticipate items with DIF in a cross-cultural comparison. Close agreement was obtained between the judges' opinions about DIF and statistical DIF analysis results.

Application of item response theory to achieve cross‐cultural comparability of occupational stress measurement

Our objective was to examine cross-cultural comparability of standard scales of the Effort-Reward Imbalance occupational stress scales by item response theory (IRT) analyses. Data were from 20,256 Japanese employees, 1464 Dutch nurses and nurses' aides, 2128 representative employees from post-communist countries, 963 Swedish representative employees, 421 Chinese female employees, 10,175 employees of the French national gas and electric company and 734 Spanish railroad employees, sanitary personnel and telephone operators. The IRT likelihood ratio model was used for differential item functioning (DIF) and differential test functioning (DTF) analyses. Despite the existence of DIF, most comparisons did not show discernible differences in the relations between Effort-Reward total score and level of the underlying trait across cultural groups. In the case that DTF was suspected, excluding an item with significant DIF improved the comparability. The full cross-cultural comparability of Effort-Reward Imbalance scores can be achieved with the help of IRT analysis.

The Effects of Referent Item Parameters on Differential Item Functioning Detection Using the Free Baseline Likelihood Ratio Test

The purpose of this simulation study is to investigate the effects of anchor subtest composition on the accuracy of item response theory (IRT) likelihood ratio (LR) differential item functioning (DIF) detection (Thissen, Steinberg, & Wainer, 1988). Here, the IRT LR test was implemented with a free baseline approach wherein a baseline model was formed by freeing all items except a referent or anchor subset and examining the changes in fit with respect to a series of models wherein 1 item at a time was constrained in addition to the referent(s). The results clearly indicated that the composition of the anchor subtest is important for accurate DIF detection. It was found that using a single highly discriminating rather than a low discriminating referent greatly enhanced the power of the procedure. Moreover, in conditions involving small DIF or smaller sample sizes or both, power appeared to improve when a group of highly discriminating referents was used. These findings have implications for applied research involving short scales and small sample sizes.

IRT-LR-DIF With Estimation of the Focal-Group Density as an Empirical Histogram

Item response theory–likelihood ratio–differential item functioning (IRT-LR-DIF) is used to evaluate the degree to which items on a test or questionnaire have different measurement properties for one group of people versus another, irrespective of group-mean differences on the construct. Usually, the latent distribution is presumed normal for both groups, but previous research shows that results are biased if the true distribution is not approximately normal. This article introduces a variation of IRT-LR-DIF, called empirical histogram–differential item functioning (EH-DIF), in which the focal-group latent density is estimated simultaneously with the item parameters as an empirical histogram (EH). A simulation study shows that if the focal-group density is nonnormal, Type I error rates and focal-group estimates of the item parameters, mean, and SD are more accurate using EH-DIF than standard IRT-LR-DIF methods that presume normality. A pseudoempirical example is analyzed to illustrate EH-DIF.

Detection of Crossing Differential Item Functioning

Differential item functioning (DIF) continues to receive attention both in applied and methodological studies. Because DIF can be an indicator of irrelevant variance that can influence test scores, continuing to evaluate and improve the accuracy of detection methods is an essential step in gathering score validity evidence. Methods for detecting uniform DIF are well established, whereas those for identifying the presence of nonuniform or crossing DIF are less clearly understood. Four such methods were compared: simultaneous item bias test (SIBTEST), logistic regression, item response theory likelihood ratio test, and confirmatory factor analysis. Factors manipulated were sample size, ability differences between groups, percentage of DIF, and the underlying model used to generate the data. Results suggest that all methods were able to control Type I error, but SIBTEST had the highest power of the approaches compared. Problems with detection rates were evident with different underlying models.

Examining the relationship between differential item functioning and differential test functioning

The present study utilized both the IRT-LR (item response theory likelihood ratio) and a series of CFA (confirmatory factor analysis) multi-sample analyses to systematically examine the relationships between DIF (differential item functioning) and DTF (differential test functioning) with a random sample of 15 000 Korean examinees. Specifically, DIF was detected using the IRT-LR method and the cumulative effect of DIF on DTF was gauged by the multi-sample analysis technique offered by the LISREL 8.5 program. The results of the current study indicate that item level DIF, once detected, may be carried to the test level bias regardless of the DIF directions, thereby showing mixed evidence to the previous findings reported in the literature. This suggests that the relation of DIF to DTF seems to be much more complex than that reported in the literature and, accordingly, more empirical studies are needed to bridge the gap in the literature about DIF-DTF relationships.

DIF for examinees with different academic backgrounds

The purpose of the present study was to investigate differential item function (DIF) on the English subtest of the 1998 Korean National Entrance Exam for Colleges and Universities for examinees with different academic backgrounds (i.e., Humanities vs. Sciences) using the Item Response Theory (IRT) Likelihood Ratio (LR) approach. The English subtest consisted of a total of 55 items, which comprised the subscales of Listening Comprehension and Reading Comprehension. Data were fitted with a modified three parameter logistic IRT model, and DIF was detected using both the Mantel-Haenszel procedure and the IRT likelihood ratio approach. To evaluate a baseline estimate of DIF due to chance, the entire DIF detection procedure was repeated, with the sample randomly divided in half. The results of the study identified 18 DIF items with 28 DIF parameters. DIF magnitude for each studied item is also provided. Results of a preliminary content analysis are discussed in terms of the relationships between DIF directions and item content. Implications and recommendations for future research are also addressed.