Differential Item Functioning Items Research Articles

Effect of Differential Item Functioning on Computer Adaptive Testing Under Different Conditions

This study provides an overview of the effect of differential item functioning (DIF) on measurement precision, test information function (TIF), and test effectiveness in computer adaptive tests (CATs). Simulated data for the study was produced and analyzed with the Rstudio. During the data generation process, item pool size, DIF type, DIF percentage, item selection method for CAT, and the test termination rules were considered changed conditions. Sample size and ability parameter distribution, Item Response Theory (IRT) model, DIF size, ability estimation method, test starting rule, and item usage frequency method regarding CAT conditions were considered fixed conditions. To examine the effect of DIF, measurement precision, TIF and test effectiveness were calculated. Results show DIF has negative effects on measurement precision, TIF, and test effectiveness. In particular, statistically significant effects of the percentage DIF items and DIF type are observed on measurement precision.

Influences of internet access on civic knowledge measurement in Taiwan

Internet-related issues have influenced how civic knowledge is educated and measured. The International Civic and Citizenship Education Study (ICCS) is a well-known large-scale assessment concerning how civic knowledge is educated and measured globally. Regardless of the emerging roles of internet access and usage, the influences of internet access on civic knowledge have yet to be investigated in ICCS research. Hence, this study aims to study whether the multiple-choice items in the civic knowledge test of ICCS 2016 are affected by internet access, what causes the effect, and to what degree Internet access influences the measurement. Results indicated that the ICCS civic knowledge test included six differential item functioning (DIF) items on internet access, of which five favored families with internet access, and one favored families without internet access. In addition, the primary source of DIF items was highly related to socioeconomic status. Overall, the students in families where the internet was available possessed higher civic knowledge than the students who were not. In conclusion, this study provides evidence of item inequivalence and identifies suspicious sources. These results can be used as a basis to investigate related research on digital learning, online teaching, and social media engagement in civic literacy.

COMPARİNG DIFFERENTIAL ITEM FUNCTIONING BASED ON MULTILEVEL MIXTURE ITEM RESPONSE THEORY, MIXTURE ITEM RESPONSE THEORY AND MANIFEST GROUPS

Studies on the differential item functioning (DIF) are usually considered in the context of manifest groups. Recently, with the increase in the number of analyses conducted with mixture models, investigating the situations that cause differences between groups has come to the forefront. In addition, it is considered important to examine the DIF with mixture models in which levels are also handled. In this study, it is aimed to compare the results of the multilevel mixture item response theory (MMIRT) model and the mixture item response theory (MIRT) model and the results of the DIF analyses based on the manifest groups. The research sample consists of students who answered the second booklet in the eTIMSS 2019 exam and coded their gender. The answers given to 15 items were analyzed with the Mantel Haenszel (MH) method for the gender variable according to the manifest groups, and with the selection of the most appropriate models by varying the number of groups and the number of levels according to the MIRT model and the MMIRT model. DIF analyses of the obtained latent groups were also performed with the MH method. In the light of the findings, the number of items displaying DIF in both the MIRT model and the MMIRT model is higher than the manifest groups. While only one item displayed DIF in the analysis according to gender, 14 items displayed DIF according to the MIRT model and seven items displayed DIF according to the MMIRT model. There is not a complete overlap in the number of DIF items and DIF effect sizes found as a result of the MIRT model and MMIRT model analyses. For this reason, a level analysis should be conducted before the analyses and if there is multi-levelness, the analyses should be conducted by taking this situation into consideration.

DIF Analysis with Unknown Groups and Anchor Items

Ensuring fairness in instruments like survey questionnaires or educational tests is crucial. One way to address this is by a Differential Item Functioning (DIF) analysis, which examines if different subgroups respond differently to a particular item, controlling for their overall latent construct level. DIF analysis is typically conducted to assess measurement invariance at the item level. Traditional DIF analysis methods require knowing the comparison groups (reference and focal groups) and anchor items (a subset of DIF-free items). Such prior knowledge may not always be available, and psychometric methods have been proposed for DIF analysis when one piece of information is unknown. More specifically, when the comparison groups are unknown while anchor items are known, latent DIF analysis methods have been proposed that estimate the unknown groups by latent classes. When anchor items are unknown while comparison groups are known, methods have also been proposed, typically under a sparsity assumption – the number of DIF items is not too large. However, DIF analysis when both pieces of information are unknown has not received much attention. This paper proposes a general statistical framework under this setting. In the proposed framework, we model the unknown groups by latent classes and introduce item-specific DIF parameters to capture the DIF effects. Assuming the number of DIF items is relatively small, an L1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_1$$\\end{document}-regularised estimator is proposed to simultaneously identify the latent classes and the DIF items. A computationally efficient Expectation-Maximisation (EM) algorithm is developed to solve the non-smooth optimisation problem for the regularised estimator. The performance of the proposed method is evaluated by simulation studies and an application to item response data from a real-world educational test.

Bayesian inference with spike-and-slab priors for differential item functioning detection in a multiple-group IRT tree model

Group differences have practical implications in analysing data from achievement tests or questionnaires. In the current study, we develop a model that accounts for between-group differences, differential item functioning (DIF), latent factors, and missing item response data simultaneously. Different from most of the present DIF studies where one has to iteratively select anchor items and detect DIF items, we achieve DIF detection and parameter estimation simultaneously by properly reparameterizing model parameters and applying some spike-and-slab priors (Ishwaran & Rao, Spike and slab variable selection: frequentist and Bayesian strategies. Ann Stat. 2005a;33:730–773; Ročková & George, The spike-and-slab LASSO. J Am Stat Assoc. 2018;113:431–444) in Bayesian estimation. Simulation studies are conducted to illustrate the validation of the proposed estimation procedure and the efficiency of DIF detection. The proposed method is further applied to a real dataset for illustration.

Detecting differential item functioning with multiple causes: A comparison of three methods

Differential item functioning (DIF) often arises from multiple sources. Within the context of multidimensional item response theory, this study examined DIF items with varying secondary dimensions using the three DIF methods: SIBTEST, Mantel-Haenszel, and logistic regression. The effect of the number of secondary dimensions on DIF detection rates was investigated with several test conditions such as the percentage of DIF items, item loadings, the correlation between primary and secondary dimensions, and DIF magnitude. Results indicate that both the Mantel–Haenszel and logistic regression procedures performed effectively in detecting DIF when multiple secondary dimensions were present.

Gender-Based Differential Item Function for the Positive and Negative Semantic Dimensions of the Relationship Satisfaction Scale with Item Response Theory.

Relationship satisfaction is at the core of a robust social life and is essential to mental health. The positive and negative semantic dimensions of the relationship satisfaction (PN-SMD) scale is considered in the field of relationship studies to be a reliable tool for assessing the quality of a person's interpersonal relationships. This study evaluated the psychometric properties of the PN-SMD scale by conducting multidimensional item response theory (MIRT) and differential item functioning (DIF) analyses, both of which are emerging assessment methods that focus on individual items. We recruited 511 Chinese undergraduate students for this study. Construct validity, internal consistency, and concurrent validity were assessed, and MIRT and DIF analyses were conducted. Five of the 14 items were found to have gender-based DIF traits, affecting the scale's construct validity. A revised nine-item scale (DIF items excluded) had a significantly better model fit and demonstrated comparable concurrent validity to the original scale. The implications of our results and future research directions are discussed.

Measurement bias in caregiver-report of early childhood behavior problems across demographic factors in an ECHO-wide diverse sample.

Research and clinical practice rely heavily on caregiver-report measures, such as the Child Behavior Checklist 1.5-5 (CBCL/1.5-5), to gather information about early childhood behavior problems and to screen for child psychopathology. While studies have shown that demographic variables influence caregiver ratings of behavior problems, the extent to which the CBCL/1.5-5 functions equivalently at the item level across diverse samples is unknown. Item-level data of CBCL/1.5-5 from a large sample of young children (N=9087) were drawn from 26 cohorts in the Environmental influences on Child Health Outcomes program. Factor analyses and the alignment method were applied to examine measurement invariance (MI) and differential item functioning (DIF) across child (age, sex, bilingual status, and neurodevelopmental disorders), and caregiver (sex, education level, household income level, depression, and language version administered) characteristics. Child race was examined in sensitivity analyses. Items with the most impactful DIF across child and caregiver groupings were identified for Internalizing, Externalizing, and Total Problems. The robust item sets, excluding the high DIF items, showed good reliability and high correlation with the original Internalizing and Total Problems scales, with lower reliability for Externalizing. Language version of CBCL administration, education level and sex of the caregiver respondent showed the most significant impact on MI, followed by child age. Sensitivity analyses revealed that child race has a unique impact on DIF over and above socioeconomic status. The CBCL/1.5-5, a caregiver-report measure of early childhood behavior problems, showed bias across demographic groups. Robust item sets with less DIF can measure Internalizing and Total Problems equally as well as the full item sets, with slightly lower reliability for Externalizing, and can be crosswalked to the metric of the full item set, enabling calculation of normed T scores based on more robust item sets.

DIF Statistical Inference Without Knowing Anchoring Items

Establishing the invariance property of an instrument (e.g., a questionnaire or test) is a key step for establishing its measurement validity. Measurement invariance is typically assessed by differential item functioning (DIF) analysis, i.e., detecting DIF items whose response distribution depends not only on the latent trait measured by the instrument but also on the group membership. DIF analysis is confounded by the group difference in the latent trait distributions. Many DIF analyses require knowing several anchor items that are DIF-free in order to draw inferences on whether each of the rest is a DIF item, where the anchor items are used to identify the latent trait distributions. When no prior information on anchor items is available, or some anchor items are misspecified, item purification methods and regularized estimation methods can be used. The former iteratively purifies the anchor set by a stepwise model selection procedure, and the latter selects the DIF-free items by a LASSO-type regularization approach. Unfortunately, unlike the methods based on a correctly specified anchor set, these methods are not guaranteed to provide valid statistical inference (e.g., confidence intervals and p-values). In this paper, we propose a new method for DIF analysis under a multiple indicators and multiple causes (MIMIC) model for DIF. This method adopts a minimal L_1 norm condition for identifying the latent trait distributions. Without requiring prior knowledge about an anchor set, it can accurately estimate the DIF effects of individual items and further draw valid statistical inferences for quantifying the uncertainty. Specifically, the inference results allow us to control the type-I error for DIF detection, which may not be possible with item purification and regularized estimation methods. We conduct simulation studies to evaluate the performance of the proposed method and compare it with the anchor-set-based likelihood ratio test approach and the LASSO approach. The proposed method is applied to analysing the three personality scales of the Eysenck personality questionnaire-revised (EPQ-R).

Evaluation of psychometric properties of the Dental Anxiety Inventory (DAI-36) questionnaire using iterative hybrid ordinal logistic: Differential item functioning (DIF).

Dental Anxiety Inventory (DAI-36) questionnaire is an instrument for assessing dental anxiety. The different perceptions of the questionnaire items in the individual at the same level of underlying dental anxiety may lead to different reported dental anxieties. This study aims to determine the differential item functioning (DIF) of the DAI-36 measure items. The DAI-36 was completed by 950 participants. An iterative hybrid ordinal logistic regression model was used to detect DIF across gender, education, and age groups. DIF analysis was done by lordif package in R3.1.3 software. The chi-square statistics declared 7, 7, and 4 nonuniform DIF items, and 2, 5, and 4 uniform DIF items across gender, education, and age groups, respectively. ΔR2 was always lower than 0.07 in all uniform and nonuniform DIF items. However, Δβ1 revealed significant uniform DIF in items 1 and 8 across gender (Δβ1(item 1)=0.0137, Δβ1(item 8)=0.0124) and items 22 and 27 across age categories (Δβ1(item 22)=0.0110, Δβ1(item 27)=0.0136). DIF items had no large magnitude or cancel out each other, so statements phrased in the DAI-36 questionnaire have equivalent meaning across participants, regardless of their gender, education, and age groups.

Wald χ2 Test for Differential Item Functioning Detection with Polytomous Items in Multilevel Data

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord’s Wald [Formula: see text] test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

Differential language performance of Mandarin-speaking children with and without hearing loss

ABSTRACT The current study investigated the differential item functioning (DIF) on language assessment between Mandarin-speaking children with and without hearing loss (HL). Particularly, it explored the differences between the children in specific linguistic aspects. A total of 253 Mandarin-speaking children, aged three to six years old, participated, including 140 children without HL and 113 with HL. All children took the Revised Preschooler Language Assessment (RPLA) to assess their language ability. The RPLA items were categorized into different linguistic aspects, including semantics, syntax and pragmatics. The DIF analysis was conducted and the results revealed several statistically significant DIF items favouring children with and without HL, showing both the advantages and difficulties faced by children with HL. Items that favoured children with HL included directional words (semantics), comparative sentences (syntax), analogies (semantics), producing complete sentences (pragmatics/syntax), and retelling stories (pragmatics). Items that favoured children without HL included wh-questions (semantics), sentences containing connectives (syntax), logical words (semantics), comprehension of long paragraphs (semantics), and identifying semantic features (semantics). The results shed light on the advantages, which may benefit children with HL due to early intervention. In addition, suggestions on the difficulties were also provided, which these children may encounter compared to their hearing counterparts.

Bayesian penalty methods for evaluating measurement invariance in moderated nonlinear factor analysis.

Measurement invariance (MI) is one of the main psychometric requirements for analyses that focus on potentially heterogeneous populations. MI allows researchers to compare latent factor scores across persons from different subgroups, whereas if a measure is not invariant across all items and persons then such comparisons may be misleading. If full MI does not hold further testing may identify problematic items showing differential item functioning (DIF). Most methods developed to test DIF focused on simple scenarios often with comparisons across two groups. In practical applications, this is an oversimplification if many grouping variables (e.g., gender, race) or continuous covariates (e.g., age) exist that might influence the measurement properties of items; these variables are often correlated, making traditional tests that consider each variable separately less useful. Here, we propose the application of Bayesian Moderated Nonlinear Factor Analysis to overcome limitations of traditional approaches to detect DIF. We investigate how modern Bayesian shrinkage priors can be used to identify DIF items in situations with many groups and continuous covariates. We compare the performance of lasso-type, spike-and-slab, and global-local shrinkage priors (e.g., horseshoe) to standard normal and small variance priors. Results indicate that spike-and-slab and lasso priors outperform the other priors. Horseshoe priors provide slightly lower power compared to lasso and spike-and-slab priors. Small variance priors result in very low power to detect DIF with sample sizes below 800, and normal priors may produce severely inflated type I error rates. We illustrate the approach with data from the PISA 2018 study. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

An investigation of item bias in the four-tier diagnostic test using Rasch model

The existence of item bias in a set of measuring instruments can threaten the instrument’s validity. Based on the Rasch model, this study evaluated item bias in the four-tier heat and temperature diagnostic test (4T-HTDT). This study used a cross-sectional quantitative survey method. There were 241 students selected using a stratified random sampling technique. The 4T-HTDT instrument consisted of 20 items grouped into five concept groups. Students’ conceptual understanding was grouped into five categories, namely scientific knowledge (Rating=5), false positive (Rating=4), false negative (Rating=3), misconceptions (Rating=2), and lack of knowledge (Rating=1). The differential item functioning (DIF) score was used to evaluate item bias in the 4T-HTDT. Bias was reviewed based on the respondent’s gender, class, and school. The item has DIF if the probability value is <5%. The results showed that 35% (7 out of 20 items) spread over five groups of heat and temperature concepts were biased. However, excluding seven DIF items from the measurement set would not significantly affect the composition and distribution of items. Thus, the 13 items in the 4T-HTDT instrument are free from bias and can be used to evaluate the conceptual understanding of high school students.

Using a Generalized Logistic Regression Method to Detect Differential Item Functioning With Multiple Groups in Cognitive Diagnostic Tests.

Items with the presence of differential item functioning (DIF) will compromise the validity and fairness of a test. Studies have investigated the DIF effect in the context of cognitive diagnostic assessment (CDA), and some DIF detection methods have been proposed. Most of these methods are mainly designed to perform the presence of DIF between two groups; however, empirical situations may contain more than two groups. To date, only a handful of studies have detected the DIF effect with multiple groups in the CDA context. This study uses the generalized logistic regression (GLR) method to detect DIF items by using the estimated attribute profile as matching criteria. A simulation study is conducted to examine the performance of the two GLR methods, GLR-based Wald test (GLR-Wald) and GLR-based likelihood ratio test (GLR-LRT), in detecting the DIF items, the results based on the ordinary Wald test are also reported. Results show that (1) both GLR-Wald and GLR-LRT have more reasonable performance in controlling Type I error rates than the ordinary Wald test in most conditions; (2) the GLR method also produces higher empirical rejection rates than the ordinary Wald test in most conditions; and (3) using the estimated attribute profile as the matching criteria can produce similar Type I error rates and empirical rejection rates for GLR-Wald and GLR-LRT. A real data example is also analyzed to illustrate the application of these DIF detection methods in multiple groups.

Combining Item Purification and Multiple Comparison Adjustment Methods in Detection of Differential Item Functioning

Many of the differential item functioning (DIF) detection methods rely on a principle of testing for DIF item by item, while considering the rest of the items or at least some of them being DIF-free. Computational algorithms of these DIF detection methods involve the selection of DIF-free items in an iterative procedure called item purification. Another aspect is the need to correct for multiple comparisons, which can be done with a number of existing multiple comparison adjustment methods. In this article, we demonstrate that implementation of these two controlling procedures together may have an impact on which items are detected as DIF items. We propose an iterative algorithm combining item purification and adjustment for multiple comparisons. Pleasant properties of the newly proposed algorithm are shown with a simulation study. The method is demonstrated on a real data example.

Rasch-tree 방법을 이용한 PISA 2015 과학검사의 차별기능문항 탐색

The purpose of this study is to introduce the Rasch-tree, which is Rasch model-based recursive partitioning, as a new method for differential item functioning(DIF) detection and to examine its usefulness by exploring DIF in PISA 2015 Science test data with the Rasch-tree. This study used 16 items binary response of PISA 2015 Science test and 41 variables related to the student, parent, family, teacher, science, and ICT. The results of this study showed that there are four subgroups splitted by the three variables, ‘Enjoyment of Science(JOYSCICE)’, ‘gender’, and ‘Child’s past science activities(PRESUPP)’, and the two DIF items were detected. These items had large differences in the estimate item difficulty parameters between these four subgroups, especially the group of students with high level of JOYSCIE and the group of female students with relatively low level of JOYSCIE. For these results, it was confirmed that the Rasch-tree could be effective for exploring DIF between latent classes by considering simultaneously multiple trait variables.

The Impact of Missing Data on the Performances of DIF Detection Methods

This study analyzed the impact of missing data techniques on performances of two differential item functioning (DIF) detection methods (Mantel Haenszel and Multiple Indicator and Multiple Causes) under missing completely at random missing data mechanism. Percentage of missing data was set at 5% and 15%. Zero imputation, listwise deletion and fractional hot-deck imputation were used to handle missing data. The data set of the study consisted of 17 items in the S12 item cluster of Programme for International Student Assessment (PISA) 2015 science test. Results showed that fractional hot-deck imputation produced the best results in identifying DIF items in all conditions and it had also the closest DIF values to the values obtained from complete data set. It was also found that multiple indicator and multiple causes method was more adversely affected than Mantel Haenszel by the presence of missing data.

Evaluating the Consequential Validity of the Research-Based Early Mathematics Assessment

Consequential validity (often referred to as “test fairness” in practice) is an essential aspect of educational measurement. This study evaluated the consequential validity of the Research-Based Early Mathematics Assessment (REMA). A sample of 627 children from PreK to second grade was collected using the short form of the REMA. We conducted two sets of analyses with different foci (item- or scale-level) for validation: differential item functioning (DIF) and consequential validity ratio (CVR) analyses. The analyses focused on the demographic subgroups of gender, English Language Learner status, and race/ethnicity. We found a low percentage of DIF items (less than 3%) and high CVRs (ranging from 96 to 98%). Both findings support the consequential validity and thus “fairness” of the REMA.

Quality of life in multiple scenarios: The impact of work mode and social contact quantity.

During the COVID-19 pandemic, many employees were encouraged to temporarily work from home as an attempt to decrease social contact with others. However, the employees' quality of life (QoL) may have been threatened by this mode of working. This study, therefore, aims to explore the employees' QoL given the new mode of working from home (WFH) as a result of the pandemic vs. working in the office (WIO), the amount of social contact that they were exposed to, and the ratio of face-to-face contact that they had. A total of 803 WFH employees and 588 WIO employees' QoL was assessed during the same time period using the WHOQOL-BREF, which contains four domains: physical health, psychological health, social relationship, and the environment. We then divided the participants into 16 groups in accordance with the levels of work mode, social contact quantity, and face-to-face contact ratio-forming a case-control study. A differential item functioning (DIF) analysis was used to analyze the responses on the WHOQOL-BREF under the 4-dimensional rating scale model fitting. The results indicated that WFH employees' QoL was superior to that of WIO employees. The relationship between the WFH mode and the employees' QoL was specifically moderated by the amount of social contact and the ratio of face-to-face contact that was experienced. The results further demonstrated that the increased amount of non-face-to-face contact was better for WFH employees' QoL than that of WIO employees. In conclusion, the WFH mode was practical during the COVID-19 pandemic, as our findings indicated that WFH employees' QoL was better than WIO employees' QoL. However, maintaining social connections is equally important as this allows employees to perform better at their jobs and maintain such performance. The employees with a higher number of social support had a better QoL. Additionally, the facets detected as DIF items provided implications for the QoL with regard to the research methodology and insight into factors affecting the employees' QoL.