Differential Item Functioning Analysis Research Articles

Alignment as an Alternative to Anchor Purification in DIF Analyses

Unbalanced differential item functioning (DIF) occurs when the items favoring group 1 are not balanced by items favoring group 2. This creates false differences in the total score used for matching examinee subgroups or inaccuracy in the scaling used to bring the groups to commensurable metrics. Historically, unbalanced DIF has been addressed by removing DIF items from the set of anchor items, often called purification. A recently developed alignment procedure minimizes the impact of the DIF items in scaling the groups without removing them. In this study, when the data followed a two-parameter model, the results using alignment or a purified anchor were generally comparable when the DIF was balanced or when the DIF was unbalanced but no more than 20% of the items contained DIF. With a larger percentage of unbalanced DIF, the alignment procedure yielded more accurate estimates of the group means and the magnitude of the DIF.

Psychometric Validation and Reorganization of the Desired Results Developmental Profile

This study reports an independent investigation of the psychometric properties of Desired Results Developmental Profile (DRDP), a teacher-rated measure of school readiness for preschool-aged children. In a sample of 2,031 low-income, 3- to 5-year-old children attending Head Start, we tested three measurement models: a higher order one-factor model, a seven-factor model, and a five-factor model. To explore the appropriateness of the DRDP for use with diverse populations of young children, we used multiple group and differential item functioning (DIF) analyses to determine whether the DRDP works differently for dual language learners (DLL) and non-DLLs. The proposed five-factor structure fits the data best, with greater face and statistical validity. Using this conceptually driven factor structure, the multiple group analyses were robust for DLL and non-DLL preschool students. More than half of the items on the DRDP displayed little DIF. Items measuring emergent language and literacy exhibited DIF favoring non-DLL children.

Evaluating Differential Item Functioning (DIF) of the Chinese Version of the Behavioral Activation for Depression Scale (C-BADS)

Abstract. Depression is prevalent in both China and Taiwan, and Behavioral Activation (BA), an Evidence-Based Treatment (EBT) for depression, is ideally suited for cross cultural implementation. As a first step, the current study examined cross cultural differences in the understanding of BA constructs, by investigating item level differences in functioning between the English and Chinese versions of Behavioral Activation for Depression Scale (BADS and C-BADS; Kanter, Mulick, Busch, Berlin, & Martell, 2007 ; Li, Ding, Kanter, Zeng, & Yang, 2014) . 752 college students were recruited from China, Taiwan, and the United States. Factorial invariance-based Differential Item Functioning (DIF) analysis was used to study item level differences in functioning for the BADS and C-BADS. Results. DIF was observed in the majority of BADS items, with items in the avoidance and impairment factors showing the greatest DIF. The constructs of avoidance and impairment demonstrate less cross-cultural generalizability compared to the activation construct. Suggestions for the implementation of DIF analysis for future cross cultural psychometric studies, and further modification of the C-BADS as a clinical assessment tool in China and Taiwan, are discussed.

Interview Administration of PROMIS Depression and Anxiety Short Forms.

Background:Health literacy reflects a person's reading and numeracy abilities applied to understanding health-related information. These skills may influence how patients report symptoms, leading to underestimates or overestimates of symptom severity. No prior studies have examined health literacy measurement bias.Objective:The purpose of the current study was to determine whether PROMIS (Patient-Reported Outcomes Measurement Information System) anxiety and depression short forms, administered by interview, capture symptoms equally across health literacy groups. We examined the psychometric properties of PROMIS anxiety and depression short forms using differential item functioning (DIF) analysis by level of health literacy.Methods:The sample analyzed included 888 adults, age 55 to 74 years, in Chicago, IL. Health literacy was measured using the Test of Functional Health Literacy in Adults. PROMIS short forms assessed anxiety and depression.Key Results:DIF was present in 3 of 8 depression items, and 3 of 7 anxiety items. All items flagged for DIF had lower item-slopes for people with limited health literacy.Conclusions:Items with DIF were less strongly related to anxiety and depression, and thus less precise. Overall, impact of DIF on PROMIS scores was negligible, likely mitigated by interview administration. Although overall test impact of health literacy was minimal, DIF analyses flagged items that were potentially too complex for people with limited health literacy. Design and validation of patient-reported surveys should incorporate respondents with a range of health literacy and methods to identify and reduce measurement bias. [HLRP: Health Literacy Research and Practice. 2019;3(3):e196–e204.]Plain Language Summary:This study suggests that people with limited health literacy may respond differently to questions about depression and anxiety than people with adequate health literacy. Therefore, it is important to be aware of differences in literacy ability when creating and using questionnaires.

Gender-based differential item functioning in measures of eating pathology.

Eating disorder (ED) symptoms are common and impairing in males, despite their perception as "female" disorders. As existing self-report symptom measures were developed and primarily validated in women, there is a need to establish the utility of these measures in men. The present study used differential item functioning (DIF) analyses to explore whether item endorsement differed by gender for three commonly used ED symptom measures. Participants were undergraduate men (n = 1,083) and women (n = 2,424) from three universities in the United States. Global scores on the Eating Attitudes Test-26 (EAT-26), Eating Disorder Examination Questionnaire (EDEQ), and Eating Disorder Diagnostic Scale for DSM-IV (EDDS) were examined. Tests of DIF were conducted by regressing each item against its composite scale score, and then comparing fit and variance explained (R2 ) to a model with the interaction of item*gender. The clinical significance threshold for DIF is ΔR2 ≥ 0.13. There was no evidence of clinically significant DIF within the EAT-26, EDEQ, or EDDS. Findings suggest that the examined measures perform similarly for undergraduate men and women, supporting their use in nonclinical male samples. However, development and testing of items reflecting ED symptoms that more commonly occur in males (e.g., muscularity-oriented behaviors) is encouraged.

Multilevel Generalized Mantel-Haenszel for Differential Item Functioning Detection

Research has demonstrated that when data are collected in a multilevel framework, standard single level differential item functioning (DIF) analyses can yield incorrect results, particularly inflated Type I error rates. Prior research in this area has focused almost exclusively on dichotomous items. Thus, the purpose of this simulation study was to examine the performance of the Generalized Mantel-Haenszel (GMH) procedure and a Multilevel GMH (MGMH) procedure for the detection of uniform differential item functioning (DIF) in the presence of multilevel data with polytomous items. Multilevel data were generated with manipulated factors (e.g., intraclass correction, subjects per cluster) to examine Type I error rates and statistical power to detect DIF. Results highlight the differences in DIF detection when the analytic strategy matches the data structure. Specifically, the GMH had an inflated Type I error rate across conditions, and thus an artificially high power rate. Alternatively, the MGMH had good power rates while maintaining control of the Type I error rate. Directions for future research are provided.

A new assessment to monitor student performance in introductory neurophysiology: Electrochemical Gradients Assessment Device.

The basis for understanding neurophysiology is understanding ion movement across cell membranes. Students in introductory courses recognize ion concentration gradients as a driving force for ion movement but struggle to simultaneously account for electrical charge gradients. We developed a 17-multiple-choice item assessment of students' understanding of electrochemical gradients and resistance in neurophysiology, the Electrochemical Gradients Assessment Device (EGAD). We investigated the internal evidence validity of the assessment by analyzing item characteristic curves of score probability and student ability for each question, and a Wright map of student scores and ability. We used linear mixed-effect regression to test student performance and ability. Our assessment discriminated students with average ability (weighted likelihood estimate: -2 to 1.5 Θ); however, it was not as effective at discriminating students at the highest ability (weighted likelihood estimate: >2 Θ). We determined the assessment could capture changes in both assessment scores (model r2 = 0.51, P < 0.001, n = 444) and ability estimates (model r2 = 0.47, P < 0.001, n = 444) after a simulation-based laboratory and course instruction for 222 students. Differential item function analysis determined that each item on the assessment performed equitably for all students, regardless of gender, race/ethnicity, or economic status. Overall, we found that men scored higher (r2 = 0.51, P = 0.014, n = 444) and had higher ability scores (P = 0.003) on the EGAD assessment. Caucasian students of both genders were positively correlated with score (r2 = 0.51, P < 0.001, n = 444) and ability (r2 = 0.47, P < 0.001, n = 444). Based on the evidence gathered through our analyses, the scores obtained from the EGAD can distinguish between levels of content knowledge on neurophysiology principles for students in introductory physiology courses.

Rasch analysis of the Katz index of independence in activities of daily living in senior citizens from Antioquia, Colombia

Improving Katz index interpretability contributes to a better understanding of its results facilitating decision making and intervention planning.Objective: To evaluate the validity of the Katz geriatric scale in senior citizens from a Rasch model perspective.Methodology: Secondary data analysis of a population-based, cross-sectional study of health status in 4,023 senior citizens conducted in Antioquia, Colombia, in 2012. Item fit, differential item functioning, unidimensionality, reliability and item-person map analyses were performed.Results: All items adjusted to the model. Transferring and Continence showed high differential item functioning with a considerable, but statistically non-significant impact on the overall estimation. Person and Item reliability were 0.80 and 0.90 respectively. The Rasch measurement explained 61.6% of variance.Conclusion: A unidimensional interval level measure can be obtained from the Katz instrument. It is important to do more research on differential functioning of Katz’s items in a way in which validity arguments on its generalization to various subpopulations could be strengthened and score interpretability could be improved.

The Vancouver Index of Acculturation (VIA): New evidence on dimensionality and measurement invariance across two cultural settings

The Vancouver Index of Acculturation (VIA) is a self-report bidimensional instrument that has been employed in several cultural contexts to assess migrants’ orientations toward mainstream and heritage traditions. Although it has shown good reliability and validity in the past, recent theoretical developments and empirical studies have suggested the VIA might assess more than two dimensions. Moreover, measurement invariance of this instrument across demographic subgroups has never been assessed before. With the aim of evaluating these psychometric issues, exploratory factor analysis and Rasch modeling were conducted on data from two samples of immigrants to Canada (N = 224) and to Italy (N = 266). Results supported the hypothesized two-factor structure, with Rasch modeling showing that items within a subscale differed in terms of “difficulty” in being endorsed, but still belonged to a single factor. Differential item functioning analysis results showed measurement invariance across gender, age and education subgroups. Most results converged across the two samples.

Nurse practitioner primary care organizational climate questionnaire: Item response theory and differential item functioning.

To validate Nurse Practitioner Primary Care Organizational Climate Questionnaire (NP-PCOCQ) items using item response theory (IRT) models and conduct differential item functioning (DIF) analysis to test the item functioning among nurse practitioners (NPs) practicing in different U.S. states with variable regulations governing NP practice. Nurse Practitioner Primary Care Organizational Climate Questionnaire is the only NP-specific tool measuring NP work environment and is being used in different U.S. states with variable NP scope of practice regulations and internationally to produce evidence about NP work environments within their organisations. Cross-sectional survey design was used to collect data from 278 primary care NPs in New York (NY) and 314 NPs in Massachusetts (MA). NPs completed the 29-item NP-PCOCQ. Data collection involved an online survey in NY and a mail survey in MA in 2012. We used Samejima's graded response model for IRT and ordinal logistic regression for DIF analysis. A STROBE checklist was completed. IRT models yielded discrimination parameters ranging from 0.98-4.65 in NY and 1.25-6.94 in MA. Item difficulty parameters were within -3 to +3 range, suggesting a fair range of item difficulties exist in the scale. Only five of the 29 items on NP-PCOCQ exhibited DIF, suggesting some other state-related factor besides the measured construct influenced item responses; thus, the items were removed. Our findings indicate that a shortened, 24-item NP-PCOCQ is capable of measuring organisational climate of NPs practicing in different U.S. states. NP-PCOCQ can be used in future research to measure NP work environment. The tool can also be used by practice administrators to assess NP work environment and identify deficiencies to address them. This evidence about NP work environment can be used by practice administrators to promote favourable work environments for NPs to deliver high-quality care.

Preliminary psychometric validation of a brief screening tool for athlete mental health among male elite athletes: the Athlete Psychological Strain Questionnaire

There is growing interest in the area of elite athlete mental health, however brief, valid and reliable screening instruments designed to identify early indictors of athlete-specific distress and potential mental health symptoms are lacking. This study sought to develop a brief screening instrument for athlete populations – the Athlete Psychological Strain Questionnaire (APSQ) – and examine convergent, divergent and construct validity. A two-stage psychometric validation study was undertaken. Self-report data was collected from 1,007 currently competing Australian elite male athletes (M = 23.67, SD = 4.16). The sample was randomly partitioned into calibration (n = 497) and validation (n = 510) samples. Exploratory and confirmatory factor analysis, and tests of differential item functioning were conducted. Exploratory factor analysis, with parallel analysis, conducted on the calibration sample supported a three-factor solution, with subscales assessing Self-Regulation, Performance and External Coping accounting for 50.44% of total scale variance. Confirmatory factor analysis supported this three-factor model, yielding excellent model fit indices (CFI = 0.976, TLI = 0.966, RMSEA = 0.055, SRMR = 0.032), with the Bayesian Information Criterion supporting the superordinate structure. Differential item functioning analysis indicated item-equivalence relative to athletes’ level of education and ethnicity. As predicted, a multivariate effect indicated higher APSQ scores for currently injured athletes (p = .040) with a univariate effect on the Performance subscale. The APSQ may help identify early symptoms of athlete psychological strain facilitating timely management. Replication and validation studies in broader samples, including female athletes and comparison with other athlete-specific, coping and stress measures are needed.

MIMIC Models for Uniform and Nonuniform DIF as Moderated Mediation Models.

In this article, the authors describe how multiple indicators multiple cause (MIMIC) models for studying uniform and nonuniform differential item functioning (DIF) can be conceptualized as mediation and moderated mediation models. Conceptualizing DIF within the context of a moderated mediation model helps to understand DIF as the effect of some variable on measurements that is not accounted for by the latent variable of interest. In addition, useful concepts and ideas from the mediation and moderation literature can be applied to DIF analysis: (a) improving the understanding of uniform and nonuniform DIF as direct effects and interactions, (b) understanding the implication of indirect effects in DIF analysis, (c) clarifying the interpretation of the "uniform DIF parameter" in the presence of nonuniform DIF, and (d) probing interactions and using the concept of "conditional effects" to better understand the patterns of DIF across the range of the latent variable.

Measurement and Implications of Perfectionism in South Korea and the United States

We evaluated the Short Almost Perfect Scale (SAPS; Rice et al., 2014) in South Korean ( N = 306) and United States ( N = 259) samples. Results supported partial metric invariance for the standards (perfectionistic strivings) and discrepancy (perfectionistic concerns) factors, and for depression and self-esteem. Item responses were unaffected by gender- or age-related bias. Differential item functioning analyses and tests of scalar invariance revealed noninvariance for several items that might reflect culturally relevant issues. There were no differences in SAPS factor means based on invariance tests using the alignment method. For both samples, the standards factor was positively and comparably associated with self-esteem and inversely predicted depression; the discrepancy factor yielded effects in the opposite direction. Results generally supported the use of the SAPS for United States–Korean correlational comparisons. Future work could develop and evaluate additional indicators of perfectionistic concerns for cross-national studies, and increase representation in the samples and breadth of criterion factors.

An Integrated Approach to Bias in the Spanish Version of the General Health Questionnaire (GHQ-12)

Abstract. The aim of the paper is to illustrate a comprehensive approach to bias in an application of the General Health Questionnaire (GHQ-12) among native Spanish people and immigrants in Spain. A comprehensive evaluation of bias can detect combined effects of different types of bias and their causes. The GHQ-12, administered as part of the 2006 Spanish National Health Survey, was used for evaluating bias. The entire sample of 1,251 participants was composed of Spanish and immigrants from Morocco, Ecuador, Romania, and Colombia. Construct, method, and items bias were evaluated, using analyses of measurement invariance, response styles, and Differential Item Functioning (DIF), respectively. Integrating the results of the bias analyses, we found that the presence of bias was related to items asking about symptoms, notably dealing with feelings and personal perceptions. Furthermore, colloquialisms in the items led to bias in the comparison of Spanish speakers from Spain and South America (all of whom had Spanish as mother tongue). The comprehensive approach to bias showed the value of examining various forms of bias in an integrated manner.

Bifactor and Item Response Theory Analyses of Health Care Workers’ Attitudes Toward HIV-Infected People

Abstract. Health care workers’ attitudes toward people living with HIV/AIDS (PLWHA) can play an important role in the quality of life of PLWHA as well as in public health efforts in HIV prevention and treatment. The main aim of this study was to develop and evaluate the psychometric properties of an instrument to measure health care workers’ attitudes toward PLWHA using bifactor and IRT analyses. We applied bifactor and Item Response Theory (IRT) analyses to questionnaire development, evaluation, and refinement. A sample of 836 Italian health care workers was recruited in 2013, including medical practitioners, nurses, health care assistants, social workers, and psychologists. We tested a hierarchical item response model (bifactor model) with model fit comparisons with one-dimensional and five-dimensional models. The bifactor model was found to provide the best fit to the observed data. The results were used to construct a 15-item short form. The test information function showed that the scale was slightly more precise at higher levels of negative attitudes. Finally, the differential item functioning (DIF) analysis revealed that the items did not function differently across gender or professional groups. The instrument was found to have good criterion validity with regard to discriminatory practices toward HIV-infected patients.

Phys-MAPS: a programmatic physiology assessment for introductory and advanced undergraduates.

We describe the development of a new, freely available, online, programmatic-level assessment tool, Measuring Achievement and Progress in Science in Physiology, or Phys-MAPS ( http://cperl.lassp.cornell.edu/bio-maps ). Aligned with the conceptual frameworks of Core Principles of Physiology, and Vision and Change Core Concepts, Phys-MAPS can be used to evaluate student learning of core physiology concepts at multiple time points in an undergraduate physiology program, providing a valuable longitudinal tool to gain insight into student thinking and aid in the data-driven reform of physiology curricula. Phys-MAPS questions have a modified multiple true/false design and were developed using an iterative process, including student interviews and physiology expert review to verify scientific accuracy, appropriateness for physiology majors, and clarity. The final version of Phys-MAPS was tested with 2,600 students across 13 universities, has evidence of reliability, and has no significant statement biases. Over 90% of the physiology experts surveyed agreed that each Phys-MAPS statement was scientifically accurate and relevant to a physiology major. When testing each statement for bias, differential item functioning analysis demonstrated only a small effect size (<0.008) of any tested demographic variable. Regarding student performance, Phys-MAPS can also distinguish between lower and upper division students, both across different institutions (average overall scores increase with each level of class standing; two-way ANOVA, P < 0.001) and within each of three sample institutions (each ANOVA, P ≤ 0.001). Furthermore, at the level of individual concepts, only evolution and homeostasis do not demonstrate the typical increase across class standing, suggesting these concepts likely present consistent conceptual challenges for physiology students.

A probe into EFL learners’ emotioncy as a source of test bias: Insights from differential item functioning analysis

The development of unbiased tests is crucial in the arena of language testing in order to ensure validity. To date, studies of bias in language testing have mainly focused on factors such as gender, native language, or academic background, inter alia. However, bias may also result from psychological factors. Therefore, the present study investigates the role of English as a Foreign Language (EFL) test takers’ emotioncy, defined as the emotions evoked by senses that one holds for an entity, in their test performance. Specifically, this study aimed to examine emotioncy for the form as well as the meaning of 20 words to find out whether it can lead to differential functioning of the items on a vocabulary test. To this end, two emotioncy scales and a vocabulary test were designed. Then, based on the data collected from 235EFL students, the participants were bisected into the Low-Group and the High-Group, once based on their emotioncy scores for each word form and then based on their emotioncy scores for each word meaning. Subsequently, Rasch model-based Differential Item Functioning (DIF) analysis was performed across the two groups. The results showed that the vocabulary test items functioned differentially across the two groups in both form and meaning classifications, favoring the High-Group. Therefore, the study provides evidence for emotioncy as a psychological source of test bias and discusses implications for language testing stakeholders.

Examining test fairness across gender in a computerised reading test: A comparison between the Rasch-based DIF technique and MIMIC

Test fairness has been recognised as a fundamental requirement of test validation. Two quantitative approaches to investigate test fairness, the Rasch-based differential item functioning (DIF) detection method and a measurement invariance technique called multiple indicators, multiple causes (MIMIC), were adopted and compared in a test fairness study of the Pearson Test of English (PTE) Academic Reading test (n = 783). The Rasch partial credit model (PCM) showed no statistically significant uniform DIF across gender and, similarly, the MIMIC analysis showed that measurement invariance was maintained in the test. However, six pairs of significant non-uniform DIF (p &lt; 0.05) were found in the DIF analysis. A discussion of the results and post-hoc content analysis is presented and the theoretical and practical implications of the study for test developers and language assessment are discussed.

A computerized adaptive testing advancing the measurement of subjective well-being

This article aimed at developing an adaptive version of the subjective well-being (SWB) scale to measure a comprehensive concept of SWB among Chinese university students. Item response theory was employed to formulate the item bank of the SWB scale and computerized adaptive testing (CAT) for SWB (CAT-SWB), based on several commonly used SWB scales, after unidimensionality testing, model selection, local dependence testing, parameter estimation, item fit test and differential item functioning (DIF) analysis were performed. Finally, two CAT simulations using simulated-data and real-data were carried out to verify and evaluate the CAT-SWB. Results indicated that the proposed CAT-SWB had an excellent performance in that it largely reduces the number of test items and the length of test time without losing measurement precision.

Development and validation of a scale to measure the quality of patient medication counseling using Rasch model.

Objective:The purpose of this study is to develop and validate the psychometric properties of a scale for measuring the quality of patient medication counseling by using the Rasch model.Methods:In this study, the scale was developed based on the literature review. It consisted of 31 items across five subscales: introduction, problem identification, content, behavior, and conclusion. A convenient sample of community pharmacists was recruited from four major cities in Vietnam: Hanoi, Da Nang, Ho Chi Minh, and Can Tho. Data collection was conducted from June 10 to October 30, 2017. A Rasch analysis for polytomous data was performed to assess the suitability of the item and the reliability of the scale.Results:The research results showed that all items had a positive point-measure correlation coefficient between 0.47 and 0.77. All items had infit and outfit values in the optimal range between 0.5 and 1.5 except for D5, but its value was within acceptable range. Differential item function analysis indicated that all items had no DIF, except for items B4 and E4 containing moderate magnitude of DIF. Response category statistics found that there was a gradual increase in difficulty level from category 1 to 5 and no presence of reversal. Infit and outfit statistics of these categories were also considered good, with their values close to 1. The test result of the item characteristic curve and the person-item map showed that there were some overlapping items. Their appearance, however, might play an important role in measuring different aspects of construct. The overall scale reliability index (0.97) was high and the overall scale separation index (6.11) was good.Conclusions:The developed scale satisfied the requirements of the Rasch model. The scale is a useful tool that could be used to measure the quality of patient medication counseling among community pharmacists.