Differential Item Functioning Analysis Research Articles

Development of mobile CAT for patient feedback on pediatric consultations based on Rasch analysis of online techniques.

The Rasch Rating Scale Model (RSM) is widely used in questionnaire analysis, providing insights into how individuals respond to item-level stimuli. Existing software for Rasch RSM parameter estimation, while powerful, often presents a steep learning curve. An accessible online tool can greatly benefit novice users, particularly students and clinicians, by simplifying the analytical process. This study introduces an online tool, an intuitive online RSM analysis tool designed to facilitate questionnaire data analysis for applied researchers, students, and clinicians. The online tool employs the joint maximum likelihood method for estimation, yielding estimates, standard errors (SE), and fit statistics iteratively. A unique feature of the tool is its ability to visualize estimates on Google Maps with an opacity setting of 0, enhancing data interpretation through a user-friendly interface. This study outlines the estimation process and key features, employing data from 200 proxy participants who answered 20 5-point questions regarding doctor-patient and doctor-family interactions in pediatric consultations. Mobile computerized adaptive testing (CAT) was employed. The online tool offers 5 essential visual displays often utilized in Rasch analyses, including the Wright Map, KIDMAP, category probability curve, performance plot, and differential item functioning (DIF) graph. DIF analysis revealed that 2 items, concerning the doctor attentiveness and empathy toward the child illness, exhibited differences in female proxy participants' responses, indicating lower satisfaction with pediatricians. The online tool emerges as a user-friendly and efficient RSM analysis tool with notable advantages for newcomers, improving data visualization and comprehension. Its capacity to pinpoint key areas of concern, such as gender-related satisfaction disparities among proxy participants, enhances its utility in questionnaire analysis. The online tool holds promise as a valuable resource for researchers, students, and clinicians seeking accessible Rasch analysis solutions.

A cross-cultural study of differential item functioning analysis of beliefs about Language Learning Inventory (BALLI)

This research examines students' beliefs about language learning by evaluating the measurement equivalence of the beliefs about Language Learning Inventory (BALLI). The graded response model (GRM) was used to examine differential item functioning (DIF) in the BALLI across four different countries, namely: Iran, Malaysia, Indonesia, and Bangladesh. The sample was drawn from 1613, including 500 males and 1113 females who completed the online version of the BALLI, comprising five subscales of beliefs about foreign language aptitude, the difficulty of language learning, the nature of language learning, learning and communication strategies, and motivations and expectations. Most BALLI items showed non-uniform DIF. This finding implies that students in different countries had different perceptions of the BALLI items. Therefore, researchers should be very cautious about using the BALLI in different countries. This cross-cultural comparison may generate new insights into revising the BALLI items or developing another scale to compare students' beliefs about language learning in different countries.

Thriving from work questionnaire: Spanish translation and validation

BackgroundThriving from Work is a construct that has been highlighted as an important integrative positive worker well-being indicator that can be used in both research and practice. Recent public discourse emphasizes the important contributions that work should have on workers’ lives in positive and meaningful ways and the importance of valid and reliable instruments to measure worker well-being. The Thriving from Work Questionnaire measures how workers’ experiences of their work and conditions of work contributes in positive ways to their thriving both at and outside of work.MethodsThe purpose of this study was to translate the Thriving from Work Questionnaire from English to Spanish, and then validate the translated questionnaire in a sample of 8,795 finance workers in Peru and Mexico. We used item response theory models replicating methods that were used for the original validation studies. We conducted a differential item functioning analysis to evaluate any differences in the performance of models between Peru and Mexico. We evaluated criterion validity with organizational leadership, flourishing, vitality, community well-being, and worker’s home location socio-economic position.ResultsThe current study demonstrates that the Spanish (Peru/Mexico) questionnaire was found to be a reliable and valid measure of workers’ thriving from work. One item was dropped from the long-form version of the original U.S. questionnaire. Both the long and short form versions of the questionnaire had similar psychometric properties. Empirical reliability was high. Criterion validity was established as hypothesized relationships between constructs was supported. There were no differences in the performance of the model between countries suggesting utility across Latin American countries.ConclusionsThe current study demonstrated that the Spanish (Peru and Mexico) version of the questionnaire is both a reliable and valid measure of worker well-being in Latin America. Specific recommendations are made for the adaptation of the questionnaire and directions of future research.

Psychometric characteristics of individuals’ intention to engage in philanthropic activities measure: An early development and tryout

This brief article reports on the psychometric characteristics of the early development of individuals’ intention to engage in philanthropic activities questionnaire. In the innitial stage of development, the questionnaire was tried out by administering it online to a targeted sample. Out of 63 sample participants completing the questionnaire, only 43 data were appropriate for further analysis using the Rasch model. While the overall scale of the questionnaire suggested multidimensionality, subscales aligned with unidimensionality criteria. Acceptable reliability was observed in the overall scale and the Funding Philanthropic Brand Trust (FPBT) subscale, contrasting with inadequate reliability in the Philanthropic Brand Preference (PBP) and Pride of Affiliation (PoA) subscales, warranting further investigation. Specific items like Q17, Q31, Q42, and Q44 posed challenges, indicating difficulties in discussing social issues, cultural conformity in donations, valuing recognition, and seeking community support in philanthropy. Differential Item Functioning (DIF) analysis revealed demographic variations in responses, indicating diverse agreement levels based on age, gender, and education. These findings offer crucial insights for refining future questionnaire iterations, highlighting the need for a second round of development to address these limitations, with an expanded sample size to ensure robustness.

Investigating Differential Item Functioning of an International Mathematics Competition Items across Gender Groups

Mathematical problem-solving competitions have existed for over a century. Scholars report the gender gap in these competitions. As a result, it is necessary to determine whether any score difference between gender groups is attributable to a genuine difference or is the result of the exam itself. Thus, the current study specifically examined bias in one of the well-known mathematics competitions: the Kangaroo Mathematics competition. Determining the fairness of Kangaroo mathematics competition items across gender groups is crucial for creating accurate comparisons and avoiding unintended construct irrelevant bias. To examine the bias, Differential Item Functioning (DIF) analyses were conducted using Logistic Regression, Mantel-Haenszel, and Item Response Theory Likelihood Ratio Test DIF detection methods. After a series of investigations, out of 336 items, it was concluded that these mathematics items were free of DIF and bias across the gender groups. Further implications were discussed in detail regarding the validity and bias.

Applicability and clinical utility of the German rivermead post-concussion symptoms questionnaire in proxies of children after traumatic brain injury: an instrument validation study.

The German Rivermead Post-Concussion Symptoms Questionnaire (RPQ) can be used to assess post-concussion symptoms (PCS) after traumatic brain injury (TBI) in adults, adolescents, and children. In this study, we examined the psychometric properties of the German RPQ proxy version (N = 146) for children (8-12 years) after TBI at the item, total and scale score level. Construct validity was analyzed using rank correlations with the proxy-assessed Post-Concussion Symptoms Inventory (PCSI-P), the Patient Health Questionnaire 9 (PHQ-9), and the Generalized Anxiety Disorder Scale 7 (GAD-7). Furthermore, sensitivity testing was performed concerning subjects' sociodemographic and injury-related characteristics. Differential item functioning (DIF) was analyzed to assess the comparability of RPQ proxy ratings for children with those for adolescents. Good internal consistency was demonstrated regarding Cronbach's α (0.81-0.90) and McDonald's ω (0.84-0.92). The factorial validity of a three-factor model was superior to the original one-factor model. Proxy ratings of the RPQ total and scale scores were strongly correlated with the PCSI-P (ϱ = 0.50-0.69), as well as moderately to strongly correlated with the PHQ-9 (ϱ = 0.49-0.65) and the GAD-7 (ϱ = 0.44-0.64). The DIF analysis revealed no relevant differences between the child and adolescent proxy versions. The German RPQ proxy is a psychometrically reliable and valid instrument for assessing PCS in children after TBI. Therefore, RPQ self- and proxy-ratings can be used to assess PCS in childhood as well as along the lifespan of an individual after TBI.

The Ethiopian Cognitive Assessment battery in Schizophrenia (ECAS): a validation study

Cognitive impairment is common in people with schizophrenia (PWS). To detect the presence and its consequences, cognitive measures with sound psychometric properties are needed. However, these are lacking especially in low-income countries. Hence, we developed the Ethiopian Cognitive Assessment battery in Schizophrenia (ECAS). In this study, we evaluated the psychometric properties of the ECAS in a cross-sectional study involving 350 PWS. Confirmatory factor analysis demonstrated a one-factor solution. ECAS score correlated significantly but weakly with a disability measure (r = −0.13, p = 0.02) and symptom dimensions of PANSS (r between −0.12 and −0.29, p < 0.05), except for positive symptoms (r = −0.10, p > 0.05). Years of education (β = 0.12, 95% CI (0.09, 0.14), p < 0.001), male sex (β = 0.22, 95% CI (0.05, 0.39)), age β = −0.02, 95% CI (−0.03, −0.01), and medication side effects (β = −0.03, 95% CI (−0.06, −0.01), p = 0.021) were significantly associated with the composite score of ECAS. The Item Response Theory analysis showed that the tool best functions among participants with moderate cognitive impairment (difficulty coefficient between −1.12 and 0.27). The Differential Item Functioning analyses showed that education had a positive contribution on Digit Symbol Substitution Test (MH OR = 2.64, 95% CI (1.34, 5.20)). The results showed that ECAS is valid in assessing cognition in PWS in low-resource settings.

Attitude Is Everything: Examining Acceptance of Violence Items for Differential Item Functioning Across Heterosexual and Sexual Minority Youth.

Acceptance of dating violence (ADV) is a cognitive risk factor for violence perpetration and a common target of prevention programs. However, frequently used items assessing ADV are characterized by heteronormative item wording, and limited research has evaluated the degree to which ADV items function equivalently for both heterosexual and sexual minority youth (SMY). The current study sought to determine if there are differences in the way heterosexual and SMY respond to ADV survey items. Secondary data from a total of 2,014 adolescents (Mage = 16.78) were used to examine differences in ADV. Results of differential item functioning analysis indicated nonuniform differential item functioning for two of eight ADV items, with heterosexual youth being more likely to express strong levels of agreements with (a) female-perpetrated physical violence in response to male-perpetrated violence and (b) female-perpetrated violence against males broadly, relative to SMY. Although these differences were of negligible magnitude and only resulted in minimal differences in overall expected average scores, heterosexual youth were more likely to strongly accept female-perpetrated dating violence compared to SMY. Findings highlight differences in ADV item response patterns across heterosexual and sexual minority identifying youth and provide preliminary evidence for group differences in acceptance of female-perpetrated dating violence. Implications for prevention programming based on current findings include greater focus on measure adaptation and development as well as more consensus on the necessity of preventing female-perpetrated violence.

Examining Differential Item Functioning from a Multidimensional IRT Perspective.

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Age-Neutral Measurement Of Personality Functioning and Maladaptive Personality Traits.

As previous studies have shown that personality disorder (PD) assessment in older adults is often hampered because assessment tools are tailored toward younger adults, establishing the age-neutrality of novel tools is crucial. This study primarily aimed to evaluate the age-neutrality of the Level of Personality Functioning Brief Form (LPFS-BF 2.0) and the Personality Inventory for DSM-5 Modified + (PID-5-BF+M), using a sample of 254 community-dwelling adults. The analysis of Differential Item Functioning (DIF) demonstrated the age-neutrality of both instruments, with only 8.3% of LPFS-BF 2.0 items and 5.6% of PID-5-BF+M items exhibiting DIF. Differential Test Functioning (DTF) analyses revealed large DTF for the LPFS-BF 2.0 total score, indicating that age-specific norms might be necessary for this score. In summary, this study supports the use of these instruments in both older and younger adults, enhancing the assessment of PDs across the life span.

Validating a performance assessment of computational thinking for early childhood using item response theory

Despite growing interest in early childhood computational thinking (CT), there is a lack of validated assessments for children who are emerging readers. This paper presents validity and reliability evidence of a performance-based assessment of CT using item response theory (IRT) from 272 children aged 4–8. Using a two-parameter logistic model IRT model (2PL IRT), we confirmed that model- and item-level fits are acceptable. Item analyses revealed a high discriminability (M = 2.26, SD = 1.12) and a moderate item difficulty (M = −0.21; SD = 0.86), on average, across 19 items. Reliability analysis demonstrated that the assessment was substantially reliable (marginal reliability: rxx = 0.87). Differential item functioning (DIF) analyses indicated that the assessment estimated children's item parameters fairly, regardless of their gender and age. However, we confirmed gaps in latent ability (θ) of CT by gender and age: boys showed higher latent ability of CT than girls, and old children (above 72 months) showed higher latent ability than young children (below 72 months). Findings suggest the assessment is a fair measure that can serve as a reliable and valid tool to assess CT for children who are emerging readers.

Rasch Calibration and Differential Item Functioning (DIF) Analysis of the School Well-Being Scale for Students

Rasch Calibration and Differential Item Functioning (DIF) Analysis of the School Well-Being Scale for Students

Can personality disorder be accurately assessed in older age? A differential item functioning analysis of ICD-11 inventories.

The contested categorical personality disorder (PD) criteria are not well suited to inform PD diagnoses in older adults. Yet, the classification of PDs is undergoing a critical transition phase with a paradigm shift to a dimensional approach for diagnosing PDs. No special attention was given to the expression of PDs in older age when the dimensional ICD-11 model was developed. Given that PDs are highly prevalent in older adults, there is an urgent need to examine if ICD-11 related instruments are able to adequately assess for PDs in older adults. The age-neutrality of ICD-11 measures was examined in a sample of 208 Dutch community-dwelling adults (N=208, M age=54.96, SD=21.65), matched on sex into 104 younger (age range 18-64) and 104 older (age range 65-93) adults. An instrument is considered not to be age-neutral if a collective large level of differential item functioning (DIF) exists in a group of items of an instrument (i.e., 25% or more with DIF). We therefore set out to detect possible DIF in the following ICD-11 self-report measures: the Standardized Assessment of Severity of Personality Disorder (SASPD), the Personality Inventory for ICD-11 (PiCD), and the Borderline Pattern Scale (BPS). DIF analyses using a non-parametric odds ratio approach demonstrated that SASPD, PiCD, and BPS were age-neutral with less than 25% of items showing DIF. Yet, impact of DIF at scale level, examined by way of differential test functioning (DTF), indicated a DTF effect on the SASPD total score. These results of age-neutrality of the PiCD and BPS are promising for measuring ICD-11 traits and the borderline pattern. Yet, the age-neutral measurement of PD severity requires further research. With a rapidly aging population, its accurate assessment across the entire adult life span, including older age, is a prerequisite for an adequate detection of PDs.

DIF Analysis with Unknown Groups and Anchor Items

Ensuring fairness in instruments like survey questionnaires or educational tests is crucial. One way to address this is by a Differential Item Functioning (DIF) analysis, which examines if different subgroups respond differently to a particular item, controlling for their overall latent construct level. DIF analysis is typically conducted to assess measurement invariance at the item level. Traditional DIF analysis methods require knowing the comparison groups (reference and focal groups) and anchor items (a subset of DIF-free items). Such prior knowledge may not always be available, and psychometric methods have been proposed for DIF analysis when one piece of information is unknown. More specifically, when the comparison groups are unknown while anchor items are known, latent DIF analysis methods have been proposed that estimate the unknown groups by latent classes. When anchor items are unknown while comparison groups are known, methods have also been proposed, typically under a sparsity assumption – the number of DIF items is not too large. However, DIF analysis when both pieces of information are unknown has not received much attention. This paper proposes a general statistical framework under this setting. In the proposed framework, we model the unknown groups by latent classes and introduce item-specific DIF parameters to capture the DIF effects. Assuming the number of DIF items is relatively small, an L1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_1$$\\end{document}-regularised estimator is proposed to simultaneously identify the latent classes and the DIF items. A computationally efficient Expectation-Maximisation (EM) algorithm is developed to solve the non-smooth optimisation problem for the regularised estimator. The performance of the proposed method is evaluated by simulation studies and an application to item response data from a real-world educational test.

대학전공선택검사의 성차에 의한 차별기능문항 탐색

Objectives Paying attention to the phenomenon of a gender gap in the choice of a major in the engineering de-partment, this study set out to examine whether there was differential item functioning by gender and what caused it by analyzing the items related to engineering in the Korean College Major Interest Inventory, 2nd Edition(KCMII-2).&#x0D; Methods For this purpose, the study used the data of 9,069 respondents who answered the activity preference items and the data of 9,040 respondents who answered the vocational preference items in the engineering sec-tion of the inventory in its analysis before checking if the data met the normal distribution and unidimensionality assumption requirements. Differential item functioning was extracted with the ‘lordif’ package of R and the PARSCALE program. The items whose differential function was found both in the programs were chosen as the final differential item functioning.&#x0D; Results The results of testing the assumption of unidimensionality suggested the possibility of other dimensions of latent trait beyond activity and career preferences. Subsequent analysis conducted through differential item functioning analysis revealed that nine out of the 30 activity preference items and seven out of the 30 vocational preference items were extracted as differential item functioning. All these items except for one activity preference item were favorable to female students. The activity preference items whose differential function included majors in life science and biotechnology, computer software, environmental studies and energy, and chemistry and chemical engineering. In terms of career preference, the differential functioning items encompassed majors in life science and biotechnology, chemistry and chemical engineering, and environmental studies and energy.&#x0D; Conclusions The study then inferred the causes of the differential item function extracted based on the findings and discussed the future direction of career education to help high school students overcome the gender gap in the choice of an engineering career and make a reasonable and sustainable career and occupation choice.

Computational Thinking Skill Level of Senior High School Students Majoring in Natural Science

Computational thinking (CT) is a skill integrated into various curricula in many countries. However, lack of student acceptance and limited assessment become challenges to integrating skill into the curriculum, specifically in developing countries like Indonesia. Therefore, this study aimed to validate the Indonesian version of the Computational Thinking Scale (CTS) and determine the CT skill level of high school students majoring in science. This study was conducted using a quantitative approach with a cross-sectional survey design. Participants were purposively selected based on certain criteria from a population of high schools in Yogyakarta, Indonesia. In this study, data were collected from 526 students with 19 items of CTS questionnaires and analysed using the Rasch model measurement. The findings showed that the adapted CTS met the fit criteria based on Rasch model measurement, except for one item. Based on the logit mean value of +1.69, the level of students' CT ability falls into the good category, where the most frequently implemented aspect is the problem-solving aspect, while the least frequently implemented aspect is algorithmic thinking. According to Differential Item Function analysis, there were differences in student responses based on coding experience. This study is expected to contribute to the field of CT assessment in science education. In addition, the results of this study can be an affirmation for educational policy makers in developing countries to integrate CT into the curriculum of natural science majors.

Detection of gender bias in numeracy skills in Islamic boarding schools: Differential item functioning

Numeracy skills are an important point in the structure of mathematics, and boarding school students are no exception. This study aims to identify gender bias in numeracy assessment in Islamic boarding schools. This research is a descriptive exploratory research using quantitative methods. The instrument used in this study was a numeracy test of 25 questions consisting of matching, multiple-choice, complex multiple-choice, and description questions. The research subjects involved 383 students in West Java consisting of 4 pesantren in 4 cities, namely West Bandung Regency, Cirebon Regency, Tasikmalaya Regency, and Tasikmalaya City. Quantitative analysis used Item Response Theory (IRT) followed by Differential Item Functioning (DIF) analysis with the Mantel-Haenzel method. The results showed that the instrument was suitable for use because it met the standards of validity and reliability. The model fit test that meets is GPCM, and DIF analysis shows that there is 1 number, namely number 21, in numeracy questions that indicate gender bias. The results of this analysis indicate that the numeracy test instrument set is suitable for use by boarding school students with minimal gender bias.

The development and psychometric validation of a survey to measure the subjective well-being of care leavers

Young people who age out of state care are at risk of a range of negative outcomes. In England, national data provides only five indicators of care leavers’ lives and there are no measures of how young people themselves feel about their transition to adulthood. To fill this gap a new survey to measure subjective wellbeing was co-produced with 31 care leavers. The survey was then distributed by 21 local authorities and completed by 1804 care leavers. The responses revealed a steep decline in wellbeing after leaving care, a wide variation in care leavers’ wellbeing depending on the local authority responsible for their care, and that some groups, such as those with a disability, were more vulnerable to low wellbeing. The survey was also validated using psychometric analyses. Latent factors were extracted, dimensionality tested and differential item functioning (DIF) was used to see if different groups of care leavers responded similarly to questions. The association between the total survey score and the commonly used Office for National Statistics four personal wellbeing questions was examined. The survey had good reliability across each of the statistics but data loaded onto a five-factor solution rather than the theorised four. DIF analysis found differences by sex, ethnicity and disability. Overall, the survey was found to be a valid and reliable measure of care leavers’ subjective wellbeing providing practitioners with information on which aspects of life were going well and where practice and policy needed to change.

Investigating differential item functioning across interaction variables in listening comprehension assessment

Differential item functioning (DIF) analysis is essential to ensuring the equity of measurement for different subgroups at the item level and is an integral part of validity. However, existing DIF research often overlooks within-group heterogeneity, commonly assuming that test takers from different subgroups comprise a homogeneous population. This study investigated DIF across gender, academic background, and their interaction in listening comprehension assessment using Rasch measurement. It found that ignoring within-group heterogeneity would lead to the under-detection of DIF, likely due to the cancellation of DIF at broader group levels. In addition, the study is the first to investigate DIF in a linked test, a scenario more prevalent in practical testing. The findings of the study highlight the importance of accounting for within-group heterogeneity in test fairness investigations in language assessment research and point to the potential effect of test linking and equating on DIF analysis and interpretation.

Rasch analysis of the dermatology life quality index (DLQI) in patients with mild to moderate-severe psoriasis.

Background The Dermatology Life Quality Index (DLQI) is a valuable tool for assessing the quality of life in adult patients with psoriasis. Aims To analyse the reliability and validity of the DLQI to measure the quality of life in patients with mild to moderate-severe psoriasis. Methods This was a secondary validation study nested in a follow-up study. The Rasch-Andrich model was utilised to perform response function, item and person fit, differential item functioning, dimensionality, and reliability analyses. Results A total of 1439 patients were analysed, 52.1% male, mean age of 48.7 years (SD 16.1). Psoriasis vulgaris was the phenotype in 43.1% of patients, and 86% had a mild Psoriasis Area Severity Index (PASI). Adequate adjustment of the response function and the items was observed in the best-fit sample, except for item 7 (work and study). The measure explained 60.9% of the variance and presented a reliability of 0.86. Differential item functioning was identified by age, with a relevant bias in the estimation for older adults. Item-person maps are provided. Limitations This study was performed at a single centre, with most patients presenting a mild PASI score, limiting generalisation of the findings. Conclusion The validity evidence favours the use of the DLQI in moderate-severe psoriasis. However, the instrument biases the estimate of older adults. This population group should consider a specific version of the instrument.