Differential Test Functioning Research Articles

Age neutrality of the Young Schema Questionnaire in patients with a substance use disorder

Young's Schema Focused Therapy (SFT) is gaining popularity in the treatment of older adults. In the context of this therapy, the Young Schema Questionnaire (YSQ) was developed to assess the early maladaptive schemas (EMS). EMS are considered to be relatively stable over time, but research shows that questionnaires often lack face validity in older adults, which makes it difficult to investigate EMS in older adults and their stability across the lifespan. In the present cross-sectional study, we investigated the age neutrality of the Young Schema Questionnaire--Long Form in young (aged 18-34 years), middle-aged (aged 35-59 years), and older (aged 60-75 years) adults in a clinical sample of substance use disorders (N = 321) by examining potential differential item functioning (DIF). While investigating the stability of the schemas, we controlled for substance dependency and clinical symptoms by means of, respectively, the Drug Use Screening Inventory - Revised and the Symptom Checklist-90-R. The Bonferroni-adjusted Liu-Agresti Cumulative Common Log-Odds Ratio confirmed large DIF for six items, divided across five schema scales (Mistrust/Abuse, Subjugation, Entitlement, Enmeshment and Self-sacrifice). Of the six items that presented DIF, only one item showed differential test functioning (Entitlement). Overall results show only 3% DIF, implying age neutrality of the questionnaire. Current results corroborate that most EMS scales are equally measured across age, and reliable comparisons can be made across the lifespan, allowing for good clinical practice and further research on SFT in older adults. Only for Entitlement, Enmeshment, and Insufficient Self-control, caution is needed when comparing mean scores across the age groups.

Gender-Related Differential Item Functioning in Vocational Interest Measurement

Large mean differences are consistently found in the vocational interests of men and women. These differences may be attributable to real differences in the underlying traits. However, they may also depend on the properties of the instrument being used. It is conceivable that, in addition to the intended dimension, items assess a second dimension that differentially influences responses by men and women. This question is addressed in the present study by analyzing a widely used German interest inventory (Allgemeiner Interessen-Struktur-Test, AIST-R) regarding differential item functioning (DIF) using a DIF estimate in the framework of item response theory. Furthermore, the impact of DIF at the scale level is investigated using differential test functioning (DTF) analyses. Several items on the AIST-R’s scales showed significant DIF, especially on the Realistic, Social, and Enterprising scales. Removal of DIF items reduced gender differences on the Realistic scale, though gender differences on the Investigative, Artistic, and Social scales remained practically unchanged. Thus, responses to some AIST-R items appear to be influenced by a secondary dimension apart from the interest domain the items were intended to measure.

Age-Neutrality of the Trait Facets Proposed for Personality Disorders in DSM-5: A DIFAS Analysis of the PID-5

An age-neutral measurement system is one of the basic conditions for an accurate personality assessment across the lifespan, both longitudinally and cross-sectionally. In this study the age-neutrality of the Personality Inventory for DSM-5 PID-5; Krueger et al. (Psychological Medicine 42, 1879–1890, 2012) was investigated. Potential Differential Item Functioning (DIF) was examined for the 25 trait facets in older versus younger adults. Overall, 33 items displayed large DIF, according to the adjusted Bonferroni corrected cutoffs (Mantel Chi-square, Liu-Agresti Cumulative Common Log-Odds Ratio [L-A LOR], and Cox’s Noncentrality Parameter Estimator [Cox’s B]). In a next step, the implications of the item level DIF across age groups was investigated on scale (i.e., facet) level. These Differential Test Functioning (DTF) analyses revealed large DTF for four of the 25 PID-5 facets (i.e., Withdrawal, Attention Seeking, Rigid Perfectionism and Unusual Beliefs). Current initial results show that most PID-5 traits are measured equally well across age, however, further research is needed to further refine this instrument and make it entirely age-neutral.

Measuring differential item and test functioning across academic disciplines

Differential item functioning (DIF) is when a test item favors or hinders a characteristic exhibited by group members of a test-taking population. DIF analyses are statistical procedures used to determine to what extent the content of an item affects the item endorsement of sub-groups of test-takers. If DIF is found for many items on the test, the final test scores do not represent the same measurement across groups in the population of test-takers. This is known as differential test functioning (DTF). DTF is of particular concern in tertiary level language tests, where test-takers often differ in academic discipline. This study examined the DIF and DTF of an in-house developed assessment designed to measure how well first year students of five academic disciplines achieved material over the course of a year of English language study. The DIF and DTF tests were performed using Rasch analysis, which controls for ability across groups, ensuring that items are only flagged if groups of test-takers of the same ability levels exhibit a significantly different probability of endorsing the item. The current analysis outlines the process for checking for DIF and DTF and finds that even though DTF is unlikely, there were several items that favored and hindered some majors. Recommendations for modification of items are made and the importance of establishing a process to check for DTF and DIF, especially when the test-takers are from different disciplines of study, is discussed.

Methodologies for Investigating Item- and Test-Level Measurement Equivalence in International Large-Scale Assessments

In this study, the Canadian English and French versions of the Problem-Solving Measure of the Programme for International Student Assessment 2003 were examined to investigate their degree of measurement comparability at the item- and test-levels. Three methods of differential item functioning (DIF) were compared: parametric and nonparametric item response theory and ordinal logistic regression. Corresponding derivations of these three DIF methods were investigated at the test-level to examine both differential test functioning (DTF) and the correspondence between findings at the item-level with those at the test-level. Item-level findings suggested consistency in DIF detection across methods; however, differences in effect sizes of DIF were found by each method. Test-level results revealed a high degree of consistency across DTF methods. Discrepancies were found between item- and test-level comparability analyses. Item-level analyses suggested moderate to low degrees of comparability, whereas test-level findings suggested a higher degree of comparability. Findings also indicated the direction of DIF was mixed as some DIF items favored English-speaking students and others favored French-speaking students, suggesting that DIF cancellation may explain why item-level incomparability was not detected at the test-level.

Mayo-Portland Adaptability Inventory: Comparing Psychometrics in Cerebrovascular Accident to Traumatic Brain Injury

Malec JF, Kean J, Altman IM, Swick S. Mayo-Portland Adaptability Inventory: comparing psychometrics in cerebrovascular accident to traumatic brain injury. Objectives(1) To evaluate the measurement reliability and construct validity of the Mayo-Portland Adaptability Inventory, 4th revision (MPAI-4) in a sample consisting exclusively of patients with cerebrovascular accident (CVA) using single parameter (Rasch) item-response methods; (2) to examine the differential item functioning (DIF) by sex within the CVA population; and (3) to examine DIF and differential test functioning (DTF) across traumatic brain injury (TBI) and CVA samples. DesignRetrospective psychometric analysis of rating scale data. SettingHome- and community-based brain injury rehabilitation program. ParticipantsIndividuals post-CVA (n=861) and individuals with TBI (n=603). InterventionsNot applicable. Main Outcome MeasureMPAI-4. ResultsItem data on admission to community-based rehabilitation were submitted to Rasch, DIF, and DTF analyses. The final calibration in the CVA sample revealed satisfactory reliability/separation for persons (.91/3.16) and items (1.00/23.64). DIF showed that items for pain, anger, audition, and memory were associated with higher levels of disability for CVA than TBI patients; whereas, self-care, mobility, and use of hands indicated greater overall disability for TBI patients. DTF analyses showed a high degree of association between the 2 sets of items (R=.92; R2=.85) and, at most, a 3.7 point difference in raw scores. ConclusionsThe MPAI-4 demonstrates satisfactory psychometric properties for use with individuals with CVA applying for interdisciplinary posthospital rehabilitation. DIF reveals clinically meaningful differences between CVA and TBI groups that should be considered in results at the item and subscale level.

Age-neutrality of the NEO-PI-R: Potential Differential Item Functioning in Older Versus Younger Adults

Geriatric researchers and clinicians often have to deal with a lack of valid personality measures for older age groups (e.g., Mroczek, Hurt, & Berman, 1999; Zweig 2008), which hampers a reliable assessment of personality in later life. An age-neutral measurement system is one of the basic conditions for an accurate personality assessment across the lifespan, both longitudinally and cross-sectionally. In the present study, we empirically investigate the age-neutrality of one of the most widely used personality measures (i.e., the NEO PI-R (Costa & McCrae, 1992)), by examining potential Differential Item Functioning (DIF). Overall, results indicate that the vast majority (92.9 % at domain-level and 95 % at facet-level) of the NEO PI-R items was similarly endorsed by younger and older age groups with the same position on the personality trait of interest, corroborating the NEO PI-R’s age neutrality. However, Differential Test Functioning (DTF) analyses revealed large DTF for Extraversion, and facet A6 (Tender-Mindedness). Results are discussed in terms of their implications for using the current format of the NEO PI-R in older aged samples.

Cross-regional validity of the Assessment of Motor and Process Skills for use in Middle Europe

To evaluate cross-regional validity of the Assessment of Motor and Process Skills (AMPS) with a specific focus on valid use with Middle Europeans. Descriptive cross-regional validation study. A total of 1346 participants from Middle Europe and 144,143 participants from North America, UK/Ireland, the Nordic Countries, other Europe, Australia/New Zealand and Asia, between the ages of 3 and 103 years, in good health and with a variety of diagnoses, were selected from the AMPS database. Many-facet Rasch analysis was used to analyse participant raw data, and effect sizes were used to evaluate for differential item functioning. Evaluation for differential test functioning was also implemented. None of the 20 activity of daily living process items, and only one of the activity of daily living motor items demonstrated differential item functioning. The activity of daily living motor item Aligns exceeded the significant effect size criterion of ± 0.55 logit, but the significant differential item functioning did not lead to differential test functioning (i.e. all measures fell within the 95% confidence bands). This study provides further evidence of validity of the AMPS when used to evaluate quality of daily living performance across world regions. The AMPS measures can be used as objective indices of activity of daily living ability in rehabilitation settings and in international collaborative research related to activity of daily living task performance.

Do Applicants and Incumbents Respond to Personality Items Similarly? A Comparison of Dominance and Ideal Point Response Models

This study used an ideal point response model to examine the extent to which applicants and incumbents differ when responding to personality items. It was hypothesized that applicants' responses would exhibit less folding at high trait levels than incumbents' responses. We used sample data from applicants (N=1,509) and incumbents (N=1,568) who completed the 16 Personality Questionnaire Select. Differential item (DIF) and test functioning (DTF) analyses were conducted using the generalized graded unfolding model, which is based on ideal point model assumptions. Out of the 90 items, 50 showed DIF; however, only 11 were in the hypothesized direction. DTF was significant for 3 of the 12 scales; 2 were in the hypothesized direction.

Differential item functioning of the UWES-17 in South Africa

Orientation: South Africa’s unique cultural diversity provides a constant challenge about the fair and unbiased use of psychological measures in respect of their cross-cultural application.Research purpose: This study assesses the Differential Item Functioning (DIF) of the Utrecht Work Engagement Scale (UWES-17) for different South African cultural groups in a South African company.Motivation for the study: Organisations are using the UWES-17 more and more in South Africa to assess work engagement. Therefore, research evidence from psychologists or assessment practitioners on its DIF across different cultural groups is necessary.Research design, approach and method: The researchers conducted a Secondary Data Analysis (SDA) on the UWES-17 sample (n = 2429) that they obtained from a cross-sectional survey undertaken in a South African Information and Communication Technology (ICT) sector company (n = 24 134). Quantitative item data on the UWES-17 scale enabled the authors to address the research question.Main findings: The researchers found uniform and/or non-uniform DIF on five of the vigour items, four of the dedication items and two of the absorption items. This also showed possible Differential Test Functioning (DTF) on the vigour and dedication dimensions.Practical/managerial implications: Based on the DIF, the researchers suggested that organisations should not use the UWES-17 comparatively for different cultural groups or employment decisions in South Africa.Contribution/value add: The study provides evidence on DIF and possible DTF for the UWES-17. However, it also raises questions about possible interaction effects that need further investigation.

Cross-Regional Validation of the School Version of the Assessment of Motor and Process Skills

The objective was to determine whether the School Version of the Assessment of Motor and Process Skills (School AMPS) is valid when used to evaluate students in different world regions. Participants were 984 students, ages 3-13 yr, from North America, Australia and New Zealand, United Kingdom, and the Nordic countries, matched for age and diagnoses. We used FACETS many-faceted Rasch analyses to generate item difficulty calibrations by region and evaluate for significant differential item functioning (DIF) and differential test functioning (DTF). Four School AMPS items demonstrated DIF but resulted in no DTF. This study provided support for occupational therapists using the School AMPS to evaluate students' quality of schoolwork task performances across regions because the School AMPS measures are free of bias associated with world region.

Differential Item Functioning of Pathological Gambling Criteria: An Examination of Gender, Race/Ethnicity, and Age

This study tested for the presence of differential item functioning (DIF) in DSM-IV Pathological Gambling Disorder (PGD) criteria based on gender, race/ethnicity and age. Using a nationally representative sample of adults from the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC), indicating current gambling (n=10,899), Multiple Indicator-Multiple Cause (MIMIC) models tested for DIF, controlling for income, education, and marital status. Compared to the reference groups (i.e., Male, Caucasian, and ages 25-59years), women (OR=0.62; P<.001) and Asian Americans (OR=0.33; P<.001) were less likely to endorse preoccupation (Criterion 1). Women were more likely to endorse gambling to escape (Criterion 5) (OR=2.22; P<.001) but young adults (OR=0.62; P<.05) were less likely to endorse it. African Americans (OR=2.50; P<.001) and Hispanics were more likely to endorse trying to cut back (Criterion 3) (OR=2.01; P<.01). African Americans were more likely to endorse the suffering losses (OR=2.27; P<.01) criterion. Young adults were more likely to endorse chasing losses (Criterion 9) (OR=1.81; P<.01) while older adults were less likely to endorse this criterion (OR=0.76; P<.05). Further research is needed to identify factors contributing to DIF, address criteria level bias, and examine differential test functioning.

Adaptation and Analysis of Motivated Strategies for Learning Questionnaire in the Chinese Setting

This article reports the adaptation and analysis of Pintrich's Motivated Strategies for Learning Questionnaire (MSLQ) in Hong Kong. First, this study examined the psychometric qualities of the existing Chinese version of MSLQ (MSLQ-CV). Based on this examination, this study developed a revised Chinese version of MSLQ (MSLQ-RCV) for junior secondary students in Hong Kong. Confirmatory factor analysis and Graded Response Model (GRM) were used to examine the internal consistency, construct validity, differential item functioning (DIF), and differential test functioning (DTF) of the revised instrument, which contains six factors and 50 items. Compared with the existing MSLQ-CV, the factor structure of MSLQ-RCV was expanded to incorporate two new factors: extrinsic value and peer learning. The reasons for these amendments and implications of the findings are discussed.

An Introduction to Using Multidimensional Item Response Theory to Assess Latent Factor Structures

This study provides an introduction to the use of multidimensional item response theory (MIRT) analysis for assessing latent factor structure, and compares this statistical technique to confirmatory factor analysis (CFA) in the evaluation of an original measure developed to assess students’ motivations for entering a social work community of practice. The Participation in a Social Work Community of Practice Scale (PSWCoP) was administered to 506 masters of social work students from 11 accredited graduate programs. The psychometric properties and latent factor structure of the scale are evaluated using MIRT and CFA techniques. Although designed as a 3-factor measure, analysis of model fit using both CFA and MIRT do not support this solution. Instead, analyses using both methods produce convergent results supporting a 4-factor solution. Discussion includes methodological implications for social work research, focusing on the extension of MIRT analysis to assessment of measurement invariance in differential item functioning, differential test functioning, and differential factor functioning.

Differential Item Functioning on Antisocial Behavior Scale Items for Adolescents and Young Adults from Single-Parent and Two-Parent Families

We investigated measurement equivalence in two antisocial behavior scales (i.e., one scale for adolescents and a second scale for young adults) by examining differential item functioning (DIF) for respondents from single-parent (n = 109) and two-parent families (n = 447). Even though one item in the scale for adolescents and two items in the scale for young adults showed significant DIF, the two scales exhibited non-significant differential test functioning (DTF). Both uniform and nonuniform DIF were investigated and examples of each type were identified. Specifically, uniform DIF was exhibited in the adolescent scale whereas nonuniform DIF was shown in the young adult scale. Implications of DIF results for assessment of antisocial behavior, along with strengths and limitations of the study, are discussed.

Are there gender differences in work ethic? An examination of the measurement equivalence of the multidimensional work ethic profile

Previous research has indicated that males and females differ on their reported levels of work ethic. However, previous studies have relied upon work ethic inventories with limited generalizability, and no study has evaluated the invariance of measures. This study examined measurement invariance by exploring the differential item and test functioning of one work ethic inventory, the multidimensional work ethic profile (MWEP; Miller, Woehr, & Hudspeth, 2002) for male and female respondents. Results did not indicate that the MWEP functioned differently by gender at the test or item level. Hence, work ethic as measured by the MWEP does not carry different socially constructed meanings for men versus women. We also conclude that in contrast with previous research, women do not have a higher level of work ethic than men.

Application of item response theory to achieve cross‐cultural comparability of occupational stress measurement

Our objective was to examine cross-cultural comparability of standard scales of the Effort-Reward Imbalance occupational stress scales by item response theory (IRT) analyses. Data were from 20,256 Japanese employees, 1464 Dutch nurses and nurses' aides, 2128 representative employees from post-communist countries, 963 Swedish representative employees, 421 Chinese female employees, 10,175 employees of the French national gas and electric company and 734 Spanish railroad employees, sanitary personnel and telephone operators. The IRT likelihood ratio model was used for differential item functioning (DIF) and differential test functioning (DTF) analyses. Despite the existence of DIF, most comparisons did not show discernible differences in the relations between Effort-Reward total score and level of the underlying trait across cultural groups. In the case that DTF was suspected, excluding an item with significant DIF improved the comparability. The full cross-cultural comparability of Effort-Reward Imbalance scores can be achieved with the help of IRT analysis.

Comparing the Performance of Native North Americans and Predominantly White Military Recruits on Verbal and Nonverbal Measures of Cognitive Ability

This study compared the cognitive ability of adult Canadian First Nations (American Indian) aboriginals (N=101) living in remote areas to recruits (N=131) undergoing military training in the Canadian Forces. Comparisons involved both verbal and nonverbal measures: The Canadian Forces Aptitude Test (CFAT), Wonderlic Personnel Test, (WPT), Raven's Standard Progressive Matrices (SPM) and Mill Hill Vocabulary (MHV) test. All measures were examined for differential test and item functioning; as well, the CFAT was analyzed for adverse impact. Confirming past research, First Nations (FNs) members scored, on average, 18 points lower in IQ estimates based on the WPT, a verbal measure of cognitive ability; however, the differences between the groups were less on the nonverbal tests: SPM (5 points lower) and the MHV (9 points lower). Differential Item Functioning (DIF) analysis detected a few items from the CFAT, SPM and MHV that displayed DIF, but none from the WPT. The SPM and WPT appear to be unbiased measures with respect to differential test functioning (DTF), once language and education are controlled. Making employment decisions solely on verbal cognitive ability test scores, however, is likely to produce adverse impact against members of FN, unless verbal ability is a bona fide occupational requirement for the positions under consideration. Both the verbal and nonverbal cognitive ability tests assessed the same latent structure suggesting that the nonverbal tests could be used in place of the verbal tests with little loss in predictive ability for occupations involving a high degree of spatial ability.

A Monte Carlo Examination of the Sensitivity of the Differential Functioning of Items and Tests Framework for Tests of Measurement Invariance With Likert Data

This article highlights issues associated with the use of the differential functioning of items and tests (DFIT) methodology for assessing measurement invariance (or differential functioning) with Likert-type data. Monte Carlo analyses indicate relatively low sensitivity of the DFIT methodology for identifying differential item functioning (DIF) under some conditions of differential functioning with previously recommended significance values. The differential test functioning index was extremely insensitive to differential functioning under all study conditions. The authors recommend alternative noncompensatory DIF cutoff values used to evaluate the significance of DIF for different DIF effect sizes. Additionally, contrasts between polytomous and dichotomous data are drawn, and problems with determining measurement invariance at the scale, rather than item, level for Likert scale data are highlighted.

Assessing Gender-Related Differential Item Functioning and Predictive Validity With the Institutional Integration Scale

This study examined the gender-related differential predictive validity of five subscales of the Institutional Integration Scale (IIS) with regard to college student withdrawal. Differential functioning of the IIS across genders was assessed using an item response theory (IRT)—based framework of differential item and test functioning. The results confirmed the absence of differential functioning and supported the predictive validity of two of the five subscales for student withdrawal. IRT analyses revealed that a number of the items did not adequately reflect the construct and should be revised or removed from the measure. A discussion of these results and the implications for higher education institutions focused on preventing student withdrawal are presented.