Differential Test Functioning Research Articles

Age-Neutral Measurement Of Personality Functioning and Maladaptive Personality Traits.

As previous studies have shown that personality disorder (PD) assessment in older adults is often hampered because assessment tools are tailored toward younger adults, establishing the age-neutrality of novel tools is crucial. This study primarily aimed to evaluate the age-neutrality of the Level of Personality Functioning Brief Form (LPFS-BF 2.0) and the Personality Inventory for DSM-5 Modified + (PID-5-BF+M), using a sample of 254 community-dwelling adults. The analysis of Differential Item Functioning (DIF) demonstrated the age-neutrality of both instruments, with only 8.3% of LPFS-BF 2.0 items and 5.6% of PID-5-BF+M items exhibiting DIF. Differential Test Functioning (DTF) analyses revealed large DTF for the LPFS-BF 2.0 total score, indicating that age-specific norms might be necessary for this score. In summary, this study supports the use of these instruments in both older and younger adults, enhancing the assessment of PDs across the life span.

Can personality disorder be accurately assessed in older age? A differential item functioning analysis of ICD-11 inventories.

The contested categorical personality disorder (PD) criteria are not well suited to inform PD diagnoses in older adults. Yet, the classification of PDs is undergoing a critical transition phase with a paradigm shift to a dimensional approach for diagnosing PDs. No special attention was given to the expression of PDs in older age when the dimensional ICD-11 model was developed. Given that PDs are highly prevalent in older adults, there is an urgent need to examine if ICD-11 related instruments are able to adequately assess for PDs in older adults. The age-neutrality of ICD-11 measures was examined in a sample of 208 Dutch community-dwelling adults (N=208, M age=54.96, SD=21.65), matched on sex into 104 younger (age range 18-64) and 104 older (age range 65-93) adults. An instrument is considered not to be age-neutral if a collective large level of differential item functioning (DIF) exists in a group of items of an instrument (i.e., 25% or more with DIF). We therefore set out to detect possible DIF in the following ICD-11 self-report measures: the Standardized Assessment of Severity of Personality Disorder (SASPD), the Personality Inventory for ICD-11 (PiCD), and the Borderline Pattern Scale (BPS). DIF analyses using a non-parametric odds ratio approach demonstrated that SASPD, PiCD, and BPS were age-neutral with less than 25% of items showing DIF. Yet, impact of DIF at scale level, examined by way of differential test functioning (DTF), indicated a DTF effect on the SASPD total score. These results of age-neutrality of the PiCD and BPS are promising for measuring ICD-11 traits and the borderline pattern. Yet, the age-neutral measurement of PD severity requires further research. With a rapidly aging population, its accurate assessment across the entire adult life span, including older age, is a prerequisite for an adequate detection of PDs.

Assessing Measurement Invariance in ASWB Exams: Regulatory Research Proposal to Advance Equity

ABSTRACT Purpose Social workers from minoritized racial, ethnic, linguistic, and age groups are far less likely to pass licensing examinations required to practice. Using a simulated data set, our study investigates measurement equivalence, or invariance, of social work licensing exams. Materials For this analysis, we simulated responses to 15 multiple-choice questions which were scored as either correct or incorrect using the R mirt package and used mirt to fit a 2-parameter logistic model (2PL) to the response data. We generated the data so that five items could demonstrate DIF and calculated their impact on the test characteristic curves and item characteristic curves. Results Small amounts of differential item functioning added up into differential test functioning, but the effect size was small. This result is one potential outcome of an analysis of ASWB exams. Discussion Most studies evaluating test characteristic curves demonstrate small effect sizes. Measuring the test characteristic curve and the test information curve will help to investigate content-irrelevant sources of variance in the exams, including unfairness, unreliability, and invalid pass scores. Conclusion Differential test functioning is a core part of measurement invariance studies. Psychometric standards require test developers to assess measurement invariance at both the item-level and test-level to protect themselves from accusations of bias.

Does food use disorder exist? Item response theory analyses of a food use disorder adapted from the DSM-5 substance use disorder criteria in a treatment seeking clinical sample

BackgroundIncreased consumption of food that are high in energy and sugar have been pointed as a major factor in the obesity epidemic. Impaired control of food intake and the concept of food addiction has been developed as a potential contributor. Our objective was to evaluate the dimensionality and psychometric validity of diagnostic criteria for food addiction adapted from the 11 DSM-5 substance use disorder (SUD) criteria (i.e.: Food Use Disorder (FUD) criteria), and to evaluate the influence of age, gender, and body mass index (BMI). MethodsCross-sectional observational study including 508 participants (56.1% male; mean age 42.2) from outpatient treatment clinics for obesity or addiction disorders at time of admission. FUD diagnostic criteria were analyzed using confirmatory factor and 2-parameter item response theory analyses. Differential Item and Test Functioning analyses were performed across age, gender, and BMI. ResultsWe demonstrated the one-factor dimensionality of the criteria set. The criterion “craving” presented the strongest factor loading and discrimination parameter and the second-lowest difficulty. We found some significant uniform differential item functioning for body mass index. We found some differential test functioning for gender and BMI. ConclusionsThis study reports, for the first time, the validity of a potential Food Use Disorder (derived from the 11 DSM-5 SUD criteria adapted to food) in a sample of treatment seeking adults. This has great implications both at the clinical level and in terms of public health policy in the context of the global obesity epidemic.

Psychometric Properties of Controlled Oral Word Association (COWA) Test and Associations With Education and Bilingualism in American Indian Adults: The Strong Heart Study.

The Controlled Oral Word Association (COWA) test is used to assess phonemic fluency and executive function. Formal validation of test scores is important for accurate cognitive evaluation. However, there is a dearth of psychometric validation among American Indian adults. Given high burden of dementia risk and key contextual factors associated with cognitive assessments, this represents a critical oversight. In a large, longitudinal population-based cohort study of adult American Indians, we examined several validity inferences for COWA, including scoring, generalization, and extrapolation inferences, by investigation of factor structure, internal consistency, test-retest reliability, and differential test functioning. We found adequate unidimensional model fit, with high factor loadings. Internal consistency reliability and test-retest reliability were 0.88 and 0.77, respectively, for the full group. COWA scores were lowest among the oldest, lowest education, bilingual speakers; group effects for sex and bilingual status were small; age effect was medium; and education effect was largest. However, Wide Range Achievement Test (WRAT) score effect was stronger than education effect, suggesting better contextualization may be needed. These results support interpretation of total COWA score, including across sex, age, or language use strata.

A Simulation and Empirical Study of Differential Test Functioning (DTF)

Detecting and understanding DTF is very important under various DIF conditions. In this study, the performance of DTF, DRF, SIBTEST, and CSIBTEST approaches in detecting DIF effects was investigated using a simulation study and a real dataset. It was observed that different DIF conditions (uniform, non-uniform), the proportion of DIF items in the test, the DIF size, and the DIF effect (balanced, unbalanced) affected the performance of the methods, and especially in the case of the non-uniform DIF condition, the power rates of sDTF, sDRF, and SIBTEST statistics were low. In addition, according to the DTF estimations with the balanced/unbalanced DIF effect condition, in some cases, the effect of DIF on the overall test could be negligible. However, it was clearly emphasized in this study that DTF analyses should accompany DIF studies since DTF analysis may change with different DIF and test conditions.

Measuring Personality Systems in Older Adults: Age-Neutrality of the Dutch BIS/BAS Questionnaire.

With a rapidly aging population, age-neutrality of psychological instruments is becoming ever more important for accurate assessment of older adults. This study aims to test the age-neutrality of the Dutch Behavioral Inhibition System/Behavioral Approach System (BIS/BAS) scales by means of Differential Item Functioning (DIF) and Differential Test Functioning (DTF) analyses. DIF and DTF analyses were conducted using an odds ratio approach. Potential DIF was examined for the two primary scales and three BAS subscales in 390 Dutch-speaking participants divided over three age groups. Overall, the BIS-BAS scales lacked age-neutrality when comparing older adults to young adults: eight out of 20 items (ie, 40%) displayed DIF according to the adjusted Bonferroni corrected cut-offs, which exceeds the threshold of 25% for large DIF. Thus, for 40% of the test items, items were differently endorsed by young and older adults with the same position on the construct measured by the item. Therefore, the implications of the item-level DIF on the scale level were investigated across age groups. DTF analyses revealed large DTF for all BIS and BAS scales according to the adjusted Bonferroni corrected cut-offs. DIF found in items on the BIS scale and BAS-Drive and BAS-Fun Seeking scale can probably be explained by a difference in degree of expression across age groups. A solution could be the development of age-specific norms. DIF on BAS-Reward Responsiveness subscale could be partly due to measuring a different construct across age groups. Rephrasing or replacing such items with DIF may increase age-neutrality of the BIS/BAS Scales.

Regional differences in the assessment of depressive symptoms in the former German Democratic Republic and Federal Republic of Germany.

The present study investigated regional differences in response behaviour for the Patient Health Quetionnaire-9. We tested for measurement invariance and differential item and test functioning between formerly divided East- and West-Germany: the former German Democratic Republic and Federal Republic of Germany. Diverging socialization experiences in socialist versus capitalist and collectivist versus individualist systems may affect culturally sensitive assessments of mental health. To test this empirically, we used factor analytic and item-response-theoretic frameworks, differentiating between East- and West-Germans by birthplace and current residence based on several representative samples of the German general population (n=3802). Across all survey, we discovered slightly higher depression sum scores for East- versus West-Germans. The majority of items did not display differential item functioning-with a crucial exception in the assessment of self-harm tendencies. The scale scores were largely invariant exhibiting only small amounts of differential test functioning. Nonetheless, they made up on average about a quarter of the observed group differences in terms of effect magnitude. We explore possible causes and discuss explanations for the item-level differences. Overall, analyses of East- and West-German depressive symptom developments in the wake of reunification are feasible and statistically grounded.

Cross-Regional Validation of the Assessment of Positive Occupation 15: Differential Item and Test Functioning Analysis

Aims: This study aimed to validate the Assessment of Positive Occupation 15 (APO-15) by differential item functioning (DIF) and differential test functioning (DTF).

Assessing Differential Item Functioning and Differential Test Functioning in an Academic Motivation Scale using Item Response Theory methods

Social work researchers and practitioners who use measurement instruments to make data-informed decisions need to ensure those decisions are based on items and scales that are free from possible bias or undesirable differential functioning. In this study, we provide an example of how a set of Item Response Theory (IRT) statistical methods and tools can be used by social work measurement researchers to assess differential item (DIF) and scale (DTF) functioning. For the example, we explored the possible race, gender, and family composition differential functioning of a scale—the Academic Motivation Scale (AMS)—developed for use by school social workers. The data used in this analysis were collected from 3,221 seventh grade students in multiple school districts in a large urban mid-western U.S. county. We used IRT methods and a multiple-step framework to assess possible race, gender, and family composition DIF/DTF. Results indicated there was minimal race, gender, or family composition differential functioning at both the item and scale level. While the AMS is recommended for use by school social workers, further research is needed to examine possible DIF/DTF by other factors such as parent involvement and family background, gender identity, sexual preference, and cultural attributes and ethnic factors.

Item response theory and differential test functioning analysis of the HBSC-Symptom-Checklist across 46 countries

BackgroundThe Symptom Checklist (SCL) developed by the Health Behaviour in School-aged Children (HBSC) study is a non-clinical measure of psychosomatic complaints (e.g., headache and feeling low) that has been used in numerous studies. Several studies have investigated the psychometric characteristics of this scale; however, some psychometric properties remain unclear, among them especially a) dimensionality, b) adequacy of the Graded Response Model (GRM), and c) measurement invariance across countries.MethodsData from 229,906 adolescents aged 11, 13 and 15 from 46 countries that participated in the 2018 HBSC survey were analyzed. Adolescents were selected using representative sampling and surveyed by questionnaire in the classroom. Dimensionality was investigated using exploratory graph analysis. In addition, we investigated whether the GRM provided an adequate description of the data. Reliability over the latent variable continuum and differential test functioning across countries were also examined.ResultsExploratory graph analyses showed that SCL can be considered as one-dimensional in 16 countries. However, a comparison of the unidimensional with a post-hoc bifactor GRM showed that deviation from a hypothesized one-dimensional structure was negligible in most countries. Multigroup invariance analyses supported configural and metric invariance, but not scalar invariance across 32 countries. Alignment analysis showed non-invariance especially for the items irritability, feeling nervous/bad temper and feeling low.ConclusionHBSC-SCL appears to represent a consistent and reliable unidimensional instrument across most countries. This bodes well for population health analyses that rely on this scale as an early indicator of mental health status.

Analysis of Race and Sex Bias in the Autism Diagnostic Observation Schedule (ADOS-2)

There are long-standing disparities in the prevalence of autism spectrum disorder (ASD) across race and sex. Surprisingly, few studies have examined whether these disparities arise partially out of systematic biases in the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2), the reference standard measure of ASD. To examine differential item functioning (DIF) of ADOS-2 items across sex and race. This is a cross-sectional study of children who were evaluated for ASD between 2014 and 2020 at a specialty outpatient clinic located in the Mid-Atlantic region of the US. Data were analyzed from July 2021 to February 2022. Child race (Black/African American vs White) and sex (female vs male). Item-level biases across ADOS-2 harmonized algorithm items, including social affect (SA; 10 items) and repetitive/restricted behaviors (RRBs; 4 items), were evaluated across 3 modules. Measurement bias was identified by examining DIF and differential test functioning (DTF), within a graded response, item response theory framework. Statistical significance was determined by a likelihood ratio χ2 test, and a series of metrics was used to examine the magnitude of DIF and DTF. A total of 6269 children (mean [SD] age, 6.77 [3.27] years; 1619 Black/African American [25.9%], 3151 White [50.3%], and 4970 male [79.4%]), were included in this study. Overall, 16 of 140 ADOS-2 diagnostic items (11%) had a significant DIF. For race, 8 items had a significant DIF, 6 of which involved SA. No single item showed DIF consistently across all modules. Most items with DIF had greater difficulty and poorer discrimination in Black/African American children compared with White children. For sex, 5 items showed significant DIF. DIF was split across SA and RRB. However, hand mannerisms evidenced DIF across all 5 algorithms, with generally greater difficulty. The magnitude of DIF was only moderate to large for 2 items: hand mannerisms (among female children) and repetitive interests (among Black/African American children). The overall estimated effect of DIF on total DTF was not large. These findings suggest that the ADOS-2 does not have widespread systematic measurement bias across race or sex. However, the findings raise some concerns around underdetection that warrant further research.

Applying a Rasch model to the Broad Autism Phenotype Questionnaire: item severity analysis and differential test functioning of the English and Spanish versions

The broad autism phenotype (BAP) represents the elevated but nonclinical levels of autism-related expressions expanding them beyond the disorder threshold towards the general population and it is assessed worldwide using the Broad Autism Phenotype Questionnaire. In this study, we applied a Rasch item response approach to the Spanish and the English BAPQ versions. This approach allows us to test the proper functioning of each version, estimate which BAP behaviours are more likeable at different levels of severity, and study whether BAPQ item’s severities and orders could be considered equivalent between both versions. A Spanish community sample of 970 participants and an English of 533 ones completed either the Spanish or the English version of the BAPQ. The results revealed a lack of unidimensionality of the Pragmatic Language subscale in both tests and the Rigid subscale in the English version. Both Aloof subscale versions and the Spanish Rigid demonstrated adequate properties but with several items showing differential functioning in the case of Aloof. We conclude by highlighting the necessity of BAP measurements paralleling the current autism spectrum disorder structure, following the severity-dimensional conceptualization (behaviours are continuously distributed according to their severity), and suitable for being adapted to different languages.

A Differential Item Functioning (DIF) Study of the Infection Control Standard Isolation Precaution Instrument Across Gender and Major Among Healthcare Workers

Background: Measuring healthcare workers’ (HCWs) knowledge, attitudes, and practices (KAPs) regarding isolation precaution is essential for infection control which needs a valid and reliable instrument. Objectives: This study aimed to assess differential item functioning (DIF) across gender and major for the knowledge and practice items of the questionnaire, previously designed in Shiraz, Iran. Methods: This cross-sectional survey was conducted on 1070 participants (males/females: 306/764; medical students/nurses: 466/624). The study instrument had three subscales with nine questions for each KAP subscale. The Mantel-Haenszel (MH) statistic was used. The DIF and differential test functioning (DTF) analyses were also performed in this study. Results: There were very similar DIF outcomes for the knowledge and practice subscales, with one or two items indicating moderate DIF but comparable total scores across genders. Across majors, several items showed large DIF for both subscales. It was found that large DTF affects major for both subscales. Conclusions: Our findings indicated large DIF and DTF levels of the questionnaire among medical students and nursing groups. More attention should be paid when developing the items. This study shows the importance of paying attention to valid evidence for instruments developed within the field of healthcare.

A New Investigation of Fake Resistance of a Multidimensional Forced-Choice Measure: An Application of Differential Item/Test Functioning

To address faking issues associated with Likert-type personality measures, multidimensional forced-choice (MFC) measures have recently come to light as important components of personnel assessment systems. Despite various efforts to investigate the fake resistance of MFC measures, previous research has mainly focused on the scale mean differences between honest and faking conditions. Given the recent psychometric advancements in MFC measures (e.g., Brown & Maydeu-Olivares, 2011; Stark et al., 2005; Lee et al., 2019; Joo et al., 2019), there is a need to investigate the fake resistance of MFC measures through a new methodological lens. This research investigates the fake resistance of MFC measures through recently proposed differential item functioning (DIF) and differential test functioning (DTF) methodologies for MFC measures (Lee, Joo, & Stark, 2020). Overall, our results show that MFC measures are more fake resistant than Likert-type measures at the item and test levels. However, MFC measures may still be susceptible to faking if MFC measures include many mixed blocks consisting of positively and negatively keyed statements within a block. It may be necessary for future research to find an optimal strategy to design mixed blocks in the MFC measures to satisfy the goals of validity and scoring accuracy. Practical implications and limitations are discussed in the paper.

The multiple group IRT measurement invariance analysis of the Self-Compassion Scale in ten international samples

The purpose of this study was to examine the measurement invariance of the Self-Compassion Scale by IRT differential test functioning in ten distinct populations (n = 13623 participants) from ten different countries: Australia (n = 517), China (n = 321), Czech Republic (n = 5081), Germany (n = 2510), Italy (n = 384), Portugal (n = 512), Slovakia (n = 1181), South Korea (n = 1813), Turkey (n = 471), and USA (n = 833). We assessed differential test functioning for the two SCS subscales, Self-compassionate responding and Self-uncompassionate responding separately, because previous bifactor and two-tier analyses of the scale showed the best fit with two separate general factors, and not for the overall score. Only 13 of the 45 comparisons for Self-compassionate responding and 13 of the 45 comparisons for Self-uncompassionate responding (analyses of every pair) demonstrated measurement invariance (no differential test functioning). Generally, our results revealed that the two subscales of Self-compassionate responding and Self-uncompassionate responding were not equivalent among all countries and groups. Therefore, it is impossible to compare overall scores across all countries. Two subscales of the Self-Compassion Scale (Self-compassionate responding and Self-uncompassionate responding) are valid and reliable instruments with substantial potential of use cross-culturally, but results reveal significant cross-cultural differences in the way these two constructs are measured by the subscales of the SCS. Future analyses of the meanings and connotations of this construct across the world are necessary to develop a scale which allows cross-cultural comparisons of various treatment outcomes related to self-compassion.

Validation of the Four-Dimensional Symptom Questionnaire (4DSQ) in a mental health setting

ABSTRACT Mental health problems are highly prevalent in primary care. Validated tools to detect mental disorders in general practice are needed. The Four-Dimensional Symptom Questionnaire (4DSQ) was designed to help GPs differentiating between psychological distress and psychopathological conditions (depression, anxiety, somatization). The aim of the current study was to examine psychometric properties of the 4DSQ in a mental health setting. Reliability, factorial, construct, and criterion validity of the English translation of the 4DSQ were analyzed in an American sample of 159 patients attending a psychotherapy outpatient clinic. Measurement equivalence across languages was determined by analyzing differential item functioning (DIF) and differential test functioning (DTF) in the American sample and a Dutch mental health sample, matched by age and sex. A confirmatory factor analysis confirmed all 4DSQ subscales to be unidimensional. All 4DSQ subscales revealed excellent reliability (Cronbach’s alpha and McDonald omega ≥.90) and high correlations with a symptom distress subscale of an instrument that is commonly used to monitor psychotherapy progress, the Outcome Questionnaire-45. Eight items were flagged with DIF. The Depression subscale was free of DIF. DTF analyses showed an impact of DIF on scale level for the lower cutoff score of the Distress scale. The 4DSQ Distress score was the best predictor of a mood disorder diagnosis and the Anxiety score outperformed other 4DSQ scales to predict an anxiety disorder. In conclusion, the 4DSQ demonstrates excellent reliability and validity in a mental health setting. Further research is needed to determine reliable cutoff values on 4DSQ subscales to predict psychiatric diagnoses.

Testing for Differential Item Functioning Under the D-Scoring Method.

This study offers an approach to testing for differential item functioning (DIF) in a recently developed measurement framework, referred to as D-scoring method (DSM). Under the proposed approach, called P-Z method of testing for DIF, the item response functions of two groups (reference and focal) are compared by transforming their probabilities of correct item response, estimated under the DSM, into Z-scale normal deviates. Using the liner relationship between such Z-deviates, the testing for DIF is reduced to testing two basic statistical hypotheses about equal variances and equal means of the Z-deviates for the reference and focal groups. The results from a simulation study support the efficiency (low Type error and high power) of the proposed P-Z method. Furthermore, it is shown that the P-Z method is directly applicable in testing for differential test functioning. Recommendations for practical use and future research, including possible applications of the P-Z method in IRT context, are also provided.

Addressing gender discrimination in cognitive assessment using the English Comprehension Test

Orientation: The empirically designed English Comprehension Test (ECT) is theorised to measure verbal reasoning and is currently undergoing validation. The test development produced two versions of the ECT, namely, ECT version 1.2 and ECT version 1.3. This study focuses on the latest test version, ECT version 1.3. Research purpose: The purpose of this study was to statistically explore the performance of men and women who were assessed by the empirically designed ECT. Motivation for the study: Cognitive assessment has often been used as a discriminatory tool against gender, race and/or languages. The discrimination against race and gender were the consequences of a patriarchal system and Apartheid in South Africa, as black men and women were deemed to be subordinate to white men. With the demise of Apartheid, measures have been put in place to guard against unfair assessment practices. In addition, legislation was developed to ensure that test developers and test users employed assessments that did not unfairly prejudice individuals based on their race, gender and language. These measures are imperative to ensure fairness and equal opportunities for men and women across race and language groups. Research design, approach and method: This study used a quantitative cross-sectional design. The ECT was administered to a non-probability convenience sample of 881 individuals. The data were analysed by differential test functioning (DTF) in Winsteps and analysis of variance (ANOVA) in the Statistical Product and Service Solutions (SPSS) package. Main findings: The results indicated that the majority of the test items did not present any bias, but five possibly biased items were identified across gender groups in the test. These five items that were possibly biased appear to be affected by language and not gendered knowledge, and this, however, necessitates further investigation. The ANOVA results only indicated statistically significant differences across the different language groups, thereby confirming the DTF results. Practical/managerial implications: A major limitation of this study is the restriction of range and lack of generalisability. Contribution/value-add: This study promotes the use of DTF and ANOVA as a means of ensuring fairness in assessment practices across gender groups. Moreover, it contributes to cross-cultural test development and validation research in South Africa.

Examining provision and sufficiency of testing accommodations for English learners

Accommodations are commonly provided to address language-related barriers students may experience during testing. Research on the validity of scores from accommodated test administrations remains somewhat inconclusive. The current study investigated item response patterns to understand whether accommodations, as used in practice among English learners (ELs) in the United States, allow for comparable measurement between ELs and non-ELs. Results indicated that although significant differences are evident in overall test scores for ELs and non-ELs, only minimal measurement concerns were evident. Very few items displayed moderate or large differential item functioning (DIF); no tests showed small, medium, or large differential test functioning. The current study adds to existing literature on measurement comparability and accommodation research on ELs; implications for practice are provided.