Measurement Invariance Testing Research Articles

The Short Dark Tetrad (SD4)

Abstract: This study aimed to test the construct and criterion validity of the Serbian adaptation of the Short Dark Tetrad (SD4). In addition to testing measurement invariance between the Serbian ( N = 488) and Canadian samples ( N = 739), the construct validity of the SD4 was also assessed through correlations with more extensive measures of the Dark Tetrad, and criterion validity was evaluated through correlations with variables related to mental health. The results indicated good model fit indices for the SD4 in both samples and partial scalar invariance across samples. Validity correlations with extensive measures confirmed the construct validity of all SD4 scales, with caution noted for the psychopathy scale, which shares similar content with sadism. Profile similarity, based on construct and criterion correlations, revealed substantial dissimilarity between narcissism and the other scales but high similarity among the Machiavellianism, psychopathy, and sadism scales. Regardless of the similarity between the scales, they showed distinctive correlations with emotional distress and positive mental health aspects.

“I Know That I Don’t Know Everything”: Validation of the General Intellectual Humility Scale in French

Intellectual humility involves recognizing one’s own epistemic limitations and is associated with several beneficial outcomes. This work presents a French version of the General Intellectual Humility Scale (Leary et al., 2017) to investigate this concept in French-speaking populations. We translated the scale and, across five studies (NTotal = 2172), provide evidence of its structure (through EFA, CFA, internal consistency, test-retest reliability, and measurement invariance across French and English speakers) and the validity of its scores (convergent, divergent, and predictive). Study 1 showed that the scale related to need for evidence. Study 2 revealed that the scale (a) positively related to openness and need for cognition, (b) negatively related to dogmatism, and (c) had a negligible relationship with social desirability. In Study 3, scores predicted attention paid to the quality of persuasive arguments over and above need for closure. Study 4 showed that the scale correlated positively with another self-report measure of intellectual humility. A test-retest analysis also indicated adequate reliability for the French version of the scale. Finally, Study 5 tested measurement invariance and revealed that the structure of the scale was similar between French and English speakers and that their latent scores were comparable. Overall, this work offers evidence of the (French) General Intellectual Humility Scale as a valid measure of intellectual humility.

Comparability of personality facets between men and women: A test of measurement invariance in IPIP-NEO facets in 49 countries

Comparability of personality facets between men and women: A test of measurement invariance in IPIP-NEO facets in 49 countries

Testing Measurement Invariance using Multi-group Latent Class Analysis: A Cross-National Study of ICT Self-Efficacy

Testing Measurement Invariance using Multi-group Latent Class Analysis: A Cross-National Study of ICT Self-Efficacy

Exercise Addiction Inventory-3 (EAI-3): Psychometric properties of the Hungarian version

Background and aims: The Exercise Addiction Inventory (EAI), based on the components model addictions, is a 6-item instrument used to assess the risk of exercise addiction (REA). Its revised version (EAI-R) was published in 2019 but only differed from the original scale in the response rating range (using a 6-point rather than 5-point Likert scale). In 2023, the EAI-3 was released with two new items (guilt when missing training and exercising despite injury). We aimed to test the validity and reliability of the Hungarian EAI-3 (EAI-3-HU). Methods: We tested 507 regular exercisers (Mage = 38.7 years, SD age = 10.63 years, range age: 18–78 years; 62.7% females) who completed the EAI-3-HU, the obsessive passion subscale of the Passion Scale, and exercise habits questions on the online Qualtrics research platform during autumn/winter 2023–2024. Results: Confirmatory factor analysis resulted in a good model fit for the two factor EAI-3-HU (CFI = .96; TLI = .94; RMSEA = .07; SRMR = .04). However, the covariance between the latent factors was .97, indicating that they measure an identical concept. Thus, a single-factor solution was appropriate (CFI = .96; TLI = .94; RMSEA = .07; SRMR = .04). Testing measurement invariance revealed the partial scalar invariance across genders. The internal reliability of the scale was good (Cronbach’s α = .81). The scale had good convergent validity with obsessive passion (r = .72), and discriminant validity based on exercise frequency as well as exercise intensity (p < .001). Conclusion: The 8-item single factor EAI-3-HU adequately assesses the Hungarian samples’ REA. Nevertheless, it should be kept in perspective that the revised tool, like its predecessors, only assesses a level of ‘risk’, which does not imply morbidity, thus it has no clinical diagnostic value.

The structured assessment of violence risk in youth demonstrates measurement invariance between Black and White justice-referred youths.

Identification and implementation of effective methods for reducing racial/ethnic bias and disparities in legal settings are paramount in the United States and other countries. One procedure originally thought to reduce bias in legal decisions is the use of risk assessment instruments, which is now being heavily scrutinized. Measurement invariance, a latent trait technique, is a robust method for assessing one form of bias. Measurement invariance involves determining whether risk items in an instrument appear to be functioning the same between racial or other groups. Thus, the present study examined measurement invariance of the Structured Assessment of Violence Risk in Youth (SAVRY) between non-Latino Black and White youths to examine racial bias. We expected the SAVRY to be invariant (lacking measurement bias) between Black and White youths. The sample included 687 Black and 361 White youths, and the study used a large, multistate data set of SAVRYs conducted by probation officers. We conducted measurement invariance testing in a series of hierarchical steps including testing configural and scalar invariance. The SAVRY demonstrated scalar invariance (equal thresholds for ratings from "low" to "moderate" and "moderate" to "high") for all items except one-community disorganization. The findings lend further credibility to the SAVRY, and the structured professional judgment approach, as a method to assess violence risk and case planning needs among youths involved in the legal system. These findings provide more confidence that significant differences in SAVRY risk level or items between Black and White youths are not based in measurement bias, with the exception of the community disorganization item. Potential fixes discussed include eliminating or deemphasizing this item in final risk level ratings. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Development and Evaluation of an Out-of-Control Scale Measuring Global Contextual Stressors Across 20 Countries and Regions

Abstract: Stressful contexts like the COVID-19 pandemic precipitated remarkable socio-economic changes, exacerbating universal uncertainty and a pervasive loss of control. Improving mental health outcomes on the operational definition and measurement of related concepts. This study analyzed cross-sectional data from 20 countries and regions, collected between January 25 and November 10, 2021. Psychometric evaluation occurred in two stages: initially, item review and factor analysis with a 2,749 adult sample from Mainland of China and Hong Kong SAR; subsequently, confirmation of factor structure, measurement invariance testing, and advanced analyses with a 4,000 adult sample from 20 countries and regions. A six-item, single-factor structure was verified, confirming satisfactory factor loading, fit indices and consistency. Measurement invariance across geographies and groupings was also confirmed. Three Out-of-Control profiles representing low, medium, and high stress levels, were identified, with age, depression/anxiety symptoms, education, socio-economic status, and exposure to COVID-19-affected acquaintances significantly influencing affiliation. The item “I think I cannot control things that I previously could” was the most critical node. This newly validated scale represents a substantial advancement in measuring stress within the context of global health crises, thus enhancing comprehension of psychological responses. Continual evaluation of the scale’s effectiveness and accuracy is crucial.

Validation of the Parent-report Pandemic Anxiety Scale (PAS-P) in the context of COVID-19

AbstractTo be able to develop effective policy and targeted support for children and young people, it is vital to develop and validate measures that enable us to understand what aspects of pandemics are associated with anxiety and stress across a wide age range. We examined the psychometric properties of the Pandemic Anxiety Scale– Parent-report (PAS-P), which measures levels of child and adolescent pandemic-related anxieties. Factor structure, reliability, and convergent and discriminant validity of the PAS-P was assessed in a convenience sample of parents/carers (N­ = 8410) over at three time points in 2020 when COVID-19 case rates and restrictions varied. Factor structure was identified via two exploratory factor analyses (EFAs; n = 5601 and n = 1005) and then tested using confirmatory factor analysis (CFA; n = 800), measurement invariance tests, and a longitudinal CFA (n = 1651). Two factors structure for disease and consequence anxiety were observed across both EFAs and were found to have good fit in the CFAs. The PAS-P demonstrated good internal consistency and discriminant validity, as well as partial scalar invariance in latent construct measurement across child age, gender, and time. The PAS-P is a robust parent-report measure of two distinct forms of pandemic-related anxiety, suitable for reporting on children and adolescents aged 4-16 years. Although the scale has been validated in the context of the COVID-19 pandemic, it is not limited to this specific pandemic and, with minor wording modifications, may be a reliable tool in other health epidemic contexts.

Posttraumatic stress disorder (PTSD) in a sample of patients evaluated for acute coronary syndrome: A factor analysis of the PTSD checklist for DSM-5 (PCL-5)

The PCL-5 is a psychometrically sound measure of Post-Traumatic Stress Disorder (PTSD) symptoms. Although the scale is commonly applied to past external traumas (e.g., combat, assault), PTSD symptoms have also been assessed about potentially life-threatening cardiovascular events that represent ongoing internal threats. To date, there is a paucity of studies that have examined the updated scale factor structure for PTSD in patients with suspected acute coronary syndromes (ACS). MethodsExploratory and confirmatory factor analyses (EFA, CFA) were conducted using PCL-5 data completed by 830 patients enrolled in the REactions to Acute Care and Hospitalization (REACH) study, an observational cohort study of patients recruited from the emergency department during evaluation for ACS. Follow-up measurement invariance tests were conducted on pre-selected models and on the best-fitting model identified by EFA to evaluate invariance across diagnosis (confirmed v. rule-out ACS), sex, and language. ResultsThe EFA identified a two-factor model with “Memories of Trauma” (MT) and “Cognitive Behavioral Symptoms” (CBS) factors offering a balanced fit and interpretability. In CFAs, the Anhedonia CFA model performed the best overall. Measurement invariance tests supported strong invariance across confirmed and ruled-out ACS, male and female sex, and English and Spanish language for all models. LimitationsOnly 34 % of the sample was diagnosed with ACS at discharge, which limits generalizability. ConclusionOur study contributes to the understanding of PTSD in the context of internal traumatic reminders and may guide future research and clinical practice by informing intervention targets to improve health and well-being after suspected ACS.

A serial mediation model of Big 5 personality traits, emotional intelligence, and psychological capital as predictors of teachers' professional well-being

This study aimed to explore the relationship among the Big Five personality traits, emotional intelligence (EIQ), psychological capital (PsyCap), and teacher well-being (TPWB) within the context of higher education. The objective was to predict TPWB in university teachers by utilizing the Big Five personality traits and PsyCap while considering EI as a mediator. Data were collected from 708 teachers in Ethiopian higher education institutions. Participants completed the Emotional Intelligence Scale (EIS-16), Big Five Personality Inventory (BFI-10), Psychological Capital Questionnaire (PCQ-12), and Teacher Professional Well-Being Scale (TPWBS). The analysis included reliability tests, correlation analysis, validity assessment, measurement invariance, and serial mediation testing. The findings unveil a significant direct positive effect of Big Five personality traits on EIQ, PsyCap, and TPWB. Furthermore, PsyCap demonstrates a direct positive effect on TPWB. PsyCap fully mediates the relationship between the Big Five personality traits, EIQ, and TPWB, while EIQ partially mediates the relationships between the Big Five personality traits and PsyCap/TPWB, accentuating a serial mediation effect. Moreover, the personality trait of openness to experience positively predicts EIQ, PsyCap, and TPWB. Conscientiousness and agreeableness also positively predict EIQ, while extraversion directly influences PsyCap in a positive manner. However, neuroticism exerts a negative direct impact on EIQ, PsyCap, and TPWB. Additionally, emotional intelligence partially mediates the relationship between the five dimensions of the Big Five personality (openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism) and TPWB. In conclusion, PsyCap fully mediates the relationship between the Big Five personality traits, EIQ, and TPWB, while EIQ partially mediates the relationships between the Big Five personality traits and PsyCap/TPWB. These findings hold significant implications for enhancing well-being among teachers.

New insights into validity, reliability, and measurement invariance of the Kansas City Cardiomyopathy Questionnaire 23

Background: People affected by heart failure (HF) often exhibit a poor health status, which places a great deal of burden on the healthcare systems. The Kansas City Cardiomyopathy Questionnaire (KCCQ) 23 is the most used tool to measure the health status in this population; however, its psychometric properties have not been thoroughly established.Objective: To evaluate the psychometric properties of the KCCQ-23 in a European cohort of HF patients.Methods: 510 patients (median age 74 years, IQR=18, 58% males) completed the KCCQ-23 along with clinical and psychosocial measures. Factorial validity was established with confirmatory factor analysis (CFA); omega and model-based internal consistency indexes were computed to examine the internal consistency of the scale. Convergent validity was established by correlating the KCCQ-23 scores with clinical and psychosocial measures. Measurement invariance tests across those with preserved vs reduced ejection fraction were conducted within a multigroup framework.Results: Two CFA solutions were tested, which confirmed the theoretical and empirical models postulated by the original author. The internal consistency coefficients for the latent dimensions were adequate (Omega range =0.77-0.93; internal consistency coefficient =0.89-96). KCCQ-23 scores were found to be correlated with ejection fraction, NYHA levels, quality of life, self-care confidence, anxiety and depression, and symptom burden, supporting its convergent validity. Finally, the KCCQ-23 was invariant across ejection fraction levels, both in the theoretical and empirical factor solution.Conclusion: Overall, this study provides evidence of satisfactory psychometric properties of the KCCQ-23, promoting its use in clinical settings and research fields.

A Study of the Psychometric Properties of the Pediatric Symptom Checklist-17 for Children With Developmental Delays and Disorders.

Primary care physicians (PCPs) frequently serve pediatric patients with developmental delays and disorders (DD/D). Although the most widely used primary care behavioral health screener, the Pediatric Symptom Checklist-17 (PSC-17), is validated for use with children without DD/D, it is unclear whether this measure accurately identifies behavioral health symptoms in youth with DD/D. Thus, the purpose of this study was to assess the psychometric properties of the PSC-17 for children with DD/D. Medical record data from 3596 pediatric patients at a primary care clinic were analyzed. Descriptive analyses, measurement invariance testing, and internal consistency evaluations were conducted to assess the psychometric properties of the PSC-17. The results of these analyses support the use of the PSC-17 for behavioral health screening for children with DD/D. Behavioral health screening in this population is critical, because the timely identification of behavioral health concerns can facilitate early intervention, which may enhance long-term functioning.

Development and validation of the Transgender Adolescent Stress Survey-Dysphoria.

Transgender and nonbinary adolescents (TNBA) may experience gender dysphoria arising from incongruities between their body and their gender. Prior dysphoria measures have largely focused on clinical diagnosis with little regard to comparability of forms for people assigned male or female at birth, overall psychometric performance, or applicability to nonbinary populations. This study develops and validates the Transgender Adolescent Stress Survey-Dysphoria (TASS-D), intended to address these gaps. The current study recruited a U.S. national sample of TNBA (N = 444, aged 12-17; 65.5% White, 9.5% Black, 9.5% Latine, 15.5% other ethnicity; 34.7% transmasculine, 17.3% transfeminine, 38.3% nonbinary, 9.5% agender). The item pool was developed from life history calendars, a modified Delphi process, and cognitive interviews with TNBA. Scale development included factor analysis, item response theory modeling, measurement invariance testing, and reliability analyses. Associations were examined between the TASS-D and existing measures of gender dysphoria (convergent validity), gender minority stress (divergent validity), and behavioral health outcomes (criterion validity). TASS-D and its subscales (body distress and gender expression burden) were significantly and strongly associated with gender dysphoria; significantly but weakly associated with gender minority stress; and significantly associated with most indicators of psychological distress including depressive, anxiety, and posttraumatic stress symptoms, suicidal behaviors and nonsuicidal self-injury. The TASS-D is a reliable and valid measure of gender dysphoria for TNBA, offering notable benefits over existing measures: It is psychometrically sound, inclusive of all gender identities, and does not assume that respondents identify binarily or desire medical transition as a terminal goal.

Test-Retest Reliability of the STRAQ-1: A Registered Report

This Registered Report provides the first test of measurement invariance across time points and estimates of test-retest reliability for the Social Thermoregulation, Risk Avoidance Questionnaire (STRAQ-1, Vergara et al., 2019). The scale was developed and validated to understand the physiological drives underlying interpersonal bonding, measured by four constructs: the desire to socially regulate one’s temperature, the desire to solitary regulate one’s temperature, the sensitivity to higher temperatures, and the desire to avoid risk. Previous studies with large samples across 12 countries showed that the STRAQ-1 has a stable factorial structure, satisfying internal consistencies for the temperature subscales, and expected correlations in its nomological network. However, to date, this instrument has no estimates of test-retest reliability. Throughout four academic years (from 2018 to 2022), N = 183 French student participants took the STRAQ-1 at least two times. Out of the four STRAQ-1 subscales, only two were longitudinally invariant across two-time points. Thus, half of the STRAQ-1 constructs and latent scores were dissimilar and incomparable across time. We then conducted test-retest reliability using Intra Class Correlation coefficient (ICC) for the Social Thermoregulation, Solitary Thermoregulation, High-Temperature Sensitivity, and Risk Avoidance subscales. ICCs estimates were respectively for agreement and consistency: .70, .70 overall moderate to good, .62, .62 overall moderate, .67, .67 overall moderate, and .53, .53 overall poor to moderate, respectively. Our study suggests that test-retest reliability was insufficient for psychological diagnosis, and that future studies should address the problem of low generalizability of the constructs.

From MIsgivings to MIse-en-scène: The role of invariance in personality science

There are increasing vocal concerns about the application of measurement invariance testing arguing that it is overly strict and arbitrary. We argue that invariance is not just a procedural hurdle but a substantive tool that enhances the understanding of psychological constructs across diverse populations and has important implications for both theory testing and theory development. First, we outline the importance of how invariance, in a broad sense, plays a role at all the major steps within a research cycle, involving both theoretical and methodological concerns. Second, we suggest a list of points linked to these invariance concerns that can benefit research reports to improve reliability, validity, and fairness. We see invariance as a crucial part of scientific inquiry and an informative tool for empirical research. We agree with Funder and Gardiner’s point that “Data are data,” but would like to add that invariance inquiries and their implications help making sense of the data and the underlying world.

Psychometric properties of the Online Learner Satisfaction Instrument (OLSI): construct validity, convergent validity, and measurement invariance

The present study evaluated the validity of the Online Learner Satisfaction Instrument (OLSI) in terms of construct and convergent validity, and measurement invariance across demographic groups. The study results revealed that the factor structures generally confirmed to the original factor structure proposed in the instrument. Also, some subfactors showed a high correlation, suggesting a potential unidimensionality within each factor. The results of the convergent validity showed that the course subscale had high correlations with learner and instructor subscales, yielding inter-subscale correlations. The results of the measurement invariance testing showed that OLSI was interpreted the same way across participants of different gender and ethnic groups. However, measurement invariance was not supported across different educational levels (undergraduate vs. graduate).

The validation of short eating disorder, body dysmorphia, and Weight Bias Internalisation Scales among UK adults

BackgroundWhen collecting data from human participants, it is often important to minimise the length of questionnaire-based measures. This makes it possible to ensure that the data collection is as engaging as possible, while it also reduces response burden, which may protect data quality. Brevity is especially important when assessing eating disorders and related phenomena, as minimising questions pertaining to shame-ridden, unpleasant experiences may in turn minimise any negative affect experienced whilst responding.MethodsWe relied on item response theory to shorten three eating disorder and body dysmorphia measures, while aiming to ensure that the information assessed by the scales remained as close to that assessed by the original scales as possible. We further tested measurement invariance, correlations among different versions of the same scales as well as different measures, and explored additional properties of each scale, including their internal consistency. Additionally, we explored the performance of the 3-item version of the modified Weight Bias Internalisation Scale and compared it to that of the 11-item version of the scale.ResultsWe introduce a 5-item version of the Eating Disorder Examination Questionnaire, a 3-item version of the SCOFF questionnaire, and a 3-item version of the Dysmorphic Concern Questionnaire. The results revealed that, across a sample of UK adults (N = 987, ages 18–86, M = 45.21), the short scales had a reasonably good fit. Significant positive correlations between the longer and shorter versions of the scales and their significant positive, albeit somewhat weaker correlations to other, related measures support their convergent and discriminant validity. The results followed a similar pattern across the young adult subsample (N = 375, ages 18–39, M = 28.56).ConclusionsThese results indicate that the short forms of the tested scales may perform similarly to the full versions.

Measurement Invariance Testing Works.

Psychometricians have argued that measurement invariance (MI) testing is needed to know if the same psychological constructs are measured in different groups. Data from five experiments allowed that position to be tested. In the first, participants answered questionnaires on belief in free will and either the meaning of life or the meaning of a nonsense concept called "gavagai." Since the meaning of life and the meaning of gavagai conceptually differ, MI should have been violated when groups were treated like their measurements were identical. MI was severely violated, indicating the questionnaires were interpreted differently. In the second and third experiments, participants were randomized to watch treatment videos explaining figural matrices rules or task-irrelevant control videos. Participants then took intelligence and figural matrices tests. The intervention worked and the experimental group had an additional influence on figural matrix performance in the form of knowing matrix rules, so their performance on the matrices tests violated MI and was anomalously high for their intelligence levels. In both experiments, MI was severely violated. In the fourth and fifth experiments, individuals were exposed to growth mindset interventions that a twin study revealed changed the amount of genetic variance in the target mindset measure without affecting other variables. When comparing treatment and control groups, MI was attainable before but not after treatment. Moreover, the control group showed longitudinal invariance, but the same was untrue for the treatment group. MI testing is likely able to show if the same things are measured in different groups.

Age-related differences in trait affect: Establishing measurement invariance of the Positive and Negative Affect Schedule (PANAS).

The present study examined trajectories of trait positive and trait negative affect as measured by the Positive and Negative Affect Schedule across the lifespan. Increasing levels of measurement invariance of the Positive and Negative Affect Schedule were tested across a wide age range (18-99 years of age; 10 age groups) in a large sample (N = 3,309; 65.82% women; 74.22% with a university entrance diploma; 92.23% with German as their mother tongue). Strong invariance was established so that the latent factor means of positive and negative affect could be meaningfully compared across age groups. Age had a small effect on positive affect, which showed a reversed U-shaped function with a maximum between the ages of 40 and 50. For negative affect, there was a strong cubic age effect. Specifically, negative affect decreased until about the age of 70 and then increased again. We also found a negative covariance between positive and negative affect, which first increased slightly and then decreased across age groups until becoming positive in the oldest age group. In summary, the present study supports previous findings on age trajectories of positive and negative affect but also highlights the importance of testing for measurement invariance. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Examining the Measurement Invariance of the Revised Sociosexual Orientation Inventory Among Hispanic and Non-Hispanic White College Women in the U.S.

The Revised Sociosexual Orientation Inventory (SOI-R) is a widely used measure in research, yet the invariance of this measure has not been established in English speaking Non-Hispanic White (NHW) and Hispanic/Latine populations. This study examined whether the SOI-R, a measure developed in Germany, was invariant between US Hispanic/Latina (N = 208) and NHW (N = 190) undergraduate women. Confirmatory factor analysis (CFA) was used to assess model fit in the Hispanic/Latina and NHW samples and fit of increasingly restrictive models was used to test configural, metric, scalar, and residual invariance of the models in both samples. CFA results revealed that data from both the Hispanic/Latina and NHW groups fit the model adequately in this sample, which consisted of highly acculturated Hispanic/Latina college women. Tests of measurement invariance found that the SOI-R was invariant across Hispanic/Latina and NHW college women. However, questions about the development of the SOI-R and the underlying assumptions made during the course of its development might be considered prior to the use of the measure in research, and further invariance testing should be conducted in future work with less acculturated Hispanic/Latine populations.