Assumption Of Measurement Invariance Research Articles

Internalized Shame Among Justice-Involved Women in Substance Use Disorder Treatment: Measurement Invariance and Changes During Treatment.

Shame is one of the leading barriers to successful recovery in substance use treatment settings. This secondary analysis study examined measurement invariance of the Internalized Shame Scale (ISS) and explored changes in shame during treatment. Participants (N=105) in the parent study were recruited from a nonprofit residential treatment center for justice-involved women and were randomized to receive mindfulness-based relapse prevention or relapse prevention treatment. A series of confirmatory factor analyses were used to assess measurement invariance in a one-factor measurement model of the ISS. Latent growth curve modeling was used to examine change in shame over time. Our findings support the assumption of measurement invariance across multiple time points and across treatment conditions, supporting comparisons of stigma scores across groups and over time. Although we observed significant reductions in shame from pre- to post-treatment, there were no differences across treatment conditions. Additional research is needed to determine how distinct treatment components relate to reductions in shame among individuals receiving treatment for a substance use disorder.

Measurement Invariance of the NIH Toolbox‐Cognition Battery Across Demographic Characteristics

AbstractBackgroundThe tablet‐based NIH Toolbox‐Cognition Battery (NIHTB‐CB) was compared against the established NACC Uniform Data Set 3.0 (UDS 3.0) neuropsychological battery (used across Alzheimer’s Disease Research Centers). NIHTB‐CB factor invariance is critical for drawing appropriate cross‐demographic conclusions. Invariance was assessed across both batteries for consensus‐driven clinical diagnosis, sex, and age in a large sample of participants from the Advancing Reliable Measurement in Alzheimer’s Disease and Cognitive Aging (ARMADA) study. Goals of this project were to establish the factor relationship of these two batteries and examine their invariance across the aforementioned demographic characteristics.MethodNIHTB‐CB tasks measure Crystalized and Fluid ability. Tests were administered to 571 participants aged 65 to 99 years (mean age 77.2±7.8, 41% female) with normal cognition or amnestic mild cognitive impairment. Groups were classified for age (< = 85 and > 85), gender, and diagnosis. Exploratory Factor Analysis (EFA) was tested for each group with confirmatory factor analysis. If the model fit sufficiently between groups for the configural model, then metric, scalar, and strict invariance models were also examined. Uncorrected standard scores were used for analyses.ResultA four‐factor model emerged as the best fitting model and was invariant across all demographic groups through adjustments. Factor 1: Language (MINT Total, Oral Reading, Picture Vocabulary); Factor 2: Auditory retention (Digits Forward Total and Digits Backward Total); Factor 3: Learning/Memory (List Sorting Working Memory, Picture Sequence Memory, Craft Story 21 Immediate Recall, Craft Story 21 Delayed Recall, Benson Delayed Recall, Animals Category, Vegetables Category, and MoCA Total Score); Factor 4: Attention/Executive Functioning (Dimensional Card Sort, Flanker, Trails A, Trails B, Pattern Comparison Processing Speed, and Benson Recall). We found strong support for strict invariance across the assessed demographic characteristics.ConclusionA four‐factor structure comprised of NIHTB‐CB and UDS 3.0 tests supports the validity of the NIHTB‐CB by matching both UDS 3.0 and NIHTB‐CB measures as expected. Strong support for strict invariance reflects the psychometric equivalence of the NIHTB‐CB construct across the groups. The psychometric properties of the NIHTB‐CB subtests therefore are generalizable across samples. By satisfying the assumption of measurement invariance, inferences made using the NIHTB‐CB about between‐group differences are presumed valid.

Examining the assumption of measurement invariance in job performance ratings across time: The role of rater experience

AbstractPerformance appraisals are widely used in organizations and most typically involve raters evaluating groups of subordinates along a set of items designed to represent job performance over a predetermined period (e.g., annually). A defining but often overlooked characteristic of performance appraisals is that they are cyclical. Since raters conduct appraisals over many cycles, it may be that measures of job performance are not equivalent across time. This is important because changes or differences in aggregated performance ratings can only be meaningfully interpreted if raters' definitions of job performance, interpretation of what the items mean, and their view of what constitutes the different levels of performance remain unchanged over time, unaffected by their experience with appraisals. Although critical to the interpretation of job performance scores, measurement invariance concerns are generally absent from the literature. The current research investigated the extent to which rater experience affected the conceptualization and measurement of performance using performance data from a major South American company which comprised information from raters and ratees through several appraisal cycles. In the between‐rater design, measurement invariance was analyzed using ratings of one performance appraisal cycle from 514 raters divided into groups according to their level of experience. The within‐rater design analyzed ratings from the same 80 raters in their first three appraisal cycles. In the between‐rater analysis, data supported measurement invariance across raters with different levels of experience. Results from the within‐rater analysis suggested that the job performance factor structure was not the same across cycles. Implications for research and practice are discussed.

Measurement Invariance of the General Health Questionnaire GHQ 12-Item Version (GHQ-12)

Abstract: Understanding how levels, patterns, predictors, and outcomes of mental health issues differs in students relative to non-students can inform more effective and better tailored prevention and intervention for mental health in higher education contexts. However, comparisons of mental health in student and non-student groups depend on the critical but seldom-tested assumption of measurement invariance. In this study, we use data from the UK household longitudinal study (UKLS) to evaluate the measurement invariance of the scores from a commonly used mental health measure: the General Health Questionnaire 12-item version (GHQ-12) across students and non-students. Using a bifactor model to take account of wording factors we found measurement invariance up to the scalar level for students and non-student groups. This provides support for the use of instruments for comparing mental health issue levels and candidate risk factors and outcomes across students and non-students.

Why Measurement Invariance is Important in Comparative Research. A Response to Welzel et al. (2021)

Welzel et al. (2021) claim that non-invariance of instruments is inconclusive and inconsequential in the field for cross-cultural value measurement. In this response, we contend that several key arguments on which Welzel et al. (2021) base their critique of invariance testing are conceptually and statistically incorrect. First, Welzel et al. (2021) claim that value measurement follows a formative rather than reflective logic. Yet they do not provide sufficient theoretical arguments for this conceptualization, nor do they discuss the disadvantages of this approach for validation of instruments. Second, their claim that strong inter-item correlations cannot be retrieved when means are close to the endpoint of scales ignores the existence of factor-analytic approaches for ordered-categorical indicators. Third, Welzel et al. (2021) propose that rather than of relying on invariance tests, comparability can be assessed by studying the connection with theoretically related constructs. However, their proposal ignores that external validation through nomological linkages hinges on the assumption of comparability. By means of two examples, we illustrate that violating the assumptions of measurement invariance can distort conclusions substantially. Following the advice of Welzel et al. (2021) implies discarding a tool that has proven to be very useful for comparativists.

Estimation Methods of the Multiple-Group One-Dimensional Factor Model: Implied Identification Constraints in the Violation of Measurement Invariance

Factor analysis is one of the most important statistical tools for analyzing multivariate data (i.e., items) in the social sciences. An essential case is the comparison of multiple groups on a one-dimensional factor variable that can be interpreted as a summary of the items. The assumption of measurement invariance is a frequently employed assumption that enables the comparison of the factor variable across groups. This article discusses different estimation methods of the multiple-group one-dimensional factor model under violations of measurement invariance (i.e., measurement noninvariance). In detail, joint estimation, linking methods, and regularized estimation approaches are treated. It is argued that linking approaches and regularization approaches can be equivalent to joint estimation approaches if appropriate (robust) loss functions are employed. Each of the estimation approaches defines identification constraints of parameters that quantify violations of measurement invariance. We argue in the discussion section that the fitted multiple-group one-dimensional factor analysis will likely be misspecified due to the violation of measurement invariance. Hence, because there is always indeterminacy in determining group comparisons of the factor variable under noninvariance, the preference of particular fitting strategies such as partial invariance over alternatives is unjustified. In contrast, researchers purposely define fitting functions that minimize the extent of model misspecification due to the choice of a particular (robust) loss function.

An Exploration of Measurement Invariance across Sex in Intimate Partner Violence Perpetration

ABSTRACT The present study examines the assumption of measurement invariance across sex from two instruments of intimate partner violence (IPV) perpetration. Data were used from two studies of self-reported IPV perpetration among adult college students. Study 1 employs a subset of the Revised Conflict Tactics Scale (CTS2) for 901 young adult females and males. Study 2 employs a subset of the Safe Dates scale for 599 young adult females and males. Multiple-group confirmatory factor analysis was estimated in a forward, sequential process to test measurement invariance of IPV perpetration across sex. Both studies indicated multiple items violated the assumption of measurement invariance across sex for the latent IPV perpetration constructs. Findings revealed the IPV perpetration scales suffered from sex-bias for certain IPV behavior items. Future research should explore sources of variance in IPV perpetration scales and their impact on key correlates.

Dyadic measurement invariance and its importance for replicability in romantic relationship science

AbstractComparisons of group means, variances, correlations, and/or regression slopes involving psychological variables rely on an assumption of measurement invariance—that the latent variables under investigation have equivalent meaning and measurement across group. When measures are noninvariant, replicability suffers, as comparisons are either conceptually meaningless, or hindered by inflated Type I error rates. We propose that the failure to account for interdependence among dyad members when testing measurement invariance may be a potential source of unreplicable findings in relationship research. We developed fully dyadic versions of invariance models, created an R package (dySEM) to make specifying dyadic invariance models easier and reporting more reproducible, and executed a Registered Report for gauging the extent of dyadic (non)invariance in romantic relationship research across measures of relationship well‐being, personality, and sexuality in a sample of 282 heterosexual couples. We found that although a number of popular measures display good evidence of dyadic invariance, a few display concerning levels and interesting patterns of noninvariance, while others appeared either noninvariant or poorly fitting for both men and women. We discuss our findings in terms of their meaning for the replicability dyadic close relationship research. We close by arguing that increased theorizing and research on dyadic invariance, and inclusive methods for analyzing invariance with indistinguishable dyads, are needed to capitalize on the opportunity to advance our field's understanding of dyadic constructions of relational concepts.

The Adolescent Peer Relations Instrument- Bully/Target: Measurement Invariance Across Gender, Age, and Clinical Status.

Adolescent Peer Relations Instrument-Bully/Target (APRI-BT) is a multidimensional scale designed to assess bullying involvement both as target and perpetrator. Although existing research has shown that the APRI-BT satisfies the assumption of measurement invariance across age and gender, these findings come from western individualistic countries (e.g., Australia). This study aimed to investigate the factorial structure and measurement invariance across age, gender, and clinical status in a sample of Romanian youths. Participants were 1,024 adolescents, 10 to 18 years, recruited from both community and clinical setting. Our results confirmed a six first-order factor structure and two second-order factors (Bully including Bullying Physical, Bullying Verbal, Bullying Social and Victimization including Physical Victimization, Verbal Victimization, Social Victimization). In addition, measurement invariance across age, gender, and clinical status was demonstrated. This study identifies APRI-BT as an instrument with solid psychometric proprieties for measuring bullying and victimization among preadolescents and adolescents.

Longitudinal Measurement Invariance of the Brief Symptom Inventory (BSI)-18 in Psychotherapy Patients

Abstract. The Brief Symptom Inventory (BSI)-18 is a widely-used tool to assess changes in general distress in patients despite an ongoing debate about its factorial structure and lack of evidence for longitudinal measurement invariance (LMI). We investigated BSI-18 scores from 1,081 patients from an outpatient clinic collected after the 2nd, 6th, 10th, 18th, and 26th therapy session. Confirmatory factor analysis (CFA) was used to compare models comprising one, three, and four latent dimensions that were proposed in the literature. LMI was investigated using a series of model comparisons, based on chi-square tests, effect sizes, and changes in comparative fit index (CFI). Psychological distress diminished over the course of therapy. A four-factor structure (depression, somatic symptoms, generalized anxiety, and panic) showed the best fit to the data at all measurement occasions. The series of model comparisons showed that constraining parameters to be equal across time resulted in very small decreases in model fit that did not exceed the cutoff for the assumption of measurement in variance. Our results show that the BSI-18 is best conceptualized as a four-dimensional tool that exhibits strict longitudinal measurement invariance. Clinicians and applied researchers do not have to be concerned about the interpretation of mean differences over time.

Consistency matters: measurement invariance of the EORTC QLQ-C30 questionnaire in patients with hematologic malignancies.

To ensure that observed differences in the scores of the European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 (EORTC QLQ-C30) reflect actual differences in health-related quality of life (HRQoL) rather than measurement bias, measurement invariance needs to be established. We investigated the assumption of measurement invariance of the EORTC QLQ-C30 in patients with hematological malignancies across age, sex, comorbidity, disease type, and time. We used a large database of patients with hematological malignancies, which included HRQoL data collected with the EORTC QLQ-C30. We used the structural equation modeling approach to test for measurement (metric and scalar) invariance across groups (age, sex, comorbidity, disease) and time (baseline, 1month and 2month follow-up). Longitudinal invariance was examined in a subgroup of patients diagnosed with myelodysplastic syndromes. Confirmatory factor analyses demonstrated full measurement invariance for age and comorbidity and over time, while support for partial scalar invariance was obtained for sex and disease. Violations of invariance for sex were observed for items of the physical functioning scale and the emotional functioning scale, while for disease type, violations of invariance were observed for items of the physical functioning scale, emotional functioning scale, and the cognitive functioning scale. Our findings support measurement invariance of the EORTC QLQ-C30 in a large sample of patients with hematological malignancies. The results showed that the number of non-invariant items was negligible, suggesting that this questionnaire is a valid and robust measurement tool in patients with hematological malignancies, also for comparisons across groups and time.

Factorial Invariance of the German Version of the Marital Offence-Specific Forgiveness Scale (MOFS)

Abstract. Spousal forgiveness is positively linked to aspects of relationship satisfaction, as well as to the physical and psychological health of both partners. Oftentimes, the respective values for men and women are compared under the untested assumption of measurement invariance of the assessment tool. In order to investigate the factorial invariance (configural, metric, scalar, and residual) of the German version of the Marital Offence-Specific Forgiveness Scale (MOFS; Paleari et al., 2009 ), we conducted several multigroup confirmatory factor analyses. In the context of comparing increasingly stringent models, tests of measurement invariance indicated at least partial measurement invariance. These findings suggest that the MOFS operates in a similar fashion among men and women, thus supporting the assessment of spousal forgiveness with the German MOFS.

Measurement invariance of TGMD-3 in children with and without mental and behavioral disorders.

This study evaluated whether the Test of Gross Motor Development 3 (TGMD-3) is a reliable tool to compare children with and without mental and behavioral disorders across gross motor skill domains. A total of 1,075 children (aged 3-11 years), 98 with mental and behavioral disorders and 977 without (typically developing), were included in the analyses. The TGMD-3 evaluates fundamental gross motor skills of children across two domains: locomotor skills and ball skills. Two independent testers simultaneously observed children's performances (agreement over 95%). Each child completed one practice and then two formal trials. Scores were recorded only during the two formal trials. Multigroup confirmatory factor analysis tested the assumption of TGMD-3 measurement invariance across disability groups. According to the magnitude of changes in root mean square error of approximation and comparative fit index between nested models, the assumption of measurement invariance across groups was valid. Loadings of the manifest indicators on locomotor and ball skills were significant (p < .001) in both groups. Item response theory analysis showed good reliability results across locomotor and the ball skills full latent traits. The present study confirmed the factorial structure of TGMD-3 and demonstrated its feasibility across normally developing children and children with mental and behavioral disorders. These findings provide new opportunities for understanding the effect of specific intervention strategies on this population. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Establishing a Longitudinal Comparable Scale of Chinese Children's Cognitive Development through Calibrated Projection Linking.

In the past decades, the longitudinal approach has been remarkably and increasingly used in the investigations of children's cognitive development. Recently, many researchers have started to realize the importance and necessity of examining measurement invariance for any further longitudinal analysis. However, there are few empirical studies demonstrating how to conduct further analysis when the assumption of measurement invariance of an instrument is violated. The primary purpose of this study is to explore how a newly-developed calibrated projection method can be applied to reduce the impact of lack of parameter invariance in a longitudinal study of preschool children's cognitive development. The sample consisted of 882 children from China who participated in two waves of the cognitive tests when they were 4 and 5 years old. Before this study was conducted, the IRT method was used to examine the measurement invariance of the instrument. The results showed that five items presented difficulty parameter drift and three items presented discrimination/slope parameter drift. In the study, the invariant items were treated as “common items” and calibrated projection linking was used to establish a comparable scale across two time points. Then the linking method was evaluated by three properties: grade-to-grade growth, grade-to-grade variability, and the separation of distributions. The results showed that the grade-to-grade growth across two waves was larger and exhibited a larger effect size; the grade-to-grade variability showed less scale shrinkage, which indicated a smaller measurement error; the separation of distributions showed a larger growth as well.

Class Enumeration and Parameter Recovery of Growth Mixture Modeling and Second-Order Growth Mixture Modeling in the Presence of Measurement Noninvariance between Latent Classes.

Population heterogeneity in growth trajectories can be detected with growth mixture modeling (GMM). It is common that researchers compute composite scores of repeated measures and use them as multiple indicators of growth factors (baseline performance and growth) assuming measurement invariance between latent classes. Considering that the assumption of measurement invariance does not always hold, we investigate the impact of measurement noninvariance on class enumeration and parameter recovery in GMM through a Monte Carlo simulation study (Study 1). In Study 2, we examine the class enumeration and parameter recovery of the second-order growth mixture modeling (SOGMM) that incorporates measurement models at the first order level. Thus, SOGMM estimates growth trajectory parameters with reliable sources of variance, that is, common factor variance of repeated measures and allows heterogeneity in measurement parameters between latent classes. The class enumeration rates are examined with information criteria such as AIC, BIC, sample-size adjusted BIC, and hierarchical BIC under various simulation conditions. The results of Study 1 showed that the parameter estimates of baseline performance and growth factor means were biased to the degree of measurement noninvariance even when the correct number of latent classes was extracted. In Study 2, the class enumeration accuracy of SOGMM depended on information criteria, class separation, and sample size. The estimates of baseline performance and growth factor mean differences between classes were generally unbiased but the size of measurement noninvariance was underestimated. Overall, SOGMM is advantageous in that it yields unbiased estimates of growth trajectory parameters and more accurate class enumeration compared to GMM by incorporating measurement models.

Is the Assessment of Personality Comparable in Persons Who Have and Have Not Experienced Depressive, Anxiety, and Substance Use Disorders? An Examination of Measurement Invariance.

Numerous studies have focused on characterizing personality differences between individuals with and without psychopathology. For drawing valid conclusions for these comparisons, the personality instruments used must demonstrate psychometric equivalence. However, we are unaware of any studies that examine measurement invariance in personality across individuals with and without psychopathology. This study conducted tests of measurement invariance for positive emotionality, negative emotionality, and disinhibition across individuals with and without histories of depressive, anxiety, and substance use disorders. We found consistent evidence that positive emotionality, negative emotionality, and disinhibition were assessed equivalently across all comparisons with each demonstrating strict invariance. Overall, results suggest that comparisons of personality measures between diagnostic groups satisfy the assumption of measurement invariance and these scales represent the same psychological constructs. Thus, mean-level comparisons across these groups are valid tests.

Assuming measurement invariance of background indicators in international comparative educational achievement studies: a challenge for the interpretation of achievement differences

BackgroundLarge-scale cross-national studies designed to measure student achievement use different social, cultural, economic and other background variables to explain observed differences in that achievement. Prior to their inclusion into a prediction model, these variables are commonly scaled into latent background indices. To allow cross-national comparisons of the latent indices, measurement invariance is assumed. However, it is unclear whether the assumption of measurement invariance has some influence on the results of the prediction model, thus challenging the reliability and validity of cross-national comparisons of predicted results.MethodsTo establish the effect size attributed to different degrees of measurement invariance, we rescaled the ‘home resource for learning index’ (HRL) for the 37 countries (n=166,709 students) that participated in the IEA’s combined ‘Progress in International Reading Literacy Study’ (PIRLS) and ‘Trends in International Mathematics and Science Study’ (TIMSS) assessments of 2011. We used (a) two different measurement models [one-parameter model (1PL) and two-parameter model (2PL)] with (b) two different degrees of measurement invariance, resulting in four different models. We introduced the different HRL indices as predictors in a generalized linear mixed model (GLMM) with mathematics achievement as the dependent variable. We then compared three outcomes across countries and by scaling model: (1) the differing fit-values of the measurement models, (2) the estimated discrimination parameters, and (3) the estimated regression coefficients.ResultsThe least restrictive measurement model fitted the data best, and the degree of assumed measurement invariance of the HRL indices influenced the random effects of the GLMM in all but one country. For one-third of the countries, the fixed effects of the GLMM also related to the degree of assumed measurement invariance.ConclusionThe results support the use of country-specific measurement models for scaling the HRL index. In general, equating procedures could be used for cross-national comparisons of the latent indices when country-specific measurement models are fitted. Cross-national comparisons of the coefficients of the GLMM should take into account the applied measurement model for scaling the HRL indices. This process could be achieved by, for example, adjusting the standard errors of the coefficients.

Measuring Cross-Cultural Supernatural Beliefs with Self- and Peer-Reports.

Despite claims about the universality of religious belief, whether religiosity scales have the same meaning when administered inter-subjectively–or translated and applied cross-culturally–is currently unknown. Using the recent “Supernatural Belief Scale” (SBS), we present a primer on how to verify the strong assumptions of measurement invariance required in research on religion. A comparison of two independent samples, Croatians and New Zealanders, showed that, despite a sophisticated psychometric model, measurement invariance could be demonstrated for the SBS except for two noninvariant intercepts. We present a new approach for inspecting measurement invariance across self- and peer-reports as two dependent samples. Although supernatural beliefs may be hard to observe in others, the measurement model was fully invariant for Croatians and their nominated peers. The results not only establish, for the first time, a valid measure of religious supernatural belief across two groups of different language and culture, but also demonstrate a general invariance test for distinguishable dyad members nested within the same targets. More effort needs to be made to design and validate cross-culturally applicable measures of religiosity.

The role of multiple-group measurement invariance in family psychology research.

Measurement invariance (MI) is a property of measurement that is often implicitly assumed, but in many cases, not tested. When the assumption of MI is tested, it generally involves determining if the measurement holds longitudinally or cross-culturally. A growing literature shows that other groupings can, and should, be considered as well. Additionally, it is noted that the standard techniques for investigating MI have been focused almost exclusively on the case of 2 groups, with very little work on the case of more than 2 groups, even though the need for such techniques is apparent in many fields of research. This paper introduces and illustrates a model building technique to investigating MI for more than 2 groups. This technique is an extension of the already-existing hierarchy for testing MI introduced by Meredith (1993). An example using data on father involvement in 5 different groups of families of children with and without developmental disabilities from the Early Childhood Longitudinal Study-Birth Cohort dataset will be given. We show that without considering the possible differential functioning of the measurements on multiple developmental groups, the differences present between the groups in terms of the measurements may be obscured. This could lead to incorrect conclusions.

Measurement Invariance of the WHODAS 2.0 in a Population-Based Sample of Youth

The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is a brief measure of global disability originally developed for adults, which has since been implemented among samples of children and youth. However, evidence of its validity for use among youth, particularly measurement invariance, is lacking. Investigations of measurement invariance assess the extent to which the psychometric properties of observed items in a measure are generalizable across samples. Satisfying the assumption of measurement invariance is critical for any inferences about between-group differences. The objective of this paper was to empirically assess the measurement invariance of the 12-item interview version of the WHODAS 2.0 measure in an epidemiological sample of youth (15 to 17 years) and adults (≥ 18 years) in Canada. Multiple-group confirmatory factor analysis using a categorical variable framework allowed for the sequential testing of increasingly restrictive models to evaluate measurement invariance of the WHODAS 2.0 between adults and youth. Findings provided evidence for full measurement invariance of the WHODAS 2.0 in youth aged 15 to 17 years. The final model fit the data well: χ2(159) = 769.04, p < .001; CFI = 0.950, TLI = 0.958, RMSEA (90% CI) = 0.055 [0.051, 0.059]. Results from this study build on previous work supporting the validity of the WHODAS 2.0. Findings indicate that the WHODAS 2.0 is valid for making substantive comparisons of disability among youth as young as 15 years of age.