Multitrait-multimethod Analysis Research Articles

Comparison of patient-reported outcomes measurement information system (PROMIS®)-29 and PROMIS global physical and mental health scores

PurposeThe Patient-Reported Outcomes Measurement and Information System (PROMIS®): includes the PROMIS-29 physical and mental health summary and the PROMIS global physical and mental health scores. It is unknown how these scores coincide with one another. This study examines whether the scores yield similar or different information.MethodsThe PROMIS-29 and the PROMIS global health items were administered to 5804 adults from Amazon’s Mechanical Turk (MTurk) in 2021–2022 and to 4060 adults in the Ipsos KnowledgePanel (KP) in 2022.ResultsThe median age of those in MTurk (KP) was 36 (54) and 53% (50%) were male. Mean T-scores on the PROMIS-29 and PROMIS global physical health scales were similar, but PROMIS global mental health was 3–4 points lower than the PROMIS-29 mental health summary score. Product-moment correlations ranged from 0.69 to 0.81 between the PROMIS-29 physical health and PROMIS global physical health scales and 0.56–0.69 between the mental health scales. Multi-trait multimethod analyses indicated that only a small proportion of the correlations between the two methods of measuring mental health were significantly more highly correlated with one another than correlations between physical and mental health.ConclusionsPROMIS-29 and PROMIS global mental health scales provide different information and, therefore, study conclusions may vary depending on which measure is used. Interpretation of results needs to consider that the PROMIS-29 mental health scale is a weighted combination of specific domains while the PROMIS global mental health scale is based on general mental health perceptions. Further comparisons of methods of assessing mental health are needed.

Psychometric properties of a Creole version of Medical Outcome Study - Short Form 36 among type II diabetes patients on Reunion Island.

Health quality of life assessment is particularly important to measure the impact of chronic diseases. The aims of this study were to provide a cross-culturally adapted Creole-translation of the Medical Outcome Study Short-Form 36 (SF-36) and to assess psychometric performance of the Creole and French versions of the SF-36 among patients with type II diabetes in Reunion Island. The Creole translation and cross-cultural adaptation processes were based on the International Quality Of Life Assessment (IQOLA) methods. Internal consistency, test-retest reliability, convergent and discriminant validity using Multi-Trait-Multi-Method analysis and structural validity using exploratory factor analysis of the SF-36 for both versions were performed. In the Creole version of the SF-36, Cronbach's alpha exceeded 0.70 for all subscales except general health. In the French SF-36, Cronbach's alpha exceeded 0.70 on all subscales except general health and bodily pain. In the Creole SF-36, intraclass correlation coefficient (ICC) for reproducibility was suboptimal. Multi-trait multi-method analysis showed that item-scale correlation exceeded 0.4 for all items except two general health items of the Creole SF-36 and one of the French SF-36. Factor analysis of 2 versions showed that the physical functioning, vitality, and mental health were each divided into two subscales. Overall, our findings provided evidence that the SF-36 is adapted to Reunion Island in both Creole and French versions. However, further research could be conducted to investigate French-Creole differences in perceived health status and a cultural adaptation of the French version will be considered.

Two of a Kind or Distant Relatives?

Abstract: Allport’s distinction of personality devaluated (personality) and personality evaluated (character) raised the question of whether the character is redundant with personality, which still remains open today. The present study hence compares the Five-Factor Model of personality and the VIA-classification (Values in Action) of character strengths across two methods (self- and peer-reports) and two levels of abstraction (domains/factors and scales/facets). A sample of 152 participants and 152 peer-raters completed the NEO-Personality Inventory-Revised and the VIA Inventory of Strengths (VIA-IS). Personality and character assessed with these inventories were found to strongly overlap, yet the different operationalizations were rarely redundant (except for three personality facets). Multitrait-multimethod analyses mostly supported the convergent and discriminant validity of personality and character. Interpersonal strengths (e.g., teamwork) and abstract character factors lacked discriminant validity to personality facets. The present investigation contributes to a better understanding of the interplay between personality and character and provides an impetus for future research on the “virtue gap” between devaluated and evaluated personality traits.

Construct-related validity of the strengths and difficulties questionnaires with three and five dimensions: A multitrait-multimethod analysis.

The Strengths and Difficulties Questionnaire (SDQ) is one of the most broadly used questionnaires to evaluate children's psychological adjustment, however its internal structure has been a target of ongoing controversy. Recent studies suggested a three-factor structure of the SDQ, however data is still scarce. The present study used the Multitrait-Multimethod analysis to examine SDQ construct related-validity with three and five dimensions, provided by children, their parents and teachers. A total of 415 participants were recruited from a Portuguese community sample. Both SDQ versions presented good convergence-related validity, with higher values for the five version. Findings from this study suggest that the SDQ with three dimensions could be more suitable as a screening measure of children's psychological adjustment in a community low-risk sample. Nevertheless, the SDQ still needs further psychometric improvements in order to properly collect information from multi-source samples about the prevalence of children's psychological adjustment.

Development and validation of the Work Style Reform Scale.

In Japan, the "Act on the Arrangement of Related Acts to Promote Work Style Reform" was enforced in 2019 to reduce long working hours. However, work style reforms largely depend on workers' and companies' awareness and abilities. This study aimed to develop the Work Style Reform scale to assess the competencies required for work style reform and to examine its validity and reliability. This study was conducted with a diverse sample of 1,641 Japanese workers, and 17 questions across three subscales-Work Style, Personal Style, and Work Environment-were developed. After a response bias analysis, some items were discarded based on an exploratory factor analysis; the reliability coefficients were calculated; and a confirmatory factor analysis was performed. The convergent and discriminant validity were evaluated through a multi-trait multi-method analysis. Finally, nine questions remained. Construct validity, reliability and convergent validity were sufficient for all Subscales, and discriminant validity was sufficient only for Work Style. In conclusion, while Work Style was sufficient, issues remained in other subscales. To improve the accuracy of them in the future, it is necessary to examine the validation of discriminant validity using different indicates and the addition of new items to the smallest subscale; Personal Style.

Construct elaboration and validity of the Pregnancy Depression Risk Scale.

to elaborate and analyze the Pregnancy Depression Risk Scale psychometric properties. methodological research, in six steps: theoretical model empirical definition; elaboration of scale items with literature review; consultation with five professional health experts and 15 pregnant women; content validity with six experts; pre-test-semantic validity with 24 pregnant women; scale factor structure definition with 350 pregnant women; pilot study with 100 pregnant women, totaling 489 pregnant women and 11 experts. Data were analyzed by content analysis, exploratory factor analysis, multitrait-multimethod analysis and internal consistency. sixty-eight risk factors were identified for item formulation. The final version of the scale consisted of 24 items in five domains. The scale demonstrated satisfactory construct content, semantic, validity and reliability. the scale proved to be valid in terms of content and semantics, with a factor structure defined according to the adopted theoretical model and satisfactory psychometric properties.

The convergent validity of the Romanian version of the Behavioral Assessment of Children: A Multitrait-Multimethod Analysis

This study assessed the validity of the Behavior Assessment System for Children 2nd Edition, by providing evidence based on a multitraitmultimethod (MTMM) analysis performed on data obtained with a culturally adapted form of the test. The study included two Romanian samples (161 adolescents aged 12-18 years and 91 children aged 8-12 years) that were each assessed with all the three age corresponding forms (self, parent, and teacher) of the test. We used a confirmatory factor analytic framework in order to test the convergent and discriminant validity of BASC-2, by generating a CFAbased multi-trait, multi-method latent factor correlation matrix. Results showed that, for both the Adolescent and Child forms, the coefficients falling under the validity diagonal (mono-trait, hetero-method) have the highest median values, compared to the coefficients encountered in the monomethod block and those in the heteromethod-heterotrait triangles. The study provides evidence for the construct validity of the multi-trait, multi-rater assessment system that is the basis of the BASC-2.

Preliminary Psychometric Validation of the Teammate Burnout Questionnaire.

The aim of the present study was to provide support for the validation of the Teammate Burnout Questionnaire (TBQ). Athletes from a variety of team sports (N = 290) completed the TBQ and the Athlete Burnout Questionnaire (ABQ). Confirmatory factor analysis revealed acceptable fit indexes for the three-dimensional models (i.e., physical and emotional exhaustion, sport devaluation, reduced accomplishment) of the TBQ and the ABQ. Multi-trait multi-method analysis revealed that the TBQ and ABQ showed acceptable convergent and discriminant validity. The preliminary validation of the TBQ indicates the utility of the scale to reflect athletes’ perceptions of their teammates’ burnout and offers researchers the opportunity to quantitatively assess an important aspect of the social environment in the development of athlete burnout.

Recruiting "clean" chronic insomnia participants: the unicorn of sleep research.

Recruiting "clean" chronic insomnia participants: the unicorn of sleep research.

Indexing dynamic collective constructs using computer-aided text analysis: Construct validity evidence and illustrations featuring team processes.

Organizational processes have been widely recognized as both multilevel and dynamic, yet traditional methods of measurements limit our ability to model and understand such phenomena. Featuring a popular model of team processes advanced by Marks et al. (2001), we illustrate a method to use individuals' communications as construct valid unobtrusive measures of collective constructs occurring over time. Thus, the purpose of this investigation is to develop computer-aided text analysis (CATA) techniques that can score members' communications into valid team process measures. We apply a deductive content validity-based method to construct CATA dictionaries for Marks et al.'s dimensions. We then demonstrate their convergent validity with subject matter experts' (SMEs) hand-coded team communications and different SMEs' behaviorally anchored rating scales based on video recordings of team interactions, using multitrait-multimethod analyses in two samples. Using a third sample of paramedics performing a high-fidelity mass casualty incident exercise, we further demonstrate the convergent validity of the CATA and SME scorings of communications. We then model the relationships among processes across episodes using all three samples. Next, we test criterion-related validity using a longitudinal dual-discontinuous change growth modeling design featuring the paramedic CATA-scored team processes as related to a dynamic performance criterion. Finally, we integrate behavioral data from wearable sensor badges to illustrate how CATA can be scored at the individual level and then leveraged to model dynamic networks of team interactions. Implications, limitations, directions for the future research, and guidelines for the application of these techniques to other collective constructs are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Convergence and Psychometric Properties of Character Strengths Measures: The VIA-IS and the VIA-IS-R

This study compares the German versions of the original measure of character strengths (VIA-IS) with its latest revision (VIA-IS-R) regarding reliability and convergent, discriminant, and criterion-related validity. A sample of 499 German-speaking adults (79% women, mean age: 33.3 years) provided self-reports of character strengths (VIA-IS, VIA-IS-R) and several criteria: Core virtues, thriving, and moral behaviors. Results suggested that both measures showed satisfactory internal consistency and converged well in a multitrait-multimethod analysis. Further, both measures were comparable regarding their relationships with the criteria. Overall, the results of the current study suggest that both questionnaires are reliable and valid, and that findings based on these instruments can be considered highly comparable.

Validity and reliability of the 8-item EUROHIS-QOL to assess Brazilian adults’ quality of life

The study evaluated the psychometric properties of the EUROHIS-QOL 8-item scale in Brazilian adults. The EUROHIS-QOL is a measurement of quality of life developed based on the generic instruments of WHOQOL-100 and WHOQOL-BREF. Based on data from a prospective cohort of 1,100 adults, participants in the baseline in 2006 and 2007, exploratory factor analysis, internal consistency, reliability, and discriminant validity were performed via multitrait-multimethod analysis, with the calculation of the average variance extracted (AVE). Confirmatory factor analysis was conducted with 573 of these participants in 2013 to 2018. Solutions were tested with one and with two factors named functional capacity and socioeconomic conditions. Both solutions presented good internal consistency and reliability. The correlation between the items was 0.535, and the AVE was 0.397 for solution with one factor, and 0.528 and 0.341 for factors one and two, respectively, indicating good discriminant validity in the bifactorial solution. Both factors had eigenvalues greater than one and factor loadings ranging from 0.398 to 0.915. The adjustment of the unifactorial solution presented: χ2 = 186 (g.l. = 18), p < 0.001, RMSEA = 0.128 (90%CI: 0.111-0.145), CFI = 0.960, TLI = 0.938, and SMRM = 0.042; while in the bifactorial presented: χ2 = 135 (g.l. = 17), p < 0.001, RMSEA = 0.110 (90%CI: 0.093-0.128), CFI = 0.972, TLI = 0.954, and SMRM = 0.035. Overall, the results provide reasonable evidence for the construct validity and reliability of the EUROHIS-QOL 8-item. Instruments with fewer items, as in our case, provide greater practicality and are ideal for use in population-based studies with extensive samples.

Positive Psychological Capital (PsyCap) Meets Multitrait-Multimethod Analysis: Is PsyCap a Robust Predictor of Well-Being and Performance Controlling for Self-Report Bias?

A growing body of evidence suggests positive psychological capital (PsyCap) strongly predicts well-being and performance at work. However, most of this empirical research has used self-report survey designs, increasing the possibility of self-report and mono-method bias. The current study used a multitrait-multimethod (MTMM) research design and condition-based regression analysis to examine the effect of PsyCap on job well-being and work role performance beyond self-report bias. Findings from 416 full-time employees within 208 coworker pairs showed that self-reported and informant-reported PsyCap were predictive of job well-being and work role performance. However, multitrait-multimethod analyses showed monomethod measures may inflate the strength of association between PsyCap and work outcomes. Implications for future applied positive psychology research controlling for self-report and monomethod bias with multiple data sources are discussed.

Development and preliminary evaluation of the Neurofibromatosis Type 1 Adult Quality of Life (NF1-AdQoL) questionnaire.

Neurofibromatosis Type 1 (NF1) is a variable and unpredictable multisystem genetic disorder that predisposes to medical complications, cognitive impairment and disfigurement, of all which can impact negatively upon the health-related quality of life (HRQoL) of affected adults. To develop and validate a disease-specific HRQoL adult questionnaire to evaluate effects of NF1 from the patient's viewpoint. The Neurofibromatosis Type 1 Adult Health-related Quality of Life questionnaire (NF1-AdQoL) was based on patient interviews (n = 8), clinician survey and questionnaire pilot study. Adults with NF1 (n = 114, aged 18-40 years) were recruited from three Australian genetics clinics and completed the NF1-AdQoL, the 29-item Skindex (Skindex-29) and the 36-item Short Form, version 2 (SF-36v2) questionnaires. An exploratory factor analysis of the NF1-AdQoL was conducted to assess construct validity. Convergent and discriminant validity of the NF1-AdQoL was determined by using multitrait multimethod analysis with Skindex-29 and SF-36v2 scores. Factor analysis indicated that 62.7% of the common variance between the questionnaires could be explained by three factors: 'emotions associated with cosmetic appearance' (12 items), 'functioning - social and learning' (11 items) and 'physical symptoms' (8 items). NF1-AdQoL had good internal consistency (Cronbach α = 0.96). Convergent validity was confirmed by moderate associations with similarly named scales of the Skindex-29 and SF-36v2. Results from all three HRQoL questionnaires indicated overall healthy HRQoL for young to early middle-aged adults with NF1. The NF1-AdQoL is a relatively valid, feasible and fairly easy to read tool to measure the HRQoL of adults with NF1. Further evaluation is required to determine the test-retest reliability, responsiveness and validity of the NF1-AdQoL in larger adult NF1 cohorts.

An Implicit Measure of Sexual Double Standard Endorsement in Emerging Adults: Reliability and Validity Aspects

Psychometric characteristics were investigated of an Implicit Association Test to assess implicit endorsement of the sexual double standard (SDS-IAT) in emerging adults. The reliability of the SDS-IAT was investigated focusing on internal consistency across different phases of the test. Convergent validity of the SDS-IAT was evaluated against the Scale for the Assessment of Sexual Standards in Youth, an explicit measure of SDS, and against gender investment. Divergent validity was evaluated against the personality characteristics of extraversion, neuroticism, and social desirability proneness. Gendered patterns were examined. Attenuation-corrected alphas demonstrated acceptable internal consistency, with alphas ranging for .65–.70. A modest level of explicit SDS endorsement was found in both female and male participants. In line with their explicit SDS level, a modest level of implicit SDS endorsement was found in male participants, whereas a reverse implicit SDS was found among young women. In agreement with our theoretical expectations, we found low convergent validity in multitrait-multimethod analysis of the SDS-IAT with a measure of explicit SDS endorsement, and with general level of investment in gender ideals. Similarly, divergent validity analysis revealed absence of significant correlations with the conceptually unrelated concepts of extraversion, neuroticism, and social desirability proneness, except for extraversion in female participants. The present findings suggest that implicit SDS endorsement can be assessed using the SDS-IAT. The finding that explicit and implicit SDS approvals differ in young female participants, while they align in young male participants, warrants further research.

Multi-Informant Assessment of Bullying in Austrian Schools

This study explores the agreement of bullying reports from different perspectives. Nominations and (self and peer) ratings of 721 students (50.5% boys, 49.5% girls, aged 9–11), 46 teachers (91.3% female, 8.7% male;Mage= 40.28 years,SDage= 1.6, teaching experience between 2 and 36 years,Mexperience= 15.16,SDexperience= 1.68), 439 mothers, and 363 fathers were analyzed. Measures included a modified version of the Revised Olweus Bully/Victim questionnaire (OBVQ) as well as nomination items, with variables on physical and relational bullying and victimization, and frequency of experience. For descriptive analyses, group comparisons and correlations were performed and to test the dimensional structure of the modified Revised OBVQ we fitted a categorical confirmatory factor analysis (CCFA) for all four raters. To estimate the degree of agreement between the four raters, we applied a simple multitrait-multimethod (MTMM) analysis in a CCFA framework. Results indicated that teachers are more likely to recognize physical bullying than relational bullying. Significant gender effects were found, with more frequent nominations for boys and a larger gender effect for physical bullying. Overall, teachers reported the highest levels of bullying, and students reported significantly higher levels of victimization compared to the other raters. In terms of bullying, the MTMM model revealed a slightly higher agreement between students and parents than between students and teachers. The findings further indicate a tendency for lower agreement between student and adult reports than among adults’ reports. The notably high agreement between the ratings of mothers and fathers for both bullying and victimization supports the research strategy to include only one parent. Although bullying is assessed from various perspectives, the study does not provide prevalence estimates of bullying in Austria.

Examining PERMA+4 and work role performance beyond self-report bias: insights from multitrait-multimethod analyses

ABSTRACT Self-report and monomethod bias threaten the validity of positive work and organizations research. The current study used multitrait-multimethod analyses to examine the relationship between PERMA+4 and work role performance (adaptivity, proactivity, and proficiency) beyond self-report and monomethod bias. Findings from 221 coworker pairs demonstrated convergence between self-reported and collateral-reported PERMA+4 (r > .85) and work role performance (r > .85). A multitrait-multimethod matrix showed that monomethod trait interrelationships were systematically higher than heteromethod trait interrelationships with inflated correlations ranging from .07 to .20. Monomethod parameter estimates and coefficients of determination were generally higher than in bias corrected self-reports, knowledgeable collateral-reports, and heteromethod parameter estimates. The results provide support for the validity of the relationship between PERMA+4 and work role performance, but also suggest the importance of including procedural design and statistical control methods in positive work and organizations surveys to correct for self-report and monomethod bias.

Comparing Two Subjective Rating Scales Assessing Cognitive Load During Technology-Enhanced STEM Laboratory Courses

Cognitive load theory is considered universally applicable to all kinds of learning scenarios. However, instead of a universal method for measuring cognitive load that suits different learning contexts or target groups, there is a great variety of assessment approaches. Particularly common are subjective rating scales, which even allow for measuring the three assumed types of cognitive load in a differentiated way. Although these scales have been proven to be effective for various learning tasks, they might not be an optimal fit for the learning demands of specific complex environments such as technology-enhanced STEM laboratory courses. The aim of this research was therefore to examine and compare the existing rating scales in terms of validity for this learning context and to identify options for adaptation, if necessary. For the present study, the two most common subjective rating scales that are known to differentiate between load types (the cognitive load scale by Leppink et al. and the naïve rating scale by Klepsch et al.) were slightly adapted to the context of learning through structured hands-on experimentation where elements such as measurement data, experimental setups, and experimental tasks affect knowledge acquisition. N = 95 engineering students performed six experiments examining basic electric circuits where they had to explore fundamental relationships between physical quantities based on the observed data. Immediately after the experimentation, the students answered both adapted scales. Various indicators of validity, which considered the scales’ internal structure and their relation to variables such as group allocation as participants were randomly assigned to two conditions with a contrasting spatial arrangement of the measurement data, were analyzed. For the given dataset, the intended three-factorial structure could not be confirmed, and most of the a priori-defined subscales showed insufficient internal consistency. A multitrait–multimethod analysis suggests convergent and discriminant evidence between the scales which could not be confirmed sufficiently. The two contrasted experimental conditions were expected to result in different ratings for the extraneous load, which was solely detected by one adapted scale. As a further step, two new scales were assembled based on the overall item pool and the given dataset. They revealed a three-factorial structure in accordance with the three types of load and seemed to be promising new tools, although their subscales for extraneous load still suffer from low reliability scores.

Multitrait Multimethod validation of Implicit Association Test as a measure of individual differences in personality: Is there personality at all?

The objective of the study was to investigate the validity of IAT in the assessment of the Big five personality traits. We employed the Multi-Trait Multi-Method (MTMM) approach on a sample of 146 respondents. We collected measures on IATs assessing Big five personality traits, self-reports, and two ratings of close others of different gender on NEO PI-R. Additionally, we assessed behavior related to the Big five personality traits in a structured interview where three experts independently provided rating measures. MTMM analyses suggest that personality IAT measures have little in common with personality traits measured traditionally using explicit methods. The evidence does not support the claim that personality IAT can serve as a valid measure of individual differences in personality traits.

DEVELOPMENT AND VALIDATION OF QUALITY OF LIFE SCALE FOR PEOPLE LIVING WITH HIV

ABSTRACT Objective: to develop and validate a scale to measure the quality of life of people living with HIV in Brazil. Method: methodological study conducted in a Brazilian care service specialized in sexually transmissible infections/AIDS between 2017 and 2019 addressing people living with HIV. The scale’s development and validation included exploratory factor analysis to describe its factor structure and psychometric properties, Multitrait-Multimethod analysis to verify its validity and Cronbach’s alpha for reliability. Floor and ceiling effects were described according to the responses’ frequency distribution. Results: a total of 460 people living with HIV participated. Most were men 276 (60.0%) aged 43 on average (SD=±12.4). The Exploratory Factor Analysis revealed four factors with 39.9% of explained variance. The total scale presented satisfactory reliability with a Cronbach’s alpha equal to 85.0%. Most items presented satisfactory convergent and divergent validity. The presence of floor and ceiling effects were found. The scale’s final version was composed of 45 items. Conclusion: the Quali-HIV Scale is a valid and reliable tool to measure the quality of life of people living with HIV.