Trait-correlated Method Research Articles

Item Wording Effects in Psychological Measures: Do Early Literacy Skills Matter?

While the inclusion of both positively and negatively worded items is a common practice in scales, using positively and negatively worded items together may threaten the validity of a scale. Several studies have been devoted to investigating the effects of item wording methods. The current study investigated item wording effects on the responses of 4028 Turkish fifth-grade students, who responded to the Students Confidence in Mathematics (SCM) and Students Confidence in Science (SCS) scales. The role of early literacy-related variables (i.e., early literacy activities undertaken before primary school, student performance on reading literacy tasks upon entering primary school, and duration of the children’s pre-primary school attendance) on item wording effects was also examined. The investigations were conducted using confirmatory factor analysis and the correlated trait–correlated method minus one CFA- CTC(M-1) model, derived from the correlated traits-correlated methods framework. The results indicate that significant item wording effects existed in both scales. Moreover, a significant and positive effect was found in both scales relating to early literacy activities undertaken before school, but no effects were found relating to student performance on reading literacy tasks upon entering primary school or duration of the children’s pre-primary school attendance. Overall, the study suggests that researchers and practitioners should consider potential effects when including both positively and negatively worded items in scales, especially scales designed for younger students.

Internal Structure of Beck Hopelessness Scale: An Analysis of Method Effects Using the CT-C(M–1) Model

ABTRACT The construct validity in relation to the dimensionality or factor structure of the Beck Hopelessness Scale (BHS) has long been debated in psychometrics. Irrelevant variance due to item wording (method effects) can distort the factor structure, and recent studies have examined the method factor’s role in the factor structure of the BHS. However, the models used to control the method effects have severe limitations, and new models are needed. One such model is the correlated trait-correlated method minus one (CT-C(M-1)), which is a powerful approach that gives the trait factor an unambiguous meaning and prevents the anomalous results associated with fully symmetrical bifactor modeling. The present work compares the fit and factor structure of the CT-C(M-1) model to bifactor models proposed in previous literature and evaluates the criterion validity of the CT-C(M-1) model and its discriminatory capacity by taking suicidal ideation as the criterion variable. This study used a large and heterogeneous open mode online sample of Argentinian people (N = 2,164). The results indicated that the CT-C(M-1) model with positive words as referenced items achieves the most adequate factor structure. The factorial scores derived from this model demonstrate good predictive and discriminating capabilities.

The Strengths and Difficulties Questionnaire: An examination of factorial, convergent, and discriminant validity using multitrait-multirater data.

This study examined the factorial, convergent, and discriminant validity of scores on the Strengths and Difficulties Questionnaire (SDQ), a tool for screening children's psychological adjustment. Data were collected from a community sample of 346 children (46% girls, M age = 54.07 months), using teacher, mother, and father reports. Results from confirmatory factor analyses indicated that the SDQ's hypothesized 5-factor structure fit the data well and partial strict measurement invariance was established across raters. Using teachers' reports as reference method, a correlated trait-correlated method minus 1 model (Eid et al., 2008) was fitted to investigate convergent and discriminant validity. The convergent validity of parents' ratings relative to teachers' ratings was modest. Mothers and fathers had a unique perspective on children's behavior above and beyond their partial overlap with teacher reports. Results indicated good discriminant validity between most of the traits measured by the SDQ. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Convergent and Discriminant Validities of SCBE-30 Questionnaire Using Correlated Trait–Correlated Method Minus One

Correlated trait–correlated method minus one was used to evaluate convergent and discriminant validity of Social Competence Behavior Evaluation questionnaire (Social Competence, Anger-Aggression, Anxiety-Withdrawal) between multiple raters. A total of 369 children (173 boys and 196 girls; Mage = 55.85, SDage = 11.54) were rated by their mothers, fathers, and teachers. Results showed more convergence between parents than parent-teacher ratings. Mother-teacher share a common view of child behavior that is not shared with father. Parents had more difficulty distinguishing internalizing and externalizing behaviors (especially fathers). Measurement invariance across child sex was explored, results imply that differences between boys and girls were not due to measure. Girls (compare to boys) were described as more social competent by their fathers and teachers, while boys as more aggressive by mothers and teachers.

The Nature of Rater Effects and Differences in Multilevel MTMM Latent Variable Models

ABSTRACT Measuring and understanding the nature of informant/rater effects and differences (Level 1) on a common trait when the target of measurement is at the organizational level (Level 2) involves a number of methodological considerations. Although previous research has discussed single-level latent variable applications of the correlated trait-correlated method minus one (CT-C(M-1)) model for multitrait-multimethod (MTMM) designs, only recently was the CT-C(M-1) model derived for multilevel data structures. The current paper examines research on the development of multilevel latent variable measurement models when both structurally different and interchangeable raters are involved in measuring Level 2 constructs. In so doing, we review and articulate often overlooked design and interpretive considerations unique to these models. We describe these elements within the context of a real-world multilevel CT-C(M-1) latent variable model that measures school climate for 294 public high schools using individual-level ratings from students (N = 58,018), teachers (N = 12,236), and health professionals (N = 742).

BIS and BAS Sensitivities at Different Levels of Personality Description: A Latent-Variable Approach with Self- and Informant-Ratings

We examine the structural overlap of the Behavioral Inhibition System (BIS) and the Behavioral Approach System (BAS) with Stability and Plasticity, the two higher-order factors encompassing the Big Five. Carver and White’s BIS/BAS and the Big Five Inventory were administered to a sample of 330 adults, serving both as targets and informants. Self- and other-ratings were modeled by using the Correlated Trait-Correlated Method model. BIS and BAS correlated highly with metatraits, after method variance and measurement error were partialled out: BIS was positively related to Stability, while BAS was positively related to Plasticity and negatively related to Stability. After the higher-order factors were controlled, the BIS was highly and positively related to Emotional stability, whereas the BAS had a small but significant relationship with Extraversion. Findings are discussed with regard to the most appropriate level of generality/specificity at which the personality correlates of BIS and BAS can be investigated.

What Do Teachers Think About Their Students' Inclusion? Consistency of Students' Self-Reports and Teacher Ratings.

The aim of this study was to investigate the consistency between the self-reports and teacher ratings of students’ emotional and social inclusion at school as well as for their academic self-concept. The German version of the Perceptions of Inclusion Questionnaire (PIQ) was administered to 329 grade 8 students (50.8% female, Mage = 14.5 years, SDage = 0.5 years) and their teachers. First, the three-dimensional structure of both PIQ versions was confirmed by confirmatory item factor analysis. The α and ω coefficients demonstrated good reliability for all scales. Second, a correlated trait-correlated method minus one model provided evidence that the method-specificity of teacher ratings was larger than the consistency between the self-reports and teacher ratings. Third, the results of a latent difference model indicated that general method effects can partly be explained by a student’s gender or special educational needs. Finally, the low consistency between self-reports and teacher rating is discussed.

Are Workaholism and Work Engagement in the Eye of the Beholder?

Abstract. Virtually all studies on workaholism and engagement – two forms of heavy work investment – rely on self-part questionnaires. However, the limitations of self-reports are widely acknowledged and in their final sections, papers on workaholism and engagement typically lament the use of such measures. Investigating data other than respondents’ self-reports, such as peer ratings, may overcome these limitations. Using a sample of 73 dyads composed of focal workers and their colleagues, the present study aimed: (1) to compare focal workers’ and coworkers’ perceptions concerning their levels of work engagement and workaholism; and (2) to explore the discriminant validity of engagement and workaholism. A multitrait-multimethod (MTMM) matrix and a correlated trait-correlated method model, the CT-C(M-1) model, were examined. Our results showed a considerable agreement between both raters (i.e., focal workers and coworkers) in levels of engagement and workaholism. In contrast to previous findings, a significant difference between raters on the cognitive dimension of workaholism (i.e., working compulsively) was observed. Moreover, our results provided further evidence for the discriminant validity between the two forms of heavy work investment.

Student–student relations from the teacher versus student perspective: A multi-level confirmatory factor analysis

ABSTRACTEducational research emphasizes the advantages of multimethod designs. However, if the design comprises different perspectives, the question of construct validity emerges. We related this question to student and teacher ratings of student–student relations, which are of high interest in research on physical education. In our study, 2,160 students and their 114 physical education teachers answered parallel items. Data was analyzed by applying the multi-level correlated trait-correlated method minus one approach, which makes it possible to consider multiple indicators per trait and to distinguish between trait, method, and error components. The student–student relations subscales student conflicts, class cohesion, and pleasure in demonstrating were reliable and valid regarding the two perspectives. Moreover, teacher and student ratings of student–student relations were consistent to a certain extent. The administered questionnaire was suitable for both perspectives. The applied model provides a good basis for similar research questions regarding construct validity.

Isolating Trait and Method Variance in the Measurement of Callous and Unemotional Traits.

To examine hypothesized influence of method variance from negatively keyed items in measurement of callous-unemotional (CU) traits, nine a priori confirmatory factor analysis model comparisons of the Inventory of Callous-Unemotional Traits were evaluated on multiple fit indices and theoretical coherence. Tested models included a unidimensional model, a three-factor model, a three-bifactor model, an item response theory-shortened model, two item-parceled models, and three correlated trait-correlated method minus one models (unidimensional, correlated three-factor, and bifactor). Data were self-reports of 234 adolescents (191 juvenile offenders, 43 high school students; 63% male; ages 11-17 years). Consistent with hypotheses, models accounting for method variance substantially improved fit to the data. Additionally, bifactor models with a general CU factor better fit the data compared with correlated factor models, suggesting a general CU factor is important to understanding the construct of CU traits. Future Inventory of Callous-Unemotional Traits analyses should account for method variance from item keying and response bias to isolate trait variance.

Structure of phonological ability at age four

Three research questions were investigated: 1) can phonological tasks be characterized in terms of two facets, a processing complexity facet and a linguistic complexity facet, 2) can performance on the phonological tasks be subsumed under one general phonological awareness factor, and 3) how are phonological abilities and other cognitive abilities related? A test battery of tasks with a complete crossing of the two phonological awareness facets was given to 364 children aged four, along with a test battery with tasks measuring cognitive abilities (Gf, Visual, Verbal). A “correlated trait-correlated method minus one” measurement model fit the data well, and supported the proposed model with two dimensions of phonological awareness. Further analyses demonstrated that the phonological tasks can be subsumed under a dominating general phonological awareness factor and orthogonal narrow factors representing the processing and linguistic complexity factors. A structural equation model in which the linguistic and processing complexity factors were regressed on Gf showed that Gf was highly related to the processing complexity factors but not to the linguistic factors. This suggests that the well-established impact of Gf on reading skills is mediated through phonological awareness. The Verbal factor was related to the linguistic complexity factors when Gf was partialed out but not to the processing complexity factors. The Visual factor did not relate to any of the phonological factors. These results support the view that the Verbal and Visual factors are modality specific, and do not represent general processing capacity.

Multilevel multitrait-multimethod latent analysis of structurally different and interchangeable raters of school climate.

Informant-based systems of assessment are common platforms for measuring a variety of educational and psychological constructs where the use of multiple informants is considered best practice. In many instances, structurally different informant types (e.g., students and teachers) are solicited on the basis of their unique roles with the target of measurement. The use of multiple informants provides an opportunity to evaluate the degree to which the obtained ratings are influenced by the trait of focus and extraneous sources that can be attributed to the rater. Data from a multilevel multitrait-multimethod design in which students (N = 35,565) and teachers (N = 9,112), from 340 middle schools, responded to items measuring 3 dimensions of school climate were evaluated through a multilevel correlated trait-correlated method latent variable model. Results indicated that ratings of school climate obtained by students and teachers demonstrated high levels of convergent validity, and that school-level ratings obtained by students and teachers were equitable in the assessment of teasing and bullying. Student ratings of support and structure yielded somewhat stronger evidence of convergent validity than ratings obtained by teachers as revealed by their respective trait factor loadings. This was explained in part by the higher levels of common method effects that were observed for teachers.

Are Autonomous and Controlled Motivations School-Subjects-Specific?

This research sought to test whether autonomous and controlled motivations are specific to school subjects or more general to the school context. In two cross-sectional studies, 252 elementary school children (43.7% male; mean age = 10.7 years, SD = 1.3 years) and 334 junior high school children (49.7% male, mean age = 14.07 years, SD = 1.01 years) were administered a questionnaire assessing their motivation for various school subjects. Results based on structural equation modeling using the correlated trait-correlated method minus one model (CTCM-1) showed that autonomous and controlled motivations assessed at the school subject level are not equally school-subject-specific. We found larger specificity effects for autonomous (intrinsic and identified) than for controlled (introjected and external) motivation. In both studies, results of factor loadings and the correlations with self-concept and achievement demonstrated that more evidence of specificity was obtained for autonomous regulations than for controlled ones. These findings suggest a new understanding of the hierarchical and multidimensional academic structure of autonomous and controlled motivations and of the mechanisms involved in the development of types of regulations for school subjects.

The validity of retrospectively reported conflict interactions in couples.

This study investigated the extent to which researchers and clinicians can obtain valid retrospective self-reports of couples' conflict interactions outside a laboratory setting. A distinction was made between relationship attribute variance, regarding a shared perspective of both partners, and informant-specific variance, regarding the unique vantage point of each partner. By examining convergent and divergent associations for each type of variance, this study clarified the risk that responses might be influenced by informant-specific biases related to levels of relationship satisfaction. This study also investigated potential moderators of validity. Participants included both members of 269 married and cohabiting couples (538 individuals) who completed online questionnaires. Results were analyzed using a correlated trait-correlated method minus one model. The total true variance included large components of both shared relationship attribute variance and informant-specific variance. Although the shared component was moderately correlated with relationship satisfaction, the informant-specific component was mostly distinct from satisfaction, suggesting minimal bias. Convergent correlations between partners were strong and mostly unrelated to potential moderating variables, albeit slightly smaller than reported in studies conducted in laboratory settings. The results generally support the validity for retrospective self-reports of conflict interactions, especially when reports are obtained from both members of a couple.

Agreement of adolescent ratings with mother ratings and teacher ratings of ADHD symptom groups: A correlated trait-correlated method minus one analysis

This study examined the level of agreement of adolescent ratings with mother ratings, and adolescent ratings with teacher ratings of the inattention (IA) and hyperactivity/impulsivity (HI) symptom groups of ADHD. A total of 214 adolescents provided self-ratings of IA and HI, and their IA and HI were also rated by their mothers and teachers. The correlated trait-correlated method minus one model was applied, with adolescent ratings as the reference method, and the other two ratings as the non-reference methods. The findings indicated no additional variance in adolescent ratings for IA and HI that could not be accounted by mother ratings of IA and HI, respectively. In contrast, there was additional variance in adolescent ratings for IA and HI that could was not accounted by teacher ratings of IA and HI, respectively. The findings suggest that when diagnosing ADHD in adolescents, their reports of their own ADHD behaviors are not needed when mother reports of such behaviors are used.

Analysis of the Convergent and Discriminant Validity of the CBCL, TRF, and YSR in a Clinic-Referred Sample

This study used the correlated trait-correlated method minus one model to examine the convergent and discriminant validity of the Child Behavior Checklist (CBCL), Teacher's Report Form (TRF) and Youth Self-Report (YSR), based on ratings of clinic-referred adolescents. A total of 294 adolescents, comprising 70.5% males, provided self-ratings on the YSR. The adolescents were also rated by their mothers and teachers on the CBCL and TRF respectively. The findings indicated some support for convergent validity for the CBCL and TRF for Anxious/Depressed, Withdrawn/Depressed and Rule Breaking Behavior, and for the convergent validity for the CBCL and YSR for Rule Breaking Behavior. There was support for the discriminant validity between virtually all the traits. The findings are discussed in relation to the construct validity, cross-informant agreement, and clinical use of the CBCL, TRF and YSR.

Cross-informant symptoms from CBCL, TRF, and YSR: Trait and method variance in a normative sample of Russian youths.

A large community-based sample of Russian youths (n = 841, age M = 13.17 years, SD = 2.51) was assessed with the Child Behavior Checklist (mothers and fathers separately), Teacher's Report Form, and Youth Self-Report. The multiple indicator-version of the correlated trait-correlated method minus one, or CT-C(M - 1), model was applied to analyze (a) the convergent and divergent validity of these instruments in Russia, (b) the degree of trait-specificity of rater biases, and (c) potential predictors of rater-specific effects. As expected, based on the published results from different countries and in different languages, the convergent validity of the instruments was rather high between mother and father reports, but rather low for parent, teacher, and self-reports. For self- and teacher reports, rater-specific effects were related to age and gender of the children for some traits. These results, once again, attest to the importance of incorporating information from multiple observers when psychopathological traits are evaluated in children and adolescents.

Analyzing Multitrait-Multimethod Data

Assessing construct validity is a core task in psychology. Since Campbell and Fiske’s (1959) seminal article on multitrait-multimethod (MTMM) analysis, several different methodological approaches for the analysis of convergent and discriminant validity of MTMM data have been developed. In this article, two MTMM approaches are transferred to the general framework of confirmatory factor analysis and compared with the extended version of the correlated trait-correlated method minus one model ( Nussbeck, Eid, Geiser, Courvoisier, & Lischetzke, 2009 ): The multilevel MTMM model ( Maas, Lensvelt-Mulders, & Hox, 2009 ) and the three-mode model ( Oort, 2009 ). Assessing the construct validity of a German Big Five MTMM data set these three MTMM approaches are compared with regard to convergent and discriminant validity estimates and with regard to method effects. Advantages and limitations of each methodological approach will be discussed in detail.

A CTC(M−1) Model for Different Types of Raters

Many psychologists collect multitrait-multimethod (MTMM) data to assess the convergent and discriminant validity of psychological measures. In order to choose the most appropriate model, the types of methods applied have to be considered. It is shown how the combination of interchangeable and structurally different raters can be analyzed with an extension of the correlated trait-correlated method minus one [CTC(M−1)] model. This extension allows for disentangling individual rater biases (unique method effects) from shared rater biases (common method effects). The basic ideas of this model are presented and illustrated by an empirical example.

The Construct Validity of Acceptance: A Multitrait-Multimethod Investigation

Despite increasing clinical and empirical attention, the construct validity of acceptance has not been extensively investigated. The present study utilized a multitrait-multimethod design and a correlated trait-correlated method minus one [CT-C(M-1)] confirmatory factor analytic model to assess acceptance's convergent validity across methods and discriminant validity in comparison to cognitive reappraisal and perceived emotional control in a sample of 210 outpatients with anxiety and mood disorders. In addition, the study evaluated acceptance's concurrent validity by investigating the extent to which it was associated with variables of clinical interest over and above the two rival constructs. Results of confirmatory factor analyses supported acceptance's convergent and discriminant validity in comparison to the two neighboring constructs, and thereby provided partial support for its construct validity. However, contrary to prediction, acceptance was not significantly associated with concurrent validation measures. These results are discussed in the context of acceptance's potential therapeutic utility and functional relationships with associated constructs.