Multitrait-multimethod Matrix Approach Research Articles

Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Organizational task environment: evidence from the Brazilian industry

Purpose – Using the industry as an approximation of the external environment of companies, Dess and Beard proposed the construct organizational task environment (OTE). If the precision of the definition of industry is desirable, it involves a multiplicity of elements that restrict the ability of generalization. This paper aims to contribute by identifying clusters of industries with similar environmental profiles. As the discriminant validity of the OTE construct was tested only by Harris in US manufacturing industry, not supporting it, this study also seeks to contribute by assessing the OTE construct validity for the Brazilian manufacturing industry. Design/methodology/approach – Authors collected data from all manufacturing sectors in Brazil, between 1996 and 2003; they used confirmatory factor analysis with the multi-trait multi-method matrix approach to assess the construct validity and ran environmental dimensions' factorial scores through the cluster analysis to find out natural groupings of industries. Findings – The results support the convergent and discriminant validity of the construct OTE, suggesting that further replication should be conducted in the US economy and in different economic contexts. Research limitations/implications – The study identified four groups of industries with similar environmental conditions, increasing possibilities of generalization of researches. The limitations stem from measurement in an extended period of time and not measuring changes in the environment. Practical implications – To expand the analytical capabilities of managers for decision making on the sharing of skills between businesses in different industries. Originality/value – The main contributions of this work are to further discussions on the validity of the OTE construct and to identify industrial clusters of homogeneous environments.

Validity and reliability of the Behavior Problems Inventory, the Aberrant Behavior Checklist, and the Repetitive Behavior Scale – Revised among infants and toddlers at risk for intellectual or developmental disabilities: A multi-method assessment approach

Reliable and valid assessment of aberrant behaviors is essential in empirically verifying prevention and intervention for individuals with intellectual or developmental disabilities (IDD). Few instruments exist which assess behavior problems in infants. The current longitudinal study examined the performance of three behavior-rating scales for individuals with IDD that have been proven psychometrically sound in older populations: the Aberrant Behavior Checklist (ABC), the Behavior Problems Inventory (BPI-01), and the Repetitive Behavior Scale - Revised (RBS-R). Data were analyzed for 180 between six and 36 months old children at risk for IDD. Internal consistency (Cronbach's α) across the subscales of the three instruments was variable. Test-retest reliability of the three BPI-01 subscales ranged from .68 to .77 for frequency ratings and from .65 to .80 for severity ratings (intraclass correlation coefficients). Using a multitrait-multimethod matrix approach high levels of convergent and discriminant validity across the three instruments was found. As anticipated, there was considerable overlap in the information produced by the three instruments; however, each behavior-rating instrument also contributed unique information. Our findings support using all three scales in conjunction if possible.

A construct validity study of clinical competence: A multitrait multimethod matrix approach

The purpose of the study was to adduce evidence for estimating the construct validity of clinical competence measured through assessment instruments used for high-stakes examinations. Thirty-nine international physicians (mean age = 41 + 6.5 y) participated in high-stakes examination and 3-month supervised clinical practice to determine the practice readiness of physicians. Three traits-doctor-patient relationship, clinical competence, and communication skills-were assessed with objective structured clinical examinations, in-training evaluation reports, and clinical assessments. These traits were intercorrelated in a multitrait multimethod matrix (MTMM). The reliability of assessments ranged from moderate to high (Cronbach's alpha: 0.58-0.98; Ep(2) = 0.79). There is evidence for both convergent and divergent validity for clinical competence, followed by doctor-patient relationships, and communications (validity coefficients = 0.12-0.85). The correlations between the same methods but different traits indicate that there is substantial method specificity in the assessment accounting for nearly one-quarter of the variance (23.7%). There is evidence for the construct validity of all 3 traits across 3 methods. The MTMM approach, currently underutilized, could be used to estimate the degree of evidence for validating complex constructs, such as clinical competence.

Measurement of Dispositional Affect: Construct Validity and Convergence with a Circumplex Model of Affect

A circumplex model of affect has recently gained tentative acceptance by prominent researchers investigating the structure of dispositional affect. The present study examines the issue of construct validity of several existing measures of affect and their fit with the circumplex model. A traditional multitrait-multimethod matrix approach is utilized first, followed by a confirmatory factor analytic approach. These analyses demonstrate that the data collected using these four scales are characterized by generally good concurrent and discriminant validity. Furthermore, the data are in partial agreement with the proposed circumplex model of affect. Issues for the further development of the circumplex model and the use of these measures are discussed.

Convergent Discriminitive, and Predictive Validity of the Prostate Cancer Specific Quality of Life Instrument (PROSQOLI) Assessment and Comparison with Analogous Scales From the EORTC QLQ-C30 and a Trial-Specific Module

The Prostate Cancer Specific Quality of Life Instrument (PROSQOLI) is a measure of health-related quality of life (HRQL) that was designed to be an outcome measure for clinical trials in advanced hormone-resistant prostate cancer. The cross-sectional validity of the PROSQOLI was assessed using baseline data from a randomized trial in which HRQL was also assessed with the European Organisation for Research and Treatment of Cancer (EORTC) Core Quality of Life Questionnaire (QLQ-C30) and a trial-specific quality of life module (QLM-P14). Convergent validity was assessed with the multitrait-multimethod matrix approach; discriminative validity was assessed according to conventional clinical criteria; and predictive validity was assessed by the ability to predict survival duration. These assessments provided strong support for the validity of all PROSQOLI scales except those for family/marriage relationships and passing urine; modifications of these two scales are under evaluation. The strength, consistency, and independence of the prognostic information provided by the HRQL scales were striking. Differences between the instruments were generally subtle. These data support validity of the PROSQOLI and the analogous scales from the QLQ-C30 and QLM-P14 in symptomatic men with advanced hormone resistant prostate cancer. The PROSQOLI is a short, simple, and valid measure of HRQL in this setting.

The pediatric cancer quality of life inventory-32 (PCQL-32): I. Reliability and validity.

Multidimensional measurement of pediatric cancer patients' health-related quality of life (HRQOL) in Phase III randomized controlled clinical trials is being recognized increasingly as an essential component in evaluating the comprehensive health outcomes of modern antineoplastic treatment protocols. The Pediatric Cancer Quality of Life Inventory-32 (PCQL-32) is a standardized patient self-report and parent proxy-report assessment instrument designed to assess systematically pediatric cancer patients' HRQOL outcomes. To validate a patient-report form and a parent-report form, the PCQL-32 was administered to 291 pediatric cancer patients and to their parents. Both forms yield a total score and five a priori multidimensional scales. Clinical validity was determined by the known-groups approach by comparing patients classified as either on or off treatment. To determine construct validity, a battery of standardized psychosocial measures was administered and a multitrait-multimethod matrix was constructed. For both patient and parent forms, internal consistency reliability of the PCQL-32 total scale was high (0.91 and 0.92, respectively). The internal consistency reliabilities of the five component scales for both patient and parent forms were in the acceptable range for group comparisons. With regard to clinical validity, the PCQL-32 total scale and the disease/treatment and physical functioning scales of the PCQL-32 distinguished between patients on and off treatment for both patient- and parent-report. The results of the multitrait-multimethod matrix approach were consistent with hypotheses and lent evidence for the construct validity of the patient and parent forms of the PCQL-32 total scale and the psychological functioning, social functioning, cognitive functioning, physical functioning, and disease/treatment scales. The PCQL-32 has demonstrated acceptable internal consistency reliability, clinical validity, and construct validity for both patient-report and parent-report forms. Further field testing of the PCQL-32 will determine its practicality and utility in multisite pediatric cancer randomized controlled clinical trials.

Focus on psychometrics. The multitrait–multimethod approach to construct validity

The multitrait-multimethod matrix approach as proposed by Campbell and Fiske (1959) was an important contribution to our understanding of the nature of validation procedures. There are, however, problems encountered when using the Campbell and Fiske (1959) approach. The purpose of this article is to discuss the method and selected problems, and to propose an alternate approach to address those problems.

Validity and Structural Reliability of Multidimensional Scaling

The author examines the convergent and discriminant validity of multidimensional scaling (MDS) using the multitrait-multimethod matrix approach. An assessment also is made of the impact on MDS solutions due to embedding of stimuli in similar stimuli and due to change in stimulus domain introduced via different kinds of stimuli. The results indicate reasonable validity and that MDS solutions are fairly robust to the impact of embedding but less robust to changes in stimulus domain.

Validating the S-O-R paradigm for consumer involvement with a convenience good

The Houston and Rothschild S-O-R consumer involvement paradigm is tested for validity with measures of involvement for a shampoo purchase decision. A multitrait-multimethod matrix approach confirms the convergent and discriminant validity of the measure employed. A causal modeling analysis suggests that enduring involvement may work through situational involvement to influence consumer responses. Marketing implications are discussed.

Measuring Career Indecision Among College Students: Toward a Valid Approach for Counseling Practitioners and Researchers

Because of the need for validated instruments to measure career indecision, this study used a modified multitrait-multimethod matrix approach to examine the construct validity of the Career Decisio...

Validation of an S-O-R Model for Situation, Enduring, and Response Components of Involvement

Despite the importance of the concept of involvement, it remains misunderstood. No attempt has been made to establish the reliability and validity of the concept. To resolve apparently conflicting research findings, Houston and Rothschild posit a paradigm which classifies involvement as situational, enduring, and response. The author assesses the reliability and validity of this tripartite classification of involvement by using a multitrait-multimethod matrix approach and a linear structural relations analysis approach. Subsequently, the S-O-R formulation and causality are also tested.

A Scale to Measure Self-Concepts, Person Concepts, and Product Concepts

The author describes the construction of a scale to measure self-concepts, person concepts, and product concepts. Multivariate analysis and multidimensional scaling procedures are employed to develop a 15-item semantic differential scale. Measures of test-retest reliability are reported. Validity of the scale is assessed by the multitrait-multimethod matrix approach.

A multitrait-multimethod analysis of variance of teachers' ratings of aggression, hyperactivity, and inattention.

The convergent and discriminant validities of three teacher rating scale measures of the traits of hyperactivity, aggression, and inattention were explored, using the multitrait-multimethod matrix approach of Campbell and Fiske (1959), as well as an analysis of variance procedure (Stanley, 1961). In the present study teachers rated children from their elementary school classrooms on the above traits. The results provided strong evidence for convergent validity. Data also indicated that these traits can be reliable differentiated by teachers, suggesting that research aimed at better understanding the unique contributions of hyperactivity, aggression, and inattention is warranted. The respective benefits of analyzing multitrait-multimethod matrices by employing the ANOVA procedure or by using the Campbell and Fiske (1959) criteria were discussed.