Multivariate Generalizability Theory Research Articles

Customizing Bayesian multivariate generalizability theory to mixed-format tests.

Mixed-format tests, which typically include dichotomous items and polytomously scored tasks, are employed to assess a wider range of knowledge and skills. Recent behavioral and educational studies have highlighted their practical importance and methodological developments, particularly within the context of multivariate generalizability theory. However, the diverse response types and complex designs of these tests pose significant analytical challenges when modeling data simultaneously. Current methods often struggle to yield reliable results, either due to the inappropriate treatment of different types of response data separately or the imposition of identical covariates across various response types. Moreover, there are few software packages or programs that offer customized solutions for modeling mixed-format tests, addressing these limitations. This tutorial provides a detailed example of using a Bayesian approach to model data collected from a mixed-format test, comprising multiple-choice questions and free-response tasks. The modeling was conducted using the Stan software within the R programming system, with Stan codes tailored to the structure of the test design, following the principles of multivariate generalizability theory. By further examining the effects of prior distributions in this example, this study demonstrates how the adaptability of Bayesian models to diverse test formats, coupled with their potential for nuanced analysis, can significantly advance the field of psychometric modeling.

Reliability of a workplace-based assessment for the United States general surgical trainees’ intraoperative performance using multivariate generalizability theory: a psychometric study

The System for Improving and Measuring Procedure Learning (SIMPL), a smartphone-based operative assessment application, was developed to assess the intraoperative performance of surgical residents. This study aims to examine the reliability of the SIMPL assessment and determine the optimal number of procedures for a reliable assessment. In this retrospective observational study, we analyzed data collected between 2015 and 2023 from 4,616 residents across 94 General Surgery Residency programs in the United States that utilized the SIMPL smartphone application. We employed multivariate generalizability theory and initially conducted generalizability studies to estimate the variance components associated with procedures. We then performed decision studies to estimate the reliability coefficient and the minimum number of procedures required for a reproducible assessment. We estimated that the reliability of the assessment of surgical trainees' intraoperative autonomy and performance using SIMPL exceeded 0.70. Additionally, the optimal number of procedures required for a reproducible assessment was 10, 17, 15, and 17 for postgraduate year (PGY) 2, PGY 3, PGY 4, and PGY 5, respectively. Notably, the study highlighted that the assessment of residents in their senior years necessitated a larger number of procedures compared to those in their junior years. The study demonstrated that the SIMPL assessment is reliably effective for evaluating the intraoperative performance of surgical trainees. Adjusting the number of procedures based on the trainees' training stage enhances the assessment process's accuracy and effectiveness.

Applying multivariate generalizability theory to psychological assessments.

Multivariate generalizability theory (GT) represents a comprehensive framework for quantifying score consistency, separating multiple sources contributing to measurement error, correcting correlation coefficients for such error, assessing subscale viability, and determining the best ways to change measurement procedures at different levels of score aggregation. Despite such desirable attributes, multivariate GT has rarely been applied when measuring psychological constructs and far less often than univariate techniques that are subsumed within that framework. Our purpose in this tutorial is to describe multivariate GT in a simple way and illustrate how it expands and complements univariate procedures. We begin with a review of univariate GT designs and illustrate how such designs serve as subcomponents of corresponding multivariate designs. Our empirical examples focus primarily on subscale and composite scores for objectively scored measures, but guidelines are provided for applying the same techniques to subjectively scored performance and clinical assessments. We also compare multivariate GT indices of score consistency and measurement error to those obtained using alternative GT-based procedures and across different software packages for analyzing multivariate GT designs. Our online supplemental materials include instruction, code, and output for common multivariate GT designs analyzed using mGENOVA and the gtheory, glmmTMB, lavaan, and related packages in R. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Supplemental Material for Applying Multivariate Generalizability Theory to Psychological Assessments

Supplemental Material for Applying Multivariate Generalizability Theory to Psychological Assessments

Analyzing Multivariate Generalizability Theory Designs within Structural Equation Modeling Frameworks

We demonstrate how to analyze complete multivariate generalizability theory (GT) designs within structural equation modeling frameworks that encompass both individual subscale scores and composites formed from those scores. Results from numerous analyses of observed scores obtained from respondents who completed the recently updated form of the Big Five Inventory (BFI-2) revealed that the lavaan SEM package in R produced results virtually identical to those obtained from the mGENOVA package, which historically has served as the gold standard for conducting multivariate GT analyses. We further extended lavaan analyses beyond what mGENOVA allows to produce Monte Carlo based confidence intervals for key GT parameters and correct score consistency and correlational indices for effects of scale coarseness characteristic of binary and ordinal data. Our comprehensive online Supplemental Material includes code for performing all illustrated analyses using lavaan and mGENOVA.

Examining the Scoring of Content Integration in a Listening-Speaking Test: A G-Theory Analysis

ABSTRACT Communication in the real world often entails the interpretation, evaluation, and integration of content from different sources. However, it appears that the ability to integrate content into discourse has not been explicitly scored for in existing studies. This study operationalizes content integration in the analytic scoring of a listening-speaking test in Chinese. International students who were non-native speakers of Chinese took the test that comprised two retell tasks and an oral presentation linked by an academic scenario. They were scored for content integration, organization, delivery, and language control for all three tasks. Multivariate generalizability theory (G-theory) was used to investigate the functioning of content integration in the analytic rubric of the retell tasks and oral presentation respectively. Overall, this study aimed to illuminate issues on dependability and construct validity for the two task types, focusing on content integration particularly. The findings suggested that content integration functioned differently to some extent when compared with the other dimensions studied.

Rater Reliability in Speaking Assessment in a Japanese Senior High School: Case of Classroom Group Discussion and Debate 日本の高校におけるスピーキング評価の採点者信頼性—教室内グループ型のディスカッションとディベートの場合

高校の授業内スピーキングテストにおいて、シンプルなルーブリックを用い、詳細な採点者トレーニングを行わない場合に採点者信頼性が十分確保できるかを、グループ型のディスカッションとディベートで検証した。227名の高校生の発話をそれぞれ教員2名で採点し、多相ラッシュ分析・一般化可能性理論等で分析した。その結果、採点者間一致度・一貫性と採点者内一貫性の観点で十分な信頼性が満たされていることが示された。グループ型タスクの場合に、ルーブリックにやり取りの適切さでなく言語面の観点を入れる、生徒が話す時間を長めに設定する、生徒の役割や発言する順番を決める、共通認識がある教員で採点を行う等の信頼性を高める方法とその問題点が示唆された。 Securing rater reliability for classroom speaking tests can be difficult because teacher-raters typically do not have much time to engage in rater training to understand and discuss rubrics and scores. Furthermore, a teacher typically faces difficulties asking colleagues to help double mark each student’s performance. Intensive rater training and double scoring are typical procedures to maintain high reliability (Knoch et al., 2021) but are not well practiced in the classroom. However, in some cases, extensive training or double scoring is not necessary when teachers use a rubric with a few criteria and levels, which is simpler than conventional detailed rubrics (Koizumi & Watanabe, 2021). Thus, we use a group discussion and a debate to explore rater reliability when Japanese senior high school teachers use simple analytic rubrics without detailed rater training. We pose the following research questions (RQs): RQ1: To what degree are raters similar in terms of interrater consensus and consistency? RQ2: To what degree do raters score students’ responses consistently? RQ3: How many raters are required to maintain reliability? We analyzed ratings for two speaking tests administrated in September or November to 227 third-year students at a public senior high school. Each test, taken by a group of four students, included either a five-minute group discussion or a 21-minute group debate; the test administration and marking were conducted during the lesson time. An analytic rubric was developed for each task and consisted of three or four criteria with three levels (e.g., content, expression, and technique). Two of the three raters scored each student’s response during the test. Teachers had no time to discuss the rubrics in detail and engaged in only a 10-minute discussion about the rubrics before the tests. The ratings were analyzed separately for each test using weighted kappa statistics, Spearman’s rank-order correlations, many-facet Rasch measurement (MFRM), and multivariate generalizability theory (mG theory). The results indicated that the overall rater reliability was adequate, but some cases required careful training. For RQ1, the kappa statistics of two raters’ scores for each criterion ranged from poor to substantial agreement (-.06 to .84). Correlations between two raters’ scores ranged from negligible to strong (-.07 to .91) and there were not large differences in rater severity (i.e., differences in fair mean-based average values of 0.07 to 0.16 with full marks of 3). In addition, the overall agreement percentages from MFRM were higher than those predicted by MFRM (e.g., 72.9% > 71.6%). The intrarater consistency examined for RQ2 using Infit and Outfit mean squares from MFRM was also adequate (e.g., 0.86 to 1.35). The number of raters needed to maintain sufficient reliability (Φ = .70) for RQ3 was one at the overall test levels and one to three at the criterion levels. Using simple rubrics, a group discussion task, and a debate task, the results showed that rater reliability can be maintained without extensive rater training. Although the current results may have been affected by study contexts, such as procedures and students’ and raters’ characteristics, they provide pedagogical and methodological implications for developing speaking assessment tasks and procedures and reporting rater reliability statistics from multiple perspectives.

Examining the Dependability and Practicality of Analytic Rubric of Summary Writing Using Multivariate Generalizability Theory: Focusing on Japanese University Students with Lower-Intermediate Proficiency in English

English teachers, especially those who teach summary writing to students with relatively lower proficiency in English face difficulty in teaching summary writing and while assessing their students’ performances. In the classroom context, an analytic rubric is pedagogically more helpful than a holistic rubric because the teacher can confirm the strengths and weaknesses of their students’ summary performance and the students can receive constructive feedback (Yamanishi et al., 2019). This study examined the practicality of the analytic rubric which consisted of four rating scales, including language use, by investigating seven in-service English teachers’ honest assessment of 160 summaries of Japanese private university students who are inexperienced in writing English summaries and have lower-intermediate proficiency of English. Furthermore, this study examined the dependability of the analytic rubric using multivariate generalizability theory (Brennan, 2001). The results showed that assessing language use and judging summaries, copied, to a lesser or greater extent, from the source text was difficult because of diverse linguistic errors and the use of paraphrasing was lacking. Therefore, it is necessary to define the gravity of language errors and that of copying in more detail to develop a rubric that suits to assess the summaries written by English learners with lower-intermediate level of English.

Examining Human and Automated Ratings of Elementary Students’ Writing Quality: A Multivariate Generalizability Theory Application

We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students ( n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3–5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.

Development and Validation of the Breast Cancer Scale QLICP-BR V2.0 Based on Classical Test Theory and Generalizability Theory.

ObjectiveThe aim of this study was to develop and validate the breast cancer scale among the system of quality-of-life instruments for cancer patients (QLICP-BR V2.0).MethodsProgrammed decision procedures and theories on instrument development were applied to develop QLICP-BR V2.0. A total of 246 breast cancer inpatients were investigated using QLICP-BR V2.0 from hospital admission until discharge. The reliability, validity, and responsiveness of the QLICP-BR V2.0 scale were evaluated by using the classical test theory combined with the generalizability theory (GT), including correlation analysis, multi-trait scaling analysis, factor analyses, t-tests, and also multivariate generalizability theory analysis.ResultsThe test–retest reliability of the total scale is 0.79, the Cronbach coefficient is 0.85, and the intra-class correlations coefficient is 0.88. The item–domain correlation analysis showed that the correlation coefficient between items and their own domain is greater than that with other domains except of item GSO4. The exploratory factor analysis showed that three principal components are obtained in the specific module. The outcome of the factor analysis coincides substantially with our theoretical conception. The score difference of each domain of the scale and the total scale before and after treatment is statistically significant (P < 0.05), with the standardized response mean of the total scale being 0.61. According to GT, the generalization coefficient of the scores in the 5 domains is between 0.626 and 0.768, and the reliability index is between 0.557 and 0.695.ConclusionQLICP-BR V2.0 exhibited reasonable degrees of validity, reliability, and responsiveness according to classical test and the generalizability theory. The number of items in the scale is appropriate.

Generalizability of Writing Scores and Language Program Placement Decisions: Score Dependability, Task Variability, and Score Profiles on an ESL Placement Test

Second Language (L2) testing has increasingly relied on performance assessment to evaluate practical command of language acquired. However, such forms of assessment entail more complex task design and subjective human scoring judgment (Bachman, 2004), raising challenges for score dependability and score use due to variability associated with task design (Deville &amp; Chalhoub-Deville, 2006; In’nami, &amp; Koizumi, 2016), differences in rater behavior (Bachman, Lynch, &amp; Mason, 1995), and rating rubric functionality, especially when consisting of multiple subscales (Grabowski &amp; Lin, 2019; Sawaki, 2007, Xi, 2007). The current study illustrates the use of Multivariate Generalizability Theory (MG-Theory) analyses for examining score variability and dependability for written performance assessment on an ESL placement test, rated using an analytic rubric with three subscales. In particular, this study identified the presence of task-related variability that did reduce score dependability for the writing scores yielded from this test. By the same token, this variability could substantively be justified as an artifact of representing the construct of L2 writing ability in a sufficiently broad manner. Simply said, should we expect test takers to have equivalent levels of proficiency when writing a review of an experience as a customer and when writing an argumentative essay as a student?

A Generalizability Analysis of the Meaning in Life Questionnaire for Chinese Adolescents

The level of meaning in life not only affects the physical health of individuals, but also is closely related to their mental health. At present, many self-reported questionnaires are being used to measure the meaning in life of Chinese adolescents. Using the multivariate generalizability theory, this study investigated the psychometric properties and the internal structure of the Meaning in Life Questionnaires (MLQs), the most widely used questionnaire for assessing the level of meaning in life of Chinese adolescents. The data were sample of 1,951 junior high school students from Guizhou, China. Multivariate random measurement mode p × i° is the primary analytic approach. Results showed that the generalizability coefficient and dependability index of the scale were 0.86 and 0.85, respectively. The generalizability coefficients of presence of meaning and search for meaning were 0.76 and 0.85, respectively, and the dependability indexes were 0.75 and 0.85 for MLQ-P and MLQ-S, respectively. The design of each factor for MLQ is reasonable in terms of score ratio and the number of projects. In brief, the reliability and factor structure of the scale are satisfactory.

Extended Multivariate Generalizability Theory With Complex Design Structures.

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and free-response items, with the latter involving variability attributable to both items and raters. In this case, two distinct designs are needed to fully characterize the design and capture potential sources of error associated with each item format. Another example involves tests containing both testlets and one or more stand-alone sets of items. Testlet effects need to be taken into account for the testlet-based items, but not the stand-alone sets of items. This article presents an extension of MGT that faithfully models such complex test designs, along with two real-data examples. Among other things, these examples illustrate that estimates of error variance, error-tolerance ratios, and reliability-like coefficients can be biased if there is a mismatch between the user-specified universe of generalization and the complex nature of the test.

Initial Validation Evidence for Clinical Case Presentations by Student Pharmacists.

There is a paucity of validation evidence for assessing clinical case-presentations by Doctor of Pharmacy (PharmD) students. Within Kane's Framework for Validation, evidence for inferences of scoring and generalization should be generated first. Thus, our objectives were to characterize and improve scoring, as well as build initial generalization evidence, in order to provide validation evidence for performance-based assessment of clinical case-presentations. Third-year PharmD students worked up patient-cases from a local hospital. Students orally presented and defended their therapeutic care-plan to pharmacist preceptors (evaluators) and fellow students. Evaluators scored each presentation using an 11-item instrument with a 6-point rating-scale. In addition, evaluators scored a global-item with a 4-point rating-scale. Rasch Measurement was used for scoring analysis, while Generalizability Theory was used for generalization analysis. Thirty students each presented five cases that were evaluated by 15 preceptors using an 11-item instrument. Using Rasch Measurement, the 11-item instrument's 6-point rating-scale did not work; it only worked once collapsed to a 4-point rating-scale. This revised 11-item instrument also showed redundancy. Alternatively, the global-item performed reasonably on its own. Using multivariate Generalizability Theory, the g-coefficient (reliability) for the series of five case-presentations was 0.76 with the 11-item instrument, and 0.78 with the global-item. Reliability was largely dependent on multiple case-presentations and, to a lesser extent, the number of evaluators per case-presentation. Our pilot results confirm that scoring should be simple (scale and instrument). More specifically, the longer 11-item instrument measured but had redundancy, whereas the single global-item provided measurement over multiple case-presentations. Further, acceptable reliability can be balanced between more/fewer case-presentations and using more/fewer evaluators.

Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R.

Multivariate generalizability theory (mG-theory) is an important framework in many behavioral and educational studies, as it describes useful psychometric properties of multidimensional assessments. Nevertheless, the use of mG-theory estimation is limited due to the lack of available software for carrying out the necessary calculations: users rely heavily on independent software programs such as mGENOVA and the BUGS/JAGS suite of programs. Considering the prevalence of R software, this paper presents a solution using the glmmTMB package to accomplish the estimation task. Users adopting the proposed method may find it more convenient for conducting both applied investigation and simulation studies without the need to switch between different software programs.

Indices of Subscore Utility for Individuals and Subgroups Based on Multivariate Generalizability Theory.

Conventional methods for evaluating the utility of subscores rely on traditional indices of reliability and on correlations among subscores. One limitation of correlational methods is that they do not explicitly consider variation in subtest means. An exception is an index of score profile reliability designated as , which quantifies the ratio of true score profile variance to observed score profile variance. has been shown to be more sensitive than correlational methods to group differences in score profile utility. However, it is a group average, representing the expected value over a population of examinees. Just as score reliability varies across individuals and subgroups, one can expect that the reliability of score profiles will vary across examinees. This article proposes two conditional indices of score profile utility grounded in multivariate generalizability theory. The first is based on the ratio of observed profile variance to the profile variance that can be attributed to random error. The second quantifies the proportion of observed variability in a score profile that can be attributed to true score profile variance. The article describes the indices, illustrates their use with two empirical examples, and evaluates their properties with simulated data. The results suggest that the proposed estimators of profile error variance are consistent with the known error in simulated score profiles and that they provide information beyond that provided by traditional measures of subscore utility. The simulation study suggests that artificially large values of the indices could occur for about 5% to 8% of examinees. The article concludes by suggesting possible applications of the indices and discusses avenues for further research.

A Generalizability Analysis of the Mobile Phone Addiction Tendency Scale for Chinese College Students.

College students’ mobile phone addiction is negatively associated with physical and mental health and academic performance. Many self-made questionnaires are currently being administered to Chinese college students to evaluate the mobile phone addiction tendency. Using the univariate generalizability theory and multivariate generalizability theory, this study investigated the psychometric properties and the internal structure of the Mobile Phone Addiction Tendency Scale (MPATS), the most widely used survey questionnaire assessing the status of Chinese college students’ mobile addiction. Data were a sample of 1,253 college students from the southwest of China. Primary analytic approaches included the generalizability design of univariate random measurement mode p × (i:h) and multivariate random measurement mode p˙ × i°. Results showed that the variance component of the participants and the variation related to the participants explained most of the variation of the scale, while the variance component of the items was small, and the generalizability coefficient and dependability index of the scale were 0.88 and 0.85. In the multivariate generalizability analysis, the variance component of the participants and the variation related to the participants accounted for most of the variation of the scale and the variance component of the items was small. The generalizability coefficients of withdrawal symptoms, salience, social comfort, and mood changes were 0.64–0.80, and the dependability indexes were 0.63–0.77. However, the generalizability coefficient and reliability index of universe score were 0.91 and 0.90. In addition, the contribution ratio of the four dimensions to the universe score variance was different from the assignment intention of the initial scale. Recommendations were discussed on the improvement of the test reliability for each dimension.

A modification of the Schutte Emotional Intelligence Scale using multivariate generalizability theory

We modified the Schutte Emotional Intelligence Scale (EIS) using multivariate generalizability theory (MGT), a measurement system in which the focus is on multiple universe scores within a specific generalization universe. Application of MGT comprises conducting a generalizability study to analyze and estimate the associations among various factors, the number of factors, and the initial measurement design process, as well as a decision study to explore changes in the generalized coefficient and reliability indices when the number of items in the measure is changed. We conducted a generalizability study with 210 students from Shanghai and Anhui Province China, and performed item analysis and exploratory factor analysis, then modified the scale using MGT to create a modified edition of the EIS and tested item and factor reliability and validity with another 1,339 students from Shanghai and Anhui Province. Results revealed better reliability and validity than the original EIS, with each factor making a more balanced contribution to the total variance. The modified Chinese EIS will be useful for future emotional intelligence research.

A Multivariate Generalizability Theory Approach to College Students' Evaluation of Teaching.

Teachers' teaching level evaluation is an important component in classroom teaching and professional promotion in the institutions of higher learning in China. Many self-made questionnaires are currently being administered to Chinese college students to evaluate teachers' classroom teaching performance. Quite often, due to the absence of strong educational, and psychological measurements and theoretical foundations for these questionnaires, their dependability remains open to doubt. Evaluation time points, the number of students, major type, and curriculum type were examined in relation to college students' perceptions on their teachers' classroom teaching performance, using Teachers' Teaching Level Evaluation Scale for Colleges (TTLES-C). Data were collected in a sample of 556 students at two time points from three Chinese universities and were analyzed using multivariate generalizability theory. Results showed that evaluations at the beginning of the spring semester produced better outcomes than did evaluations at the end of the fall semester, and 20 student evaluators were sufficient to ensure good dependability. Results also revealed that the evaluation dependability of science curriculum appeared higher than that of liberal arts curriculum. Recommendations were discussed on the evaluation criteria and mode.

The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores.

Conventional methods for evaluating the utility of subscores rely on reliability and correlation coefficients. However, correlations can overlook a notable source of variability: variation in subtest means/difficulties. Brennan introduced a reliability index for score profiles based on multivariate generalizability theory, designated as , which is sensitive to variation in subtest difficulty. However, there has been little, if any, research evaluating the properties of this index. A series of simulation experiments, as well as analyses of real data, were conducted to investigate under various conditions of subtest reliability, subtest correlations, and variability in subtest means. Three pilot studies evaluated in the context of a single group of examinees. Results of the pilots indicated that indices were typically low; across the 108 experimental conditions, ranged from .23 to .86, with an overall mean of 0.63. The findings were consistent with previous research, indicating that subscores often do not have interpretive value. Importantly, there were many conditions for which the correlation-based method known as proportion reduction in mean-square error (PRMSE; Haberman, 2006) indicated that subscores were worth reporting, but for which values of fell into the .50s, .60s, and .70s. The main study investigated within the context of score profiles for examinee subgroups. Again, not only indices were generally low, but it was also found that can be sensitive to subgroup differences when PRMSE is not. Analyses of real data and subsequent discussion address how can supplement PRMSE for characterizing the quality of subscores.