Applying Generalizability Theory Research Articles

Interrater Reliability of Standardized Actors Versus Nonactors in a Simulation Based Assessment of Interprofessional Collaboration.

There is a need for reliable and practical interprofessional simulations that measure collaborative practice in outpatient/community scenarios where most health care takes place. The authors applied generalizability theory to examine reliability in an ambulatory care scenario using the following 2 trained observer groups: standardized patient (SP, actor) raters and those who received rater training alone (non-SPs). Twenty-one graduate health professions students participated as health care providers in an interprofessional care simulation involving an SP, caregiver, and clinicians. Six observers in each group received frame-of-reference training and rated aspects of collaborative care using a behavioral observation checklist. The authors examined sources of measurement variance using generalizability theory and extended this technique to statistically compare the rater types and compute reliability for subsets of raters. Standardized patient ratings were significantly more reliable than non-SPs' despite both groups receiving extensive rater training. A single SP was predicted to generate scores with a reliability of 0.74, whereas a single non-SP rater's scores were predicted at a reliability of 0.40. Removing each rater one by one from the full 6-member SP sample reduced reliability similarly for all raters (reliability, 0.86-0.89). However, removing individual raters from the full 6-member non-SP sample led to more variable reductions in reliability (0.58-0.72). Ongoing experience rating performance from within a particular simulation-based assessment may be a valuable rater characteristic and more effective than rater training alone. The extensions of reliability estimation introduced here can also be used to support more insightful reliability research and subsequent improvement of rater training and assessment protocols.

Applying generalizability theory to examine the antecedents of perceived coach support.

Although social support is integral to the coaching process, there is only a limited understanding of the antecedents of perceived coach support. We applied generalizability theory to examine perceived coach support and its antecedents at perceiver, provider, and relational levels of analysis. Two studies were conducted in which athletes rated the degree to which they identified with a selection of coaches, and the personality, competency, and supportiveness of those coaches. Univariate analyses demonstrated that the relational component accounted for a significant amount of variance in perceived coach support in both studies. Multivariate analyses demonstrated that when athletes perceive specific coaches to be highly agreeable, competent, and individuals with whom they share a common identity, they also perceive these same coaches to be particularly supportive in comparison with other coaches.

Assessing the Writing Achievement of Young Struggling Writers

This study examined the number of writing samples needed to obtain a reliable estimate of young struggling writers’ capabilities. It further assessed if performance in one genre was reflective of performance in other genres for these children. Second- and third-grade students (81 boys, 56 girls), who were identified as struggling writers in need of special assistance by their teacher and scored at the 25th percentile or lower on a norm-referenced story-writing test, wrote four compositions: a story, personal narrative, opinion essay, and informative text. Applying generalizability theory (G-theory), students’ scores on three writing measures (total number of words [TNW], vocabulary diversity, and writing quality) for the four compositions were each portioned into variance due to the following sources: students, writing tasks, and the interaction between students and writing tasks. We found that 14, 8, and 11 compositions, respectively, would be needed to obtain a reliable estimate of these students’ writing capabilities in terms of TNW, vocabulary diversity, and writing quality. Furthermore, how well these students wrote in one genre provided a weak prediction of how well they wrote in other genres.

A Generalizability Approach to the Measurement of Score Reliability of the Teacher Assessment Literacy Questionnaire

Classroom assessment is one of the main responsibilities of the teachers. Sound classroom assessment practices require teachers to have adequate levels of knowledge and skills in the educational assessment. The Teacher Assessment Literacy Questionnaire (TALQ) was developed by Plake and Impara (1992) to measure teachers’ knowledge and understanding of the basic principles of the educational assessment. This study applied generalizability theory to estimate the dependability of the TALQ’s scores for pre-service teachers (N = 259) enrolled in an educational measurement course at Sultan Qaboos University in Oman. Results showed that the scores are highly generalizable across items with relatively small item variance components. Implications are discussed in relation to the reliability theory.

Generalizability Analyses of NBDE Part II

This research applied generalizability theory to assess the effect of varying the number of cases and items nested within cases on generalizability of scores on Part II of the National Board Dental Examinations (NBDE Part II). In this research, sources of error were defined. Measurement conditions were classified. Error variances and generalizability coefficients for different conditions were computed. The data analyzed were the item responses of 1,535 candidates enrolled in accredited dental education programs who all took the same test form in 2007. Results showed that using more cases of fewer items might lead to a greater increase in generalizability than using more items per case. Other practical considerations such as time and cost constraints must be taken into account when applying the results of this research in other testing situations.

Analysis of Error Sources and Estimation of Reliability in a Analytic Evaluation on Writing Ability for Academic Purpose Korean by Applying Generalizability Theory.

Analysis of Error Sources and Estimation of Reliability in a Analytic Evaluation on Writing Ability for Academic Purpose Korean by Applying Generalizability Theory.

Analysis of Error Sources and Estimation of Reliability in a Korean Speaking Achievement Test by Applying Generalizability Theory

Analysis of Error Sources and Estimation of Reliability in a Korean Speaking Achievement Test by Applying Generalizability Theory

The efficiency of behavior rating scales to assess inattentive–overactive and oppositional–defiant behaviors: Applying generalizability theory to streamline assessment

Although the efficiency with which a wide range of behavioral data can be obtained makes behavior rating scales particularly attractive tools for the purposes of screening and evaluation, feasibility concerns arise in the context of formative assessment. Specifically, informant load, or the amount of time informants are asked to contribute to the assessment process, likely has a negative impact on the quality of data over time and the informant's willingness to participate. Two important determinants of informant load in progress monitoring are the length of the rating scale (i.e., the number of items) and how frequently informants are asked to provide ratings (i.e., the number of occasions). The purpose of the current study was to investigate the dependability of the IOWA Conners Teacher Rating Scale (Loney & Milich, 1982), which is used to differentiate inattentive–overactive from oppositional–defiant behaviors. Specifically, the facets of items and occasions were examined to identify combinations of these sources of error necessary to reach an acceptable level of dependability for both absolute and relative decisions. Results from D studies elucidated a variety of possible item–occasion combinations reaching the criteria for adequate dependability. Recommendations for research and practice are discussed.

일반화가능도 이론을 이용한 집단논리적사고력검사(GALT)의 신뢰도 분석

본 연구의 목적은 집단논리적사고력검사의 이용 목적에 따라 일반화가능도이론을 적용하여 문항과 피험자만을 고려한 단일국면의 오차원, 그리고 문항과 피험자, 그리고 영역을 고려한 다국면의 오차원을 분석하는 데 있다. 연구는 지방 소재 초 중 고등학생 총 1016명을 대상으로 이루어졌으며, 21문항의 GALT 완본을 40분 동안 실시하고, 이 중 축소본에 해당하는 12문항을 별도로 추출하여 일반화가능도이론을 이용한 신뢰도 분석에 이용하였다. 자료의 분석을 위해 일반화가능도이론을 적용하여 <TEX>$p{\times}i$</TEX>설계와 <TEX>$p{\times}(i:h)$</TEX>설계로 나누어 G 연구와 D 연구를 실시하였다. 분석결과는 다음과 같다. 첫째, 완본과 축소본을 <TEX>$p{\times}I$</TEX>설계로 D 연구를 수행한 결과 완본의 경우 21문항을 평가했을 때 0.87로 적정 수준의 일반화가능도 계수인 0.80을 상회하였으며, 13문항에서도 적정 수준의 일반화가능도 계수에 도달하였다. 축소본의 경우 12문항을 평가했을 때 0.77로 적정 수준의 일반화가능도 계수에 미치지 못하였으며, 최소 15문항 이상에서 신뢰도가 적정 수준에 도달하였다. 둘째, 축소본을 <TEX>$p{\times}(I:H)$</TEX>설계로 D 연구를 수행한 결과 6영역에 대해 영역별로 2문항씩 구성될 경우 0.71로 적정 수준의 일반화가능도계수인 0.80 보다 낮게 측정되었으며, 최소 영역별 5문항 이상에서 신뢰도가 적정 수준에 도달하였다. The purpose of this study lies in applying generalizability theory depending on the aim of the usage of GALT to analyze the sources of error of single-facet considering item and person only and to analyze the sources of error of multi-facet considering item, person and domain. The study was conducted with 1016 students of local elementary, middle, and high schools. The 21 items of a full version were answered for 40 minute and then the 12 items of short version were sampled to analyze reliability using generalizability theory. Both the full version and the short version of the items were analyzed using Cronbach's alpha for data analysis, and we applied generalizability theory and separate <TEX>$p{\times}i$</TEX> design and <TEX>$p{\times}(i:h)$</TEX> design, G study and D study were performed. Results of analysis are as follows: First, the result of D study after <TEX>$p{\times}I$</TEX> design both on the full version and the short version showed that in the case of the full version, the generalizability coefficient was 0.87 exceeding a normal level of 0.80, and the normal level of generalizability coefficient was achieved in 13 items as well. In case of short version, when 12 items were evaluated, generalizability coefficient was 0.77 not reaching the normal level, and the normal level was achieved in case of more than 15 items. Second, the result of D study after <TEX>$p{\times}(I:H)$</TEX> design on the short version showed that once one domain consists of 2 items in 6 domains, generalizability coefficient was 0.71 which is lower than the normal level of 0.80, the normal level was achieved in more than 5 item cases.

Applying Generalizability Theory to High-Stakes Objective Structured Clinical Examinations in a Naturalistic Environment

Purpose The purpose of this project was to determine if generalizability theory could be successfully applied to a high-stakes licensure objective structured clinical examination as part of its normal administrative procedures and whether the analysis could yield useful information with regard to sources of variance. Methods The anonymized data received from the Canadian Chiropractic Examining Board for its June 2005 Clinical Skills Examination were analyzed with generalizability theory. Variance components were estimated with SPSS 11.5 (SPSS Inc, Chicago, Ill) as partially nested data. The data included 182 candidates, 43 raters, 40 standardized patient actors, and 18 individual cases. Results Internal consistency estimates (Cronbach α) were .86 for day 1 and .91 for day 2. The α estimates for stations averaged .68 for day 1 and .74 for day 2. The generalizability-coefficient for the day 1 exam was .65 and for the day 2 was .42. G-coefficients for stations averaged .63 for day 1 and .74 for day 2. On day 1, the raters contributed 7% of the variance, and on day 2, the raters contributed 8%. Conclusions Generalizability theory can contribute to the understanding of sources of variance and provide direction for the improvement of individual stations. The size of the rater variance in a station may also indicate the need for increased training in that station or the need to make the scoring checklist more clear and definitive. Generalizability theory, however, must be cautiously applied, and it requires careful selection of the floating raters and vigorous training of the raters in each station.

Investigating the relative effects of persons, items, sections, and languages on TOEIC score dependability

This study applied generalizability theory to investigate the contributions of persons, items, sections, and language backgrounds to the score dependability of the Test of English for International Communication (TOEIC). I replicated and extended Brown’s (1999) study of the Test of English as a Foreign Language (TOEFL), using data from two language groups who took the same form of the TOEIC: 45 156 Japanese examinees and 46 067 Korean examinees. Both classical reliability analyses and generalizability analyses found the TOEIC scores of the examinees in question to be highly reliable/generalizable. The items within a section varied greatly in difficulty. An examinee who scored relatively high in one section was likely to score high in the other section. The findings regarding the effect of language backgrounds on TOEIC score dependability were inconclusive, which suggests directions for future research. TOEIC test developers might explore design alternatives with fewer items per section.

A validated, reliable method of scoring the severity of medication errors.

A method of scoring the severity of medication errors that does not require knowledge of patient outcomes was developed and tested. Thirty health care professionals from four U.K. hospitals scored 50 medication errors in terms of potential patient outcomes on a scale of 0 to 10, where 0 represented a case with no potential effect and 10 a case that would result in death. Sixteen error cases reported in the literature with actual patient outcomes were included among the cases to assess the validity of the scores. Ten of the errors were scored twice. The severity of the error cases, the occasion on which they were scored, the judge, each judge's profession, and the interactions between these were considered as potential sources of variability in scoring. The data were analyzed by applying generalizability theory to two models: one based on the 10 cases that were scored twice and ignoring the effect of differences in profession and one based on all 50 cases and ignoring the effect of the occasion of scoring. Generalizability coefficients for different numbers of judges and scoring occasions were calculated. A generalizability coefficient of 0.8 or more was considered to represent acceptable reliability. Most of the variance was attributable to differences in the cases. The analysis showed that, to achieve a generalizability coefficient of more than 0.8, at least four judges would have to score each case, each on one occasion, with the mean score used as a severity indicator. A reliable, valid method of scoring the severity of medication errors that did not require knowledge of patient outcomes was developed; at least four judges were required in order to achieve reliable scores, and reliability was not affected by the professions of the judges or the number of occasions on which the errors were scored.

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.