Generalizability Analysis Research Articles

Symbolic meanings of sex in relationships: Developing the Meanings of Sexual Behavior Inventory.

Consistent with symbolic interactionism and motivation research, the study explored the meanings of sexual behavior in romantic relationships in a sample of 3,003 online respondents. Starting with a pool of 104 respondent-generated items, Exploratory and Confirmatory Factor analyses in separate sample halves revealed a stable set of 9 dimensions within that item pool that formed 2 higher-order factors representing positive (to share pleasure, to bond, to de-stress, to energize the relationship, to learn more about each other) and negative (to manage conflict, as an incentive, to express anger, and to control partner) meanings of sexual behavior within relationships. Item Response Theory analyses helped select the 4-5 most effective items of each dimension for inclusion in the Meanings of Sexual Behavior Inventory (MoSBI). Generalizability analyses suggested that the MoSBI subscale scores continued to show high levels of internal consistency across a broad range of demographic subgroups (e.g., racial/ethnic groups, gay and lesbian respondents, and various levels of education). The MoSBI subscales demonstrated moderate and distinct patterns of association with a range of conceptual boundary scales (e.g., relationship and sexual satisfaction, emotional support, negative conflict behavior, and frequency of sexual behavior) suggesting that these scales represent novel relationship processes. Consistent with this, analyses in the 862 respondents completing a 2-month follow-up assessment suggested that the meanings of sexual behavior predicted residual change in relationship satisfaction, even after controlling for frequency of sexual behavior within the relationships. Implications are discussed. (PsycINFO Database Record

Toward a better judgment of item relevance in progress testing

BackgroundItems must be relevant to ensure item quality and test validity. Since “item relevance” has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement.MethodsMembers of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group.ResultsThe proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC.ConclusionsUse of the rubric results in a stricter evaluation of items’ appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.

Multivariate generalizability analysis of automated scoring for short answer items of social studies in large-scale assessment

With increased use of constructed response items in large scale assessments, the cost of scoring has been a major consideration (Noh et al. in KICE Report RRE 2012-6, 2012; Wainer and Thissen in Applied Measurement in Education 6:103–118, 1993). In response to the scoring cost issues, various forms of automated system for scoring constructed response items have been developed and used. The purpose of this research is to provide a comprehensive analysis for the generalizability of automated scoring results and compare it to that of scores produced by human raters. The results of this study provide evidence supporting the argument that the automated scoring system offers outcomes nearly as reliable as those produced by human scoring. Based on these findings, the automated scoring system appears to be a promising alternative to human scoring particularly for short factual answer items.

El análisis de la influencia del tiempo de juego y la puntuación parcial sobre el comportamiento táctico en el ataque en balonmano, utilizando la técnica de coordenadas polares

&lt;p&gt;&lt;span style="font-size: medium;"&gt;Different studies have referred that the playing time and partial score, as well as the interaction between these two influences the behaviors of players and teams. With this study we intended to realize the complex reality of tactical behavior in the attack in Men´s European Cup, by detecting behavior patterns and linking them to playing time and partial score.&lt;/span&gt;&lt;/p&gt;&lt;p&gt;&lt;span style="font-size: medium;"&gt;We used the observational methodology and for data analysis we used the polar coordinates. An instrument of observation has been constructed &lt;em&gt;ad hoc&lt;/em&gt; and validated. The tool optimization included the quality data control, using the analysis of intra and inter observer reliability by Cohen's Kappa index (range between .85 and 1), beyond the application of generalizability analysis (G indexes are suitable for all models estimated) confirming the instrument reliability. The sample contained all of 2&lt;em&gt;vs&lt;/em&gt;2 situations that occurred during the organized offensive process in numerical equality 6&lt;em&gt;vs&lt;/em&gt;6, in sixteen games (n = 390). &lt;/span&gt;&lt;/p&gt;&lt;p&gt;&lt;span style="font-size: medium;"&gt;As main results we can refer that the playing time influenced the decision-making in relation to the tactical means used, excepting the last ten minutes, where no patterns were found. The results also show that there are fewer behavior patterns when the score is a draw.&lt;/span&gt;&lt;/p&gt;

Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context

Aim: Many instruments for evaluating clinical teaching have been developed, albeit most in Western countries. This study aims to develop a validated cultural and local context sensitive instrument for clinical teachers in an East Asian setting (Japan), Japanese Clinical Teacher Evaluation Sheet (JaCTES).Methods: A multicenter, cross-sectional evaluation study was conducted. We collected a total of 1368 questionnaires on 304 clinical teachers, completed by residents in 16 teaching hospitals. The construct validity was examined by conducting a factor analysis and using structural equation modeling (SEM). We also assessed the reliability using generalizability analysis and decision study.Results: Exploratory factor analysis resulted in three-factor (role model, teaching activities, and accessibility) model including 18 items. Confirmatory factor analysis was performed, using SEM. The comparative fit index was 0.931 and the root mean square error of approximation was 0.087, meaning an acceptable goodness of fit for this model. To obtain a reliable dependability-coefficient of at least 0.70 or higher, 5–8 resident responses are necessary.Discussion and conclusion: JaCTES is the first reported instrument with validity evidence of content and internal structure and high feasibility in Japan, an East Asian setting. Medical educators should be aware of the local context and cultural aspects in evaluating clinical teachers.

Investigating a self-scoring interview simulation for learning and assessment in the medical consultation.

Experience with simulated patients supports undergraduate learning of medical consultation skills. Adaptive simulations are being introduced into this environment. The authors investigate whether it can underpin valid and reliable assessment by conducting a generalizability analysis using IT data analytics from the interaction of medical students (in psychiatry) with adaptive simulations to explore the feasibility of adaptive simulations for supporting automated learning and assessment. The generalizability (G) study was focused on two clinically relevant variables: clinical decision points and communication skills. While the G study on the communication skills score yielded low levels of true score variance, the results produced by the decision points, indicating clinical decision-making and confirming user knowledge of the process of the Calgary–Cambridge model of consultation, produced reliability levels similar to what might be expected with rater-based scoring. The findings indicate that adaptive simulations have potential as a teaching and assessment tool for medical consultations.

A Generalizability Analysis of the Reliability of Measurements: "An Example of Cell Division and Heredity Unit"

The purpose of the study is to measure students' performance through different measurement tools and compare the findings through G Theory in order to identify the errors associated with the raters and items to improve the future applications. The sample consisted of 48 eighth graders in Kars. Two different types of exams (a multiple choice test and an essay) were applied and essays were graded by three raters. G and K analyses were performed on the results. According to the findings, the error rate was higher for the essays in comparison to multiple-choice test. The mean score was higher for the multiple-choice test, the variances were similar. There were no differences among the essay scores given by different raters. Findings of decision study indicated Student facet as the main source of the variation in the data for both types of the measurements.

Measuring Harm in Health Care: Optimizing Adverse Event Review.

The objective of this study was to identify modifiable factors that improve the reliability of ratings of severity of health care-associated harm in clinical practice improvement and research. A diverse group of clinicians rated 8 types of adverse events: blood product, device or medical/surgical supply, fall, health care-associated infection, medication, perinatal, pressure ulcer, surgery. We used a generalizability theory framework to estimate the impact of number of raters, rater experience, and rater provider type on reliability. Pharmacists were slightly more precise and consistent in their ratings than either physicians or nurses. For example, to achieve high reliability of 0.83, 3 physicians could be replaced by 2 pharmacists without loss in precision of measurement. If only 1 rater was available for rating, ∼5% of the reviews for severe harm would have been incorrectly categorized. Reliability was greatly improved with 2 reviewers. We identified factors that influence the reliability of clinician reviews of health care-associated harm. Our novel use of generalizability analyses improved our understanding of how differences affect reliability. This approach was useful in optimizing resource utilization when selecting raters to assess harm and may have similar applications in other settings in health care.

수학적 창의성 태도 검사에서 수학영재와 일반학생의 다집단 일반화가능도 분석

본 연구의 목적은 측정학적 이론을 바탕으로 수학영재 집단과 일반학생 집단에서 수학적 창의성 태도 검사 점수에 영향을 미치는 오차 요인들의 상대적인 영향력과 적정 수준의 신뢰도에 도달하는 효율적인 측정 조건을 탐색하는 데 있다. 이를 위해 109명의 중등 수학영재와 125명의 일반 중학생을 대상으로 실시한 수학적 창의성 태도 검사 결과에 다변량 일반화가능도 분석을 수행하였다. 주요 분석 결과는 다음과 같다. 첫째, 수학적 창의성 태도 검사는 신뢰도를 기준으로는 일반학생 집단에서, 조건부 측정 오차를 기준으로는 수학영재 집단에서 좀 더 적합한 것으로 나타났지만, 두 집단에서 모두 시행할 수 있는 신뢰로운 측정도구인 것으로 나타났다. 둘째, 수학적 창의성 태도 검사에서 신뢰도를 높이기 위해서는 수학영재 집단의 경우 수렴적 태도를 높게. 반면에 일반학생 집단의 경우 발산적 태도와 문제해결 태도를 높게 반영하여야 하는 것으로 나타났다. 셋째, 수학영재 집단의 경우 발산적 태도, 문제해결 태도, 수렴적 태도의 다차원적인 요소를 반영하는 경우, 그리고 일반학생 집단의 경우 발산적 태도만으로도 적정 수준의 신뢰도에 도달하는 것으로 나타났다. 마지막으로 이러한 연구결과들을 바탕으로 수학적 창의성 태도 검사 활용 방안 및 향후 연구 방향을 제시하였다. The purpose of this study was to investigate the relative influence of multiple error sources and to find optimal measurement conditions that obtain a desired level of reliability of a creative attitude test in mathematical creativity. This study analyzed the scores of the Creative Attitude Scale-Korea allowed to access publicly of 125 general students and 109 mathematically gifted students by performing a multivariate generalizability analysis. The main results were as follows. First, based on reliability, the Creative Attitude Scale-Korea was measured less precisely for mathematically gifted students. On the contrary, based on the conditional standard error of measurement, it was measured less precisely for general students. However, the Creative Attitude Scale-Korea showed strong reliability in both groups. Second, the optimal weights should adjust to .3, .3, .4 in mathematically gifted students and .4, .4, .2 in general students with three scoring components of divergent attitude, problem solving attitude, and convergent attitude based on the maximum reliability. Third, to approach desirable reliability, it is possible to use one component of divergent attitude in general students but three components of divergent attitude, problem solving attitude, and convergent attitude in mathematically gifted students. Finally this study proposed application plans for the Creative Attitude Scale-Korea and future directions of research.

Generalizability analysis for clinical trials: a simulation study.

Subjects are rarely selected on a random basis from a well-defined patient population of interest into a clinical trial, with women, children, the elderly, and those with common comorbidities who are frequently underrepresented. Decades of clinical experience have demonstrated that the application of trial findings to individual patients is permissible by using efficacy as a measure of effectiveness and assuming that the characteristics of patients are sufficiently similar. In order to investigate this issue in greater depth, we simulated a patient population with treatment effect size of 0.5 (Cohen's d) and five covariates that included gender, health insurance, comorbidity, age, and motivation. To demonstrate how selection of patients for a clinical trial can bias the results when treatment effect varies across individuals, we created 50 nonrandom clinical trials based on this patient population and showed relative bias to range from 1.68% to 99.70%. We calculated and evaluated three indexes: C-statistics, standardized mean difference (SMD), and Tipton's index (β) of generalization for the 50 nonrandom trials. Findings indicated that (i) the ranges were 0.56-0.98, 0.23-11.17, and 0.99-0.73 for C-statistics, SMD, and β, respectively, when treatment effect bias increased from 1.68% to 99.70% and (ii) C-statistics < 0.86, SMD < 1.95, and β > 0.91 when treatment effect bias <50%. Recommendations are made using existing generalization indexes on the basis of our simulation results. An example from a real clinical trial is provided for illustration. Copyright © 2017 John Wiley & Sons, Ltd.

Measuring Mindfulness: Applying Generalizability Theory to Distinguish between State and Trait

Mindfulness can be conceptualized as either a state or a trait, but currently, there is no reliable psychometric method to distinguish clearly between the two in psychological measures. Notwithstanding the clinical effectiveness of mindfulness, any specific element of mindfulness treatment can only be evaluated by comparing state and trait changes using techniques that allow such changes to be measured. Generalizability Theory (GT) is a suitable method to differentiate between state and trait variance components, and its application is illustrated here with an empirical example using the Toronto Mindfulness Scale (TMS). Person × occasion interaction is a marker of individual state changes and should explain the largest amount of variance in a valid state measure. To assess state variability, data were collected on three separate occasions: (i) after a holiday, (ii) immediately after a mindfulness exercise, and (iii) before a stressful event (i.e., exam). Generalizability analysis was applied to examine sources of true and error variances. The TMS captured a larger amount of variance attributed to a state and only a small amount associated with trait mindfulness, which is consistent with the purpose of the measure. This study has demonstrated that Generalizability Theory can be usefully applied to distinguish between state and trait components in a measure, and it is recommended as an appropriate psychometric method to validate state and trait measurement tools. These findings have far-reaching implications to improve the accuracy of the distinction between state and trait in mindfulness measurement and other areas of psychological assessment.

Reliability of a viva assessment of clinical reasoning in an Australian pre-professional osteopathy program assessed using generalizability theory.

Clinical reasoning is situation-dependent and case-specific; therefore, assessments incorporating different patient presentations are warranted. The present study aimed to determine the reliability of a multi-station case-based viva assessment of clinical reasoning in an Australian pre-registration osteopathy program using generalizability theory. Students (from years 4 and 5) and examiners were recruited from the osteopathy program at Southern Cross University, Lismore, Australia. The study took place on a single day in the student teaching clinic. Examiners were trained before the examination. Students were allocated to 1 of 3 rounds consisting of 5 10-minute stations in an objective structured clinical examination-style. Generalizability analysis was used to explore the reliability of the examination. Fifteen students and 5 faculty members participated in the study. The examination produced a generalizability coefficient of 0.53, with 18 stations required to achieve a generalizability coefficient of 0.80. The reliability estimations were acceptable and the psychometric findings related to the marking rubric and overall scores were acceptable; however, further work is required in examiner training and ensuring consistent case difficulty to improve the reliability of the examination.

Redesign of the System for Evaluation of Teaching Qualities in Anesthesiology Residency Training (SETQ Smart)

Given the increasing international recognition of clinical teaching as a competency and regulation of residency training, evaluation of anesthesiology faculty teaching is needed. The System for Evaluating Teaching Qualities (SETQ) Smart questionnaires were developed for assessing teaching performance of faculty in residency training programs in different countries. This study investigated (1) the structure, (2) the psychometric qualities of the new tools, and (3) the number of residents' evaluations needed per anesthesiology faculty to use the instruments reliably. Two SETQ Smart questionnaires-for faculty self-evaluation and for resident evaluation of faculty-were developed. A multicenter survey was conducted among 399 anesthesiology faculty and 430 residents in six countries. Statistical analyses included exploratory factor analysis, reliability analysis using Cronbach α, item-total scale correlations, interscale correlations, comparison of composite scales to global ratings, and generalizability analysis to assess residents' evaluations needed per faculty. In total, 240 residents completed 1,622 evaluations of 247 faculty. The SETQ Smart questionnaires revealed six teaching qualities consisting of 25 items. Cronbach α's were very high (greater than 0.95) for the overall SETQ Smart questionnaires and high (greater than 0.80) for the separate teaching qualities. Interscale correlations were all within the acceptable range of moderate correlation. Overall, questionnaire and scale scores correlated moderately to highly with the global ratings. For reliable feedback to individual faculty, three to five resident evaluations are needed. The first internationally piloted questionnaires for evaluating individual anesthesiology faculty teaching performance can be reliably, validly, and feasibly used for formative purposes in residency training.

Daily Encounter Cards-Evaluating the Quality of Documented Assessments.

Concerns over the quality of work-based assessment (WBA) completion has resulted in faculty development and rater training initiatives. Daily encounter cards (DECs) are a common form of WBA used in ambulatory care and shift work settings. A tool is needed to evaluate initiatives aimed at improving the quality of completion of this widely used form of WBA. The completed clinical evaluation report rating (CCERR) was designed to provide a measure of the quality of documented assessments on in-training evaluation reports. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Six experts in resident assessment grouped 60 DECs into 3 quality categories (high, average, and poor) based on how informative each DEC was for reporting judgments of the resident's performance. Eight supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared to determine if the CCERR could discriminate based on DEC quality. Statistically significant differences in CCERR scores were observed between all quality groups (P < .001). A generalizability analysis demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. The CCERR is a reliable and valid tool to evaluate DEC quality. It can serve as an outcome measure for studying interventions targeted at improving the quality of assessments documented on DECs.

Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework.

Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework.

Observing instructional quality in the context of school evaluation

Observational measures of instructional quality are used in both research and evaluation contexts. In European school evaluation systems, one main intended use of classroom observation results is to inform schools' instructional improvement efforts. This study examined the quality of such diagnostic information in the context of a school evaluation system in Germany. The factor analytic results indicate that the empirical structure of the observation instrument lacks correspondence with its original normative model but does mirror a five-factor model based on recent literature, including classroom management, two different aspects of student orientation, cognitive activation, and classroom assessment. The generalizability analyses show that both indicators and observers represent important sources of measurement error. Additionally, their magnitude is larger for shorter observations (lesson segments). At least two, sometimes three, observers and between five and ten indicators per dimension are needed to achieve sufficiently reliable results, but results importantly depend on which dimension of instructional quality is being assessed.

Using generalizability analysis to estimate parameters for anatomy assessments: A multi-institutional study.

With integrated curricula and multidisciplinary assessments becoming more prevalent in medical education, there is a continued need for educational research to explore the advantages, consequences, and challenges of integration practices. This retrospective analysis investigated the number of items needed to reliably assess anatomical knowledge in the context of gross anatomy and histology. A generalizability analysis was conducted on gross anatomy and histology written and practical examination items that were administered in a discipline-based format at Indiana University School of Medicine and in an integrated fashion at the University of Alabama School of Medicine and Rush University Medical College. Examination items were analyzed using a partially nested design s×(i:o) in which items were nested within occasions (i:o) and crossed with students (s). A reliability standard of 0.80 was used to determine the minimum number of items needed across examinations (occasions) to make reliable and informed decisions about students' competence in anatomical knowledge. Decision study plots are presented to demonstrate how the number of items per examination influences the reliability of each administered assessment. Using the example of a curriculum that assesses gross anatomy knowledge over five summative written and practical examinations, the results of the decision study estimated that 30 and 25 items would be needed on each written and practical examination to reach a reliability of 0.80, respectively. This study is particularly relevant to educators who may question whether the amount of anatomy content assessed in multidisciplinary evaluations is sufficient for making judgments about the anatomical aptitude of students. Anat Sci Educ 10: 109-119. © 2016 American Association of Anatomists.

Direct Observation of Clinical Skills Feedback Scale: Development and Validity Evidence

ABSTRACTConstruct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace. Background: Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use. Approach: Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS. Four rater types (residents n = 21, medical students n = 8, faculty n = 12, and educators n = 12) used the DOCS-FBS to rate videotaped feedback encounters of variable quality. The psychometric properties of the scale were determined using a generalizability analysis. Participants also completed a survey to gather data on a 5-point Likert scale to inform the ease of use, clarity, knowledge acquisition, and acceptability of the scale. Results: Mean video ratings ranged from 1.38 to 2.96 out of 3 and followed the intended pattern suggesting that the tool allowed raters to distinguish between examples of higher and lower quality feedback. There were no significant differences between rater type (range = 2.36–2.49), suggesting that all groups of raters used the tool in the same way. The generalizability coefficients for the scale ranged from 0.97 to 0.99. Item-total correlations were all above 0.80, suggesting some redundancy in items. Participants found the scale easy to use (M = 4.31/5) and clear (M = 4.23/5), and most would recommend its use (M = 4.15/5). Use of DOCS-FBS was acceptable to both trainees (M = 4.34/5) and supervisors (M = 4.22/5). Conclusions: The DOCS-FBS can reliably differentiate between feedback encounters of higher and lower quality. The scale has been shown to have excellent internal consistency. We foresee the DOCS-FBS being used as a means to provide objective evidence that faculty development efforts aimed at improving feedback skills can yield results through formal assessment of feedback quality.

Baseline Characteristics and Generalizability of Participants in an Internet Smoking Cessation Randomized Trial.

The potential for sampling bias in Internet smoking cessation studies is widely recognized. However, few studies have explicitly addressed the issue of sample representativeness in the context of an Internet smoking cessation treatment trial. The purpose of the present study is to examine the generalizability of participants enrolled in a randomized controlled trial of an Internet smoking cessation intervention using weighted data from the National Health Interview Survey (NHIS). A total of 5290 new users on a smoking cessation website enrolled in the trial between March 2012 and January 2015. Descriptive statistics summarized baseline characteristics of screened and enrolled participants, and multivariate analysis examined predictors of enrollment. Generalizability analyses compared demographic and smoking characteristics of trial participants to current smokers in the 2012-2014 waves of NHIS (n = 19,043) and to an NHIS subgroup based on Internet use and cessation behavior (n = 3664). Effect sizes were obtained to evaluate the magnitude of differences across variables. Predictors of study enrollment were age, gender, race, education, and motivation to quit. Compared to NHIS smokers, trial participants were more likely to be female, college educated, and daily smokers and to have made a quit attempt in the past year (all effect sizes 0.25-0.60). In comparisons with the NHIS subgroup, differences in gender and education were attenuated, while differences in daily smoking and smoking rate were amplified. Few differences emerged between Internet trial participants and nationally representative samples of smokers, and all were in expected directions. This study highlights the importance of assessing generalizability in a focused and specific manner. CLINICALTRIALS.GOV: #NCT01544153.

MP015: Daily encounter cards: evaluating the quality of documented assessments

Introduction: In response to concerns in the literature over the quality of completed work-based assessments (WBAs), faculty development and rater training initiatives have been developed. The Completed Clinical Evaluation Report Rating (CCERR) was designed to evaluate these interventions by providing a measure of the quality of documented assessments on In-Training Evaluation Reports (ITERs). Daily Encounter Cards (DECs) are a common form of WBA used in the Emergency Department setting. A tool to evaluate initiatives aimed at improving the quality of completion of this widely used WBA is also needed. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Methods: This study was conducted in the Department of Emergency Medicine at the University of Ottawa. Six experts in resident assessment grouped 60 DECs into three quality categories (high, average, poor) based on their perception of how informative each DEC was for reporting judgments of the resident’s performance. Eight clinical supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared using a univariate ANOVA to determine if the CCERR was able to discriminate DEC quality. Reliability for the CCERR scores was determined using a generalizability analysis. Results: Mean CCERR scores for the high (37.3, SD=1.2), average (24.2, SD=3.3), and poor (14.4, SD=1.4) quality groups differed (p&lt;0.001). A pairwise comparison demonstrated that differences between all three quality groups were statistically significant (p&lt;0.001), indicating that the CCERR was able to discriminate DEC quality as judged by experts. A generalizability study demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. Conclusion: There is strong validity evidence to support the use of the CCERR to evaluate DEC quality. It can be used to provide feedback to supervisors for improving assessment reporting, and offers a quantitative measure of change in assessor behavior when utilized as a program evaluation instrument for determining the quality of completed DECs.