Completed Clinical Evaluation Report Rating Scores Research Articles

Exploring gender influences in the quality of workplace-based assessments.

Workplace-based assessments are an important tool for trainee feedback and as a means of reporting expert judgments of trainee competence in the workplace. However, the literature has demonstrated that gender bias can exist within these assessments. We aimed to determine whether gender differences in the quality of workplace-based assessment data exist in our residency training program. This study was conducted at the University of Ottawa in the Department of Emergency Medicine. Four end-of-shift workplace-based assessments completed by men faculty and four completed by women faculty were randomly selected for each resident during the 2018-2019 academic year. Two blinded raters scored each workplace-based assessment using the Completed Clinical Evaluation Report Rating (CCERR), a published nine-item quantitative measure of workplace-based assessment quality. A 2 × 2 mixed measures analysis of variance (ANOVA) of resident gender and faculty gender was conducted, with mean CCERR score as the dependent variable. The ANOVA was repeated with mean workplace-based assessment rating as the dependent variable. A total of 363 workplace-based assessments were analyzed for 46 residents. There were no significant effects of faculty or resident gender on the quality of workplace-based assessments (p = 0.30). There was no difference in mean workplace-based assessment ratings between women and men residents (p = 0.92), and no interaction between resident and faculty gender (p = 0.62). Mean CCERR score was 25.8, SD = 4.2, indicating average quality assessments. We did not find faculty or resident gender differences in the quality of workplace-based assessments completed in our training program. While the literature has previously demonstrated gender bias in trainee assessments, our results are not surprising as assessment culture varies by institution and program. Our study cautions against generalizing gender bias across contexts, and offers an approach that educators can use to evaluate whether gender bias in the quality of trainee assessments exists within their program.

Comparing the Ottawa Emergency Department Shift Observation Tool (O-EDShOT) to the traditional daily encounter card: measuring the quality of documented assessments.

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a workplace-based assessment designed to assess a trainee's performance across an entire shift. It was developed in response to validity concerns with traditional end-of-shift workplace-based assessments, such as the daily encounter card. The O-EDShOT previously demonstrated strong psychometric characteristics; however, it remains unknown whether the O-EDShOT facilitates measurable improvements in the quality of documented assessments compared to daily encounter cards. Three randomly selected daily encounter cards and three O-EDShOTs completed by 24 faculty were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published 9-item quantitative measure of the quality of a completed workplace-based assessment. Automated-CCERR (A-CCERR) scores, which do not require raters, were also calculated. Paired sample t tests were conducted to compare the quality of assessments between O-EDShOTs and DECs as measured by the CCERR and A-CCERR. CCERR scores were significantly higher for O-EDShOTs (mean(SD) = 25.6(2.6)) compared to daily encounter cards (21.5(3.9); t(23) = 5.2, p < 0.001, d = 1.1). A-CCERR scores were also significantly higher for O-EDShOTs (mean(SD) = 18.5(1.6)) than for daily encounter cards (15.5(1.2); t(24) = 8.4, p < 0.001). CCERR items 1, 4 and 9 were rated significantly higher for O-EDShOTs compared to daily encounter cards. The O-EDShOT yields higher quality documented assessments when compared to the traditional end-of-shift daily encounter card. Our results provide additional validity evidence for the O-EDShOT as an assessment tool for capturing trainee on-shift performance that can be used as a stimulus for actionable feedback and as a source for high-quality workplace-based assessment data to inform decisions about emergency medicine trainee progress and promotion.

Nudging clinical supervisors to provide better in-training assessment reports.

IntroductionIn-training assessment reports (ITARs) summarize assessment during a clinical placement to inform decision-making and provide formal feedback to learners. Faculty development is an effective but resource-intensive means of improving the quality of completed ITARs. We examined whether the quality of completed ITARs could be improved by ‘nudges’ from the format of ITAR forms.MethodsOur first intervention consisted of placing the section for narrative comments at the beginning of the form, and using prompts for recommendations (Do more, Keep doing, Do less, Stop doing). In a second intervention, we provided a hyperlink to a detailed assessment rubric and shortened the checklist section. We analyzed a sample of 360 de-identified completed ITARs from six disciplines across the three academic years where the different versions of the ITAR were used. Two raters independently scored the ITARs using the Completed Clinical Evaluation Report Rating (CCERR) scale. We tested for differences between versions of the ITAR forms using a one-way ANOVA for the total CCERR score, and MANOVA for the nine CCERR item scores.ResultsChanges to the form structure (nudges) improved the quality of information generated as measured by the CCERR instrument, from a total score of 18.0/45 (SD 2.6) to 18.9/45 (SD 3.1) and 18.8/45 (SD 2.6), p = 0.04. Specifically, comments were more balanced, more detailed, and more actionable compared with the original ITAR.DiscussionNudge interventions, which are inexpensive and feasible, should be included in multipronged approaches to improve the quality of assessment reports.

Daily Evaluation Cards Are Superior for Student Assessment Compared to Single Rater In-Training Evaluations.

The University of Manitoba's ambulatory pediatric clerkship transitioned to daily encounter cards (DECs) from single in-training evaluation reports (ITERs). The impact of this change on quality of student assessment was unknown. Using the validated Completed Clinical Evaluation Report Rating (CCERR) scale, we compared the assessment quality of the single ITER to the DEC-based system. Block randomization was used to select from a cohort of ITER- and DEC-based assessments during equivalent points in clerkship training. Data were transcribed and anonymized and scored by two blinded raters using the CCERR. Inter-rater reliability for total CCERR scores was substantive (> 0.6). Mean total CCERR score for the DEC cohort was significantly higher than for the ITER cohort (25.2 vs. 16.8, p < 0.001), as were the mean scores for each item (2.81 vs. 1.86, p < 0.05). Multivariate logistical regression supported the significant influence of assessment method on assessment quality. There is improvement in the average quality of student assessments associated with the transition from an ITER-based system to a DEC-based system. However, the improvement to only average CCERR scores for the DEC cohort suggests an unmet need for faculty development.

Faculty Development- Is Some Better Than None?

This article was migrated. The article was marked as recommended. Introduction: With the advent of competency-based medical education there is an emphasis on formative workplace based assessment. The quality of these assessments is a concern for medical educators and their trainees. Faculty development (FD) strategies to improve assessment quality have resulted in some success. However, few faculty participate, and those who do are likely more motivated to improve, making it difficult to demonstrate a conclusive benefit. To address these weaknesses, we designed a FD initiative to improve the quality of completed in-training evaluation reports (ITERs). All faculty within a division participated. We hypothesized that clinical supervisors would improve their ITER quality based on feedback, regardless of their own motivation to do so, with a simple, point-in-time intervention. Methods: In this three-phase study, two independent raters used the Completed Clinical Evaluation Report Rating (CCERR) to assess the quality of ITERs completed by all faculty in the Division of Orthopedic Surgery at the University of Ottawa. In phase one, ITERs from the previous nine months were evaluated. In phase two, the participants were aware that their ITERs were being evaluated, but they did not receive feedback. In phase three, participants received regular feedback on their performance in the form of their mean CCERR scores. Mean CCERR scores from the different phases of the study were compared. Results: CCERR scores were similar for all three phases (one: 17.56 ± 1.02, two: 17.65 ± 0.96, three: 17.54 ± 0.75, p=0.98). Discussion and Conclusions: There was no evidence in our study that participants' improved their ITER quality despite being aware that they were being evaluated and/or receiving feedback. Potentially, this was related to a lack of motivation. Alternatively, the intensity and/or frequency of the feedback may have been inadequate to create change. These results raise concerns that some faculty development may not necessarily be better than none.

Does faculty development influence the quality of in-training evaluation reports in pharmacy?

BackgroundIn-training evaluation reports (ITERs) of student workplace-based learning are completed by clinical supervisors across various health disciplines. However, outside of medicine, the quality of submitted workplace-based assessments is largely uninvestigated. This study assessed the quality of ITERs in pharmacy and whether clinical supervisors could be trained to complete higher quality reports.MethodsA random sample of ITERs submitted in a pharmacy program during 2013–2014 was evaluated. These ITERs served as a historical control (control group 1) for comparison with ITERs submitted in 2015–2016 by clinical supervisors who participated in an interactive faculty development workshop (intervention group) and those who did not (control group 2). Two trained independent raters scored the ITERs using a previously validated nine-item scale assessing report quality, the Completed Clinical Evaluation Report Rating (CCERR). The scoring scale for each item is anchored at 1 (“not at all”) and 5 (“exemplary”), with 3 categorized as “acceptable”.ResultsMean CCERR score for reports completed after the workshop (22.9 ± 3.39) did not significantly improve when compared to prospective control group 2 (22.7 ± 3.63, p = 0.84) and were worse than historical control group 1 (37.9 ± 8.21, p = 0.001). Mean item scores for individual CCERR items were below acceptable thresholds for 5 of the 9 domains in control group 1, including supervisor documented evidence of specific examples to clearly explain weaknesses and concrete recommendations for student improvement. Mean item scores for individual CCERR items were below acceptable thresholds for 6 and 7 of the 9 domains in control group 2 and the intervention group, respectively.ConclusionsThis study is the first using CCERR to evaluate ITER quality outside of medicine. Findings demonstrate low baseline CCERR scores in a pharmacy program not demonstrably changed by a faculty development workshop, but strategies are identified to augment future rater training.

Supervisor-trainee continuity and the quality of work-based assessments.

Work-based assessments (WBAs) represent an increasingly important means of reporting expert judgements of trainee competence in clinical practice. However, the quality of WBAs completed by clinical supervisors is of concern. The episodic and fragmented interaction that often occurs between supervisors and trainees has been proposed as a barrier to the completion of high-quality WBAs. The primary purpose of this study was to determine the effect of supervisor-trainee continuity on the quality of assessments documented on daily encounter cards (DECs), a common form of WBA. The relationship between trainee performance and DEC quality was also examined. Daily encounter cards representing three differing degrees of supervisor-trainee continuity (low, intermediate, high) were scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR), a previously published nine-item quantitative measure of DEC quality. An analysis of variance (anova) was performed to compare mean CCERR scores among the three groups. Linear regression analysis was conducted to examine the relationship between resident performance and DEC quality. Differences in mean CCERR scores were observed between the three continuity groups (p=0.02); however, the magnitude of the absolute differences was small (partial eta-squared=0.03) and not educationally meaningful. Linear regression analysis demonstrated a significant inverse relationship between resident performance and CCERR score (p<0.001, r2 =0.18). This inverse relationship was observed in both groups representing on-service residents (p=0.001, r2 =0.25; p=0.04, r2 =0.19), but not in the Off-service group (p=0.62, r2 =0.05). Supervisor-trainee continuity did not have an educationally meaningful influence on the quality of assessments documented on DECs. However, resident performance was found to affect assessor behaviours in the On-service group, whereas DEC quality remained poor regardless of performance in the Off-service group. The findings suggest that greater attention should be given to determining ways of improving the quality of assessments reported for off-service residents, as well as for those residents demonstrating appropriate clinical competence progression.

Effectiveness of a Focused Faculty Development Program on the Written Comments Component of Resident Evaluations

OBJECTIVE: Evaluate effectiveness of faculty development program combined with focused feedback on quality of written comments in resident evaluations. METHODS: Faculty were educated to improve quality of written comments on resident evaluations. Monthly evaluations concurrently changed, requiring comments before ratings. Quality was determined using validated CCERR (Completed Clinical Evaluation Report Rating) on de-identified evaluations, final scores representing median score from both authors. CCERR scores vary from nine to 45 correlating with quality. Baseline data were abstracted from the last half of 2015. Faculty development seminar took place May 2016. Faculty then received feedback at the monthly faculty meetings as well as individual coaching. After 9 months, a survey of faculty and residents will gauge comfort writing comments. CCERR scores were compared using nonparametric test, P&lt;.05 defining significance. RESULTS: Two hundred five evaluations (50% with written comments) comprised the baseline with 22, 28, 38, 27, 28, 32 and 23 evaluations completed (May through November). The median CCERR score increased from baseline of 10–22.5, 20, 20, 21, 22, 21 and 19 (May through November, P&lt;.0001). CONCLUSION: Previous studies confirm CCERR validity, eight points representing meaningful change. Educational sessions and ongoing feedback individually have been shown to increase scores, but not meaningfully. We achieved meaningful change, but our baseline was lower than previous studies. Ongoing feedback was not effective at generating further improvement. It is not clear if the improvement seen resulted from faculty development or changes in the structure of monthly evaluations requiring comments.

Daily Encounter Cards-Evaluating the Quality of Documented Assessments.

Concerns over the quality of work-based assessment (WBA) completion has resulted in faculty development and rater training initiatives. Daily encounter cards (DECs) are a common form of WBA used in ambulatory care and shift work settings. A tool is needed to evaluate initiatives aimed at improving the quality of completion of this widely used form of WBA. The completed clinical evaluation report rating (CCERR) was designed to provide a measure of the quality of documented assessments on in-training evaluation reports. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Six experts in resident assessment grouped 60 DECs into 3 quality categories (high, average, and poor) based on how informative each DEC was for reporting judgments of the resident's performance. Eight supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared to determine if the CCERR could discriminate based on DEC quality. Statistically significant differences in CCERR scores were observed between all quality groups (P < .001). A generalizability analysis demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. The CCERR is a reliable and valid tool to evaluate DEC quality. It can serve as an outcome measure for studying interventions targeted at improving the quality of assessments documented on DECs.

MP002: Beyond rater cognition: the impact of supervisor continuity on the quality of documented work-based assessments

Introduction: Barriers to completing high quality work-based assessments (WBAs) include relational factors such as the episodic and fragmented interaction that often exists between clinical supervisors and trainees. In an effort to increase supervisor-trainee continuity, the Department of Emergency Medicine at the University of Ottawa created Clinical Teaching Teams (CTT) in which a resident and clinical supervisor work matched shifts together throughout the year. The aim of this study was to determine the impact of supervisor-trainee continuity on the quality of assessments documented on Daily Encounter Cards (DECs). Methods: DECs completed by 20 clinical supervisors were collected and sorted into three groups representing differing degrees of supervisor-trainee continuity (Group 1: CTT emergency resident; Group 2: non-CTT emergency resident; Group 3: non-CTT off-service resident). DECs were scored using the Completed Clinical Evaluation Report Rating (CCERR), a 9-item instrument that has been shown to have reliable ratings and the ability to discriminate the quality of completed DECs. Scores were analyzed using a univariate ANOVA with “mean CCERR score” as the dependent variable and “continuity group” and “supervisor” as between-subject variables. The relationship between CCERR scores and number of CTT encounters over time was examined using a repeated measures ANOVA with “encounter number” as the within-subject factor. Results: Mean CCERR scores for the CTT (21.0, SD=5.8), non-CTT (21.9, SD=4.2), and off-service (20.7, SD=4.0) groups differed (p=0.019). A subsequent pairwise comparison demonstrated a statistically significant difference in means between the non-CTT and off-service groups (p=0.04); however, this 1.2 difference on the 45-point CCERR scale is unlikely to be of any educational significance. The number of repeated encounters did not have a statistically significant effect on CCERR scores (p=0.43) indicating that DEC quality did not improve with greater supervisor-trainee interaction. Conclusion: DEC quality as scored by the CCERR was low for all three groups. Increasing supervisor continuity alone did not result in higher quality assessments of clinical performance. Additional research focusing on the educational alliance that develops between supervisor and trainee may hold greater promise.

LO080: Performance and proximity: exploring resident factors that impact the quality of work-based assessments

Introduction: Much of the literature investigating the challenges associated with completing high quality work-based assessments (WBAs) have raised specific concerns over the appropriate documentation of assessments of underperforming trainees or trainees in difficulty. The purpose of this study was to examine the relationship between resident performance and the quality of assessments documented by supervisors on Daily Encounter Cards (DECs). The effect of trainee proximity (i.e. on-service versus off-service status) on this relationship was also examined. Methods: A series of DECs from the Department of Emergency Medicine at the University of Ottawa was scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR). The CCERR is a 9-item instrument that has previously demonstrated reliable ratings and the ability to discriminate the quality of completed DECs. A proxy measure of resident performance was calculated by averaging the scores across performance items on the DEC to produce a “mean DEC rating”. Linear regression analysis was conducted with “mean DEC rating” as the independent measure and CCERR score as the dependent measure. Separate linear regression analyses were repeated for DECs completed for on-service versus off-service residents. Results: Linear regression analysis demonstrated a small but significant inverse relationship between mean DEC rating and CCERR score (p&lt;0.001, r=-0.184), suggesting that when residents performed poorly, their supervisors tended to document higher quality assessments, and conversely, when residents performed well, their supervisors provided lower quality assessments. Further analysis demonstrated that this relationship was present for the on-service group (p&lt;0.001, r=-0.24). However, no relationship was observed in the off-service group (p=0.62, r=-0.05). Conclusion: Resident performance and trainee proximity are important factors impacting the quality of documented clinical performance assessments. Greater attention needs to be given to determining ways of improving the quality of assessments reported for residents who are appropriately progressing in their clinical competence as well as for off-service trainees.

MP015: Daily encounter cards: evaluating the quality of documented assessments

Introduction: In response to concerns in the literature over the quality of completed work-based assessments (WBAs), faculty development and rater training initiatives have been developed. The Completed Clinical Evaluation Report Rating (CCERR) was designed to evaluate these interventions by providing a measure of the quality of documented assessments on In-Training Evaluation Reports (ITERs). Daily Encounter Cards (DECs) are a common form of WBA used in the Emergency Department setting. A tool to evaluate initiatives aimed at improving the quality of completion of this widely used WBA is also needed. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Methods: This study was conducted in the Department of Emergency Medicine at the University of Ottawa. Six experts in resident assessment grouped 60 DECs into three quality categories (high, average, poor) based on their perception of how informative each DEC was for reporting judgments of the resident’s performance. Eight clinical supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared using a univariate ANOVA to determine if the CCERR was able to discriminate DEC quality. Reliability for the CCERR scores was determined using a generalizability analysis. Results: Mean CCERR scores for the high (37.3, SD=1.2), average (24.2, SD=3.3), and poor (14.4, SD=1.4) quality groups differed (p&lt;0.001). A pairwise comparison demonstrated that differences between all three quality groups were statistically significant (p&lt;0.001), indicating that the CCERR was able to discriminate DEC quality as judged by experts. A generalizability study demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. Conclusion: There is strong validity evidence to support the use of the CCERR to evaluate DEC quality. It can be used to provide feedback to supervisors for improving assessment reporting, and offers a quantitative measure of change in assessor behavior when utilized as a program evaluation instrument for determining the quality of completed DECs.

In-training evaluations: developing an automated screening tool to measure report quality.

In-training evaluation (ITE) is used to assess resident competencies in clinical settings. This assessment is documented on an evaluation report (In-Training Evaluation Report [ITER]). Unfortunately, the quality of these reports can be questionable. Therefore, training programmes to improve report quality are common. The Completed Clinical Evaluation Report Rating (CCERR) was developed to assess completed report quality and has been shown to do so in a reliable manner, thus enabling the evaluation of these programmes. The CCERR is a resource-intensive instrument, which may limit its use. The purpose of this study was to create a screening measure (Proxy-CCERR) that can predict the CCERR outcome in a less resource-intensive manner. Using multiple regression, the authors analysed a dataset of 269 ITERs to create a model that can predict the associated CCERR scores. The resulting predictive model was tested on the CCERR scores for an additional sample of 300 ITERs. The quality of an ITER, as measured by the CCERR, can be predicted using a model involving only three variables (R(2) =0.61). The predictive variables included the total number of words in the comments, the variability of the ratings and the proportion of comment boxes completed on the form. It is possible to model CCERR scores in a highly predictive manner. The predictive variables can be easily extracted in an automated process. Because this model is less resource-intensive than the CCERR, it makes it possible to provide feedback from ITER training programmes to large groups of supervisors and institutions, and even to create automated feedback systems using Proxy-CCERR scores.

Critical Synthesis Package: Completed Clinical Evaluation Report Rating (CCERR)

OPEN ACCESSApril 23, 2014Critical Synthesis Package: Completed Clinical Evaluation Report Rating (CCERR) Steven Rougas, MD Steven Rougas, MD The Warren Alpert Medical School of Brown University Google Scholar More articles by this author https://doi.org/10.15766/mep_2374-8265.9772 SectionsAbout ToolsDownload Citations ShareFacebookTwitterEmail AbstractAbstractThis Critical Synthesis Package contains: (1) a Critical Analysis of the psychometric properties and the application to health science education of the Completed Clinical Evaluation Report Rating (CCERR), and (2) a copy of the CCERR instrument developed by Nancy Dudek, MD, MEd.The CCERR is a short, low-resource instrument developed to evaluate the quality of how a clinical evaluation form has been filled out beyond the actual design of the form. The instrument is intended for use by faculty supervisors, with a heavy focus on program directors and evaluation coordinators, to assess completed clinical evaluation forms (with quantitative and qualitative information) for any medical trainee after a clinical rotation. The instrument is a paper-based form with nine items; each item is scored on a 5-point Likert scale with “1” representing the lowest score (not at all) and “5” representing the highest score (exemplary). The literature on the CCERR supports the validity of its scores and each item in the instrument is supported by appropriate psychometric analysis, which is the measure's greatest strength. The greatest disadvantages of the CCERR is its lack of a standardized cut-point to establish clear definitions of in-training evaluation reports (ITERs) that receive poor, average, and high scores, and the inadequate evidence to support consequence validity. Though the instrument has only been tested on faculty who assess medical learners on clerkships, it has the potential to be applied to other forms of evaluation of resident physicians and other allied health professionals. Further studies outlining the impact of low or high CCERR scores of ITERs on learners' academic standing, remediation, or subsequent promotion would be of use to the academic community in order to establish relevance for its widespread use among educators. Educational Objectives To describe the purpose and basic properties of the Completed Clinical Evaluation Report Rating (CCERR), including number of items and scales, and psychometric properties;To describe the application of the CCERR to the field of health sciences education;To evaluate the relative strengths and weaknesses of the CCERR; andTo provide the CCERR and supplemental materials to aid in its administration. Sign up for the latest publications from MedEdPORTAL Add your email below FILES INCLUDEDReferencesRelatedDetails FILES INCLUDED Included in this publication: Critical Analysis of the Completed Clinical Evaluation Report Rating (CCERR).pdf Completed Clinical Evaluation Report Rating (CCERR Instrument).pdf To view all publication components, extract (i.e., unzip) them from the downloaded .zip file. Download editor’s noteThis publication may contain technology or a display format that is no longer in use. Copyright & Permissions© 2014 Rougas. This is an open-access publication distributed under the terms of the Creative Commons Attribution-NonCommercial-Share Alike license.KeywordsClinical ClerkshipsRotationDREAMQuality ImprovementDirectory and Repository of Educational Assessment Measures Disclosures None to report. Funding/Support None to report. Loading ...

Quality In-Training Evaluation Reports—Does Feedback Drive Faculty Performance?

Clinical faculty often complete in-training evaluation reports (ITERs) poorly. Faculty development (FD) strategies should address this problem. An FD workshop was shown to improve ITER quality, but few physicians attend traditional FD workshops. To reach more faculty, the authors developed an "at-home" FD program offering participants various types of feedback on their ITER quality based on the workshop content. Program impact is evaluated here. Ninety-eight participants from four medical schools, all clinical supervisors, were recruited in 2009-2010; 37 participants completed the study. These were randomized into five groups: a control group and four other groups with different feedback conditions. ITER quality was assessed by two raters using a validated tool: the completed clinical evaluation report rating (CCERR). Participants were given feedback on their ITER quality based on group assignment. Six months later, participants submitted new ITERs. These ITERs were assessed using the CCERR, and feedback was sent to participants on the basis of their group assignment. This process was repeated two more times, ending in 2012. CCERR scores from the participants in all feedback groups were collapsed (n=27) and compared with scores from the control group (n=10). Mean CCERR scores significantly increased over time for the feedback group but not the control group. The results suggest that faculty are able to improve ITER quality following a minimal "at-home" FD intervention. This also adds to the growing literature that has found success with improving the quality of trainee assessments following rater training.

Quality evaluation reports: Can a faculty development program make a difference?

Background: The quality of medical student and resident clinical evaluation reports submitted by rotation supervisors is a concern. The effectiveness of faculty development (FD) interventions in changing report quality is uncertain.Aims: This study assessed whether faculty could be trained to complete higher quality reports.Method: A 3-h interactive program designed to improve evaluation report quality, previously developed and tested locally, was offered at three different Canadian medical schools. To assess for a change in report quality, three reports completed by each supervisor prior to the workshop and all reports completed for 6 months following the workshop were evaluated by three blinded, independent raters using the Completed Clinical Evaluation Report Rating (CCERR): a validated scale that assesses report quality.Results: A total of 22 supervisors from multiple specialties participated. The mean CCERR score for reports completed after the workshop was significantly higher (21.74 ± 4.91 versus 18.90 ± 5.00, p = 0.02).Conclusions: This study demonstrates that this FD workshop had a positive impact upon the quality of the participants’ evaluation reports suggesting that faculty have the potential to be trained with regards to trainee assessment. This adds to the literature which suggests that FD is an important component in improving assessment quality.