In-training Evaluation Reports Research Articles

The Influence of Saudi Board of Emergency Medicine Residency Educators on Residents' Academic and Clinical Performance in Riyadh, Saudi Arabia.

The Saudi Board of Emergency Medicine (SBEM) is required to assign educators including program directors to supervise their residents. These educators may impact the residents' academic and clinical performances. After thorough review, we noticed that the current literature lacks information about the direct influence of emergency medicine educators on their residents' academic and clinical performances. The main purpose of this study is to assess the residents' confidence level, and to measure the program directors' satisfaction of the residents' performances during their SBEM training yearsin Riyadh hospitals. This concurrent mixed-methods study was conducted in nine training hospitals in Riyadh, Saudi Arabia, during the 2021-2022 academic year. For the quantitative aspect, a cross-sectional survey was used, based on a questionnaire administered to postgraduate year (PGY)-2 to PGY-4 SBEM residents (n=120) using a nonprobability convenient sampling technique. The survey aimed to assess residents' confidence in their academic and clinical performance, influenced by their educators. A Likert scale with a total of 25 items, covering the seven roles of the CANMEDs framework, was employed. This assessment utilized a published tool called the In-Training Evaluation Report. For the qualitative aspect, program directors' perspectives were gathered through one-on-one unstructured interviews (n=9), guided by theoretical saturation. A purposive sampling technique was employed to select program directors. The interview tool collected demographic data, including gender, years of experience, and training hospital, and included 17 open-ended questions to explore program directors' opinions. The result of mixed methods showed that both quantitative and qualitative data sets were divergent with all independent variables (resident's age, gender, and training level) with the exception of the training hospital which was convergent. Thus, the residents' confidence toward their academic and clinical performances induced by their institutional educators was high. However, the program directors indicated changes that related to residents' training level. The residents' confidence toward their academic and clinical performance induced by their institutional educators was high.The educators had a great deal of influence over the academic and clinical performance of the residents. However,the program directors thought there were some issues with the performance of the residents.Most of the program directors also believed that several influential factors that may have affected the residents' overall performance includestress and receiptof constructive feedback.

Novel In-Training Evaluation Report in an Internal Medicine Residency Program: Improving the Quality of the Narrative Assessment.

To determine whether incorporating our novel in-training evaluation report (ITER), which prompts each resident to list at least three self-identified learning goals, improved the quality of narrative assessments as measured by the Narrative Evaluation Quality Instrument (NEQI). A total of 1468 narrative assessments from a single institution from 2017 to 2021 were deidentified, compiled, and sorted into the pre-intervention form arm and post-intervention form arm. Due to limitations in our residency management suite, incorporating learning goals required switching from an electronic form to a hand-deliver form. Comments were graded by two research personnel utilizing the NEQI's scale of 0-12, with 12 representing the maximum quality for a comment. The outcome of the study was the mean difference in NEQI score between the electronic pre-intervention period and paper post-intervention period. The mean NEQI score for the pre-intervention period was 2.43 ± 3.34, and the mean NEQI score for the post-intervention period was 3.31 ± 1.71, with a mean difference of 0.88 (p < 0.001). In the pre-intervention period, 46% of evaluations were submitted without a narrative assessment (scored as a zero) while 1% of post-intervention period evaluations had no narrative assessment. Internal consistency reliability, as measured by Ebel's intraclass correlation coefficient (ICC), showed high agreement between the two raters (ICC = 0.92). Our findings suggest that implementing a timely, hand-delivered paper ITER that incorporates resident learning goals can lead to overall higher-quality narrative assessments.

Longitudinal assessment of competency development at The Ohio State University using the competency-based veterinary education (CBVE) model.

With the development of the American Association of Veterinary Medical Colleges' Competency-Based Veterinary Education (CBVE) model, veterinary schools are reorganizing curricula and assessment guidelines, especially within the clinical rotation training elements. Specifically, programs are utilizing both competencies and entrustable professional activities (EPAs) as opportunities for gathering information about student development within and across clinical rotations. However, what evidence exists that use of the central tenets of the CBVE model (competency framework, milestones and EPAs) improves our assessment practices and captures reliable and valid data to track competency development of students as they progress through their clinical year? Here, we report on validity evidence to support the use of scores from in-training evaluation report forms (ITERs) and workplace-based assessments of EPAs to evaluate competency progression within and across domains described in the CBVE, during the final year clinical training period of The Ohio State University's College of Veterinary Medicine (OSU-CVM) program. The ITER, used at the conclusion of each rotation, was modified to include the CBVE competencies that were assessed by identifying the stage of student development on a series of descriptive milestones (from pre-novice to competent). Workplace based assessments containing entrustment scales were used to assess EPAs from the CBVE model within each clinical rotation. Competency progression and entrustment scores were evaluated on each of the 31 rotations offered and high-stakes decisions regarding student performance were determined by a collective review of all the ITERs and EPAs recorded for each learner across each semester and the entire year. Results from the class of 2021, collected on approximately 190 students from 31 rotations, are reported with more than 55 299 total competency assessments combined with milestone placement and 2799 complete EPAs. Approximately 10% of the class was identified for remediation and received additional coaching support. Data collected longitudinally through the ITER on milestones provides initial validity evidence to support using the scores in higher stakes contexts such as identifying students for remediation and for determining whether students have met the necessary requirements to successfully complete the program. Data collected on entrustment scores did not, however, support such decision making. Implications are discussed.

An overview of Internal Medicine Point-of-Care Ultrasound rotations in Canada

BackgroundPoint-of-care ultrasound (POCUS) is a growing part of internal medicine training programs. Dedicated POCUS rotations are emerging as a particularly effective tool in POCUS training, allowing for longitudinal learning and emphasizing both psychomotor skills and the nuances of clinical integration. In this descriptive paper, we set out to review the state of POCUS rotations in Canadian Internal Medicine training programs.ResultsWe identify five programs currently offering a POCUS rotation. These rotations are offered over two to thirteen blocks each year, run over one to four weeks and support one to four learners. Across all programs, these rotations are set up as a consultative service that offers POCUS consultation to general internal medicine inpatients, with some extension of scope to the hospitalist service or surgical subspecialties. The funding model for the preceptors of these rotations is predominantly fee-for-service using consultation codes, in addition to concomitant clinical work to supplement income. All but one program has access to hospital-based archiving of POCUS exams. Preceptors dedicate ten to fifty hours to the rotation each week and ensure that all trainee exams are reviewed and documented in the patient’s medical records in the form of a consultation note. Two of the five programs also support a POCUS fellowship. Only two out of five programs have established learner policies. All programs rely on In-Training Evaluation Reports to provide trainee feedback on their performance during the rotation.ConclusionsWe describe the different elements of the POCUS rotations currently offered in Canadian Internal Medicine training programs. We share some lessons learned around the elements necessary for a sustainable rotation that meets high educational standards. We also identify areas for future growth, which include the expansion of learner policies, as well as the evolution of trainee assessment in the era of competency-based medical education. Our results will help educators that are endeavoring setting up POCUS rotations achieve success.

Impact of Gender on Clinical Evaluation of Trainees in the Intensive Care Unit.

Background: Gender disparities in medical education are increasingly demonstrated, including in trainee assessment.Objective: This study aimed to evaluate whether gender differences exist in trainees’ evaluation during intensive care unit (ICU) rotations, which has not been previously studied.Methods: We reviewed the in-training evaluation reports (ITERs) for trainees rotating through five academic ICUs at the University of Toronto over a 10-year period (2007–2017). We compared the mean global score for the rotation and the mean score for seven training subdomains between men and women trainees. All scores were reported on a scale of 1 (unsatisfactory) to 5 (outstanding).Results: Over the 10-year period, there were 3,203 ITERS overall, representing 1,207 women and 1,996 men trainees. The mean overall score was lower for women than for men trainees: 4.26 (standard deviation [SD], 0.58) for women and 4.30 (SD, 0.60) for men (P = 0.04). This difference was driven by anesthesia trainees, in whom the mean overall score was 4.21 for women and 4.37 for men (P < 0.001), with men trainees scoring consistently higher across all seven training subdomains. Within surgical, internal medicine, and critical care residents, there were no differences between men and women in the overall score or the scores across any of the seven subdomains. Across all ITERS, women were less likely than men to receive an overall rating of 5 (outstanding) for the ICU rotation (33% women vs. 37% men; odds ratio, 0.83; 95% confidence interval, 0.71–0.96).Conclusion: Overall, quantitative evaluation scores between women and men trainees in the ICU are relatively similar. Within anesthesia trainees, scores for men were consistently higher across all domains of evaluation, a finding that requires further investigation.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

Hot Seat Diagnosis.

Competency-based medical education relies on frequent formative in-service assessments to ascertain trainee progression. Currently at our institution, trainees receive a summative end-of-rotation In-Training Evaluation Report based on feedback collected from staff pathologists. There is no method of simulating report sign-out. To develop a formative in-service assessment tool that is able to simulate report sign-out and provide case-by-case feedback to trainees. Further, to compare time- versus competency-based assessment models. Twenty-one pathology trainees were assessed for 20 months. Hot Seat Diagnosis by trainees and trainee assessment by pathologists were recorded in the laboratory information system. In the first iteration, trainees were assessed by using a time-based assessment scale on their ability to diagnose, report, use ancillary tests, comment on clinical implications, and provide intraoperative consultation and/or gross cases. The second iteration used a competency-based assessment scale. Trainees and pathologists completed surveys on the effectiveness of the In-Training Evaluation Report versus the Hot Seat Diagnosis tool. Scores from both iterations correlated significantly with other assessment tools including the Resident In-Service Examination (r = 0.93, P = .04 and r = 0.87, P = .03). The competency-based model was better able to demonstrate improvement over time and stratify junior versus senior trainees than the time-based model. Trainees and pathologists rated Hot Seat Diagnosis as significantly more objective, detailed, and timely than the In-Training Evaluation Report, and effective at simulating report sign-out. Hot Seat Diagnosis is an effective tool for the formative in-service assessment of pathology trainees and simulation of report sign-out, with the competency-based model outperforming the time-based model.

Comparing Entrustment and Competence: AnExploratory Look at Performance-Relevant Information in the Final Year of a Veterinary Program.

Workplace-based assessments and entrustment scales have two primary goals: providing formative information to assist students with future learning; and, determining if and when learners are ready for safe, independent practice. To date, there has not been an evaluation of the relationship between these performance-relevant information pieces in veterinary medicine. This study collected quantitative and qualitative data from a single cohort of final-year students (n = 27) across in-training evaluation reports (ITERs) and entrustment scales in a distributed veterinary hospital environment. Here we compare progression in scoring and performance within and across student, within and across method of assessment, over time. Narrative comments were quantified using the Completed Clinical Evaluation Report Rating (CCERR) instrument to assess quality of written comments. Preliminary evidence suggests that we may be capturing different aspects of performance using these two different methods. Specifically, entrustment scale scores significantly increased over time, while ITER scores did not. Typically, comments on entrustment scale scores were more learner specific, longer, and used more of a coaching voice. Longitudinal evaluation of learner performance is important for learning and demonstration of competence; however, the method of data collection could influence how feedback is structured and how performance is ultimately judged.

Idiosyncrasy in Assessment Comments: Do Faculty Have Distinct Writing Styles When Completing In-Training Evaluation Reports?

Written comments are gaining traction as robust sources of assessment data. Compared with the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what aspects are determined by the trainee being commented upon. The authors compiled in-training evaluation report comment data, generated from 2012 to 2015 by 4 large North American Internal Medicine training programs. The Linguistic Index and Word Count (LIWC) was used to categorize and quantify the language contained. Generalizability theory was used to determine whether faculty could be reliably discriminated from one another based on writing style. Correlations and ANOVAs were used to determine what styles were related to faculty or trainee demographics. Datasets contained 23-142 faculty who provided 549-2,666 assessments on 161-989 trainees. Faculty could easily be discriminated from one another using a variety of LIWC metrics including word count, words per sentence, and the use of "clout" words. These patterns appeared person specific and did not reflect demographic factors such as gender or rank. These metrics were similarly not consistently associated with trainee factors such as postgraduate year or gender. Faculty seem to have detectable writing styles that are relatively stable across the trainees they assess, which may represent an under-recognized source of construct irrelevance. If written comments are to meaningfully contribute to decision making, we need to understand and account for idiosyncratic writing styles.

Development of a certification examination for orthopedic sports medicine fellows

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship. Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair. There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

Necessary but not sufficient: identifying conditions for effective feedback during internal medicine residents' clinical education.

Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles underlying meaningful feedback could be supported or constrained across a variety of feedback opportunities. Using case-study qualitative methodology, interviews and focus groups were conducted to explore 19 internal medicine residents' and 7 faculty members' perceptions of feedback across a variety of feedback opportunities: coaching, mini-CEXs, in-training evaluation reports and routine clinical supervision. Our data analysis moved iteratively between developing conceptual understandings and fine-grained analyses, while attending to both deductive and inductive analysis. Our results suggest that all feedback opportunities, including those created through formalized assessments, can foster meaningful feedback if faculty establish a trusting relationship with the resident, base their feedback on direct observation and support resident learning. However, formalized assessments were often perceived as inhibiting the conditions for meaningful feedback. A coaching program provided a context in which meaningful feedback could arise, in part because faculty were supported in shifting their focus from patient to resident. Meaningful feedback in clinical education may be fostered across a variety of feedback opportunities, however, it is often constrained by assessment. We must consider whether increasing the frequency of formative assessments may inhibit efforts to improve our feedback cultures while, in contrast, freeing up faculty to focus on supporting resident learning could improve these cultures.

Daily Evaluation Cards Are Superior for Student Assessment Compared to Single Rater In-Training Evaluations.

The University of Manitoba's ambulatory pediatric clerkship transitioned to daily encounter cards (DECs) from single in-training evaluation reports (ITERs). The impact of this change on quality of student assessment was unknown. Using the validated Completed Clinical Evaluation Report Rating (CCERR) scale, we compared the assessment quality of the single ITER to the DEC-based system. Block randomization was used to select from a cohort of ITER- and DEC-based assessments during equivalent points in clerkship training. Data were transcribed and anonymized and scored by two blinded raters using the CCERR. Inter-rater reliability for total CCERR scores was substantive (> 0.6). Mean total CCERR score for the DEC cohort was significantly higher than for the ITER cohort (25.2 vs. 16.8, p < 0.001), as were the mean scores for each item (2.81 vs. 1.86, p < 0.05). Multivariate logistical regression supported the significant influence of assessment method on assessment quality. There is improvement in the average quality of student assessments associated with the transition from an ITER-based system to a DEC-based system. However, the improvement to only average CCERR scores for the DEC cohort suggests an unmet need for faculty development.

Detection of Residents With Progress Issues Using a Keyword-Specific Algorithm.

The literature suggests that specific keywords included in summative rotation assessments might be an early indicator of abnormal progress or failure. This study aims to determine the possible relationship between specific keywords on in-training evaluation reports (ITERs) and subsequent abnormal progress or failure. The goal is to create a functional algorithm to identify residents at risk of failure. A database of all ITERs from all residents training in accredited programs at Université Laval between 2001 and 2013 was created. An instructional designer reviewed all ITERs and proposed terms associated with reinforcing and underperformance feedback. An algorithm based on these keywords was constructed by recursive partitioning using classification and regression tree methods. The developed algorithm was tuned to achieve 100% sensitivity while maximizing specificity. There were 41 618 ITERs for 3292 registered residents. Residents with failure to progress were detected for family medicine (6%, 67 of 1129) and 36 other specialties (4%, 78 of 2163), while the positive predictive values were 23.3% and 23.4%, respectively. The low positive predictive value may be a reflection of residents improving their performance after receiving feedback or a reluctance by supervisors to ascribe a "fail" or "in difficulty" score on the ITERs. Classification and regression trees may be helpful to identify pertinent keywords and create an algorithm, which may be implemented in an electronic assessment system to detect future residents at risk of poor performance.

Student feedback experiences in a cross-border medical education curriculum.

ObjectivesTo determine non-Western situated health professional student experiences and preferences for feedback in workplace-based settings. MethodsWe conducted five focus groups with 27 students of Arab-origin enrolled in a Canadian-accredited cross-border pharmacy program in Qatar. Transcripts of recorded discussions were analyzed using the framework method. Hofstede’s and Hall’s cultural dimension models were employed to understand described feedback encounters and behaviours. ResultsWe identified three themes associated with cultural influences on student feedback experiences, namely: 1) collectivism; 2) power distance; and 3) context. Trainees described clinical supervisors who inadequately recognized individual performance, rejected critique, and insufficiently documented feedback onto the written in-training evaluation report. Conversely, students expected specific and timely feedback, invited criticism for learning, and desired clear written commentary. ConclusionsFeedback behaviours of clinical supervisors, but not those of trainees, were consistent with local cultural norms as described by Hofstede and Hall. Instead, feedback expectations of pharmacy students in Qatar largely echo those of other trainees enrolled in professional curricula situated outside the Middle East. Principles for optimal feedback in clinical training largely arise from Western perspectives but are not necessarily universal. Our work demonstrates that practices, in part, may be subject to local socio-cultural influences. This is of particular importance in the experiential training component of cross-border medical education programs adopted by overseas institutions. Our findings also further add to the growing body of literature reporting suboptimal feedback in workplace-based learning, reinforcing the need to cultivate more student-centered practices in health professional training globally.

Contextualizing Work-Based Assessments of Faculty and Residents: Is There a Relationship Between the Clinical Practice Environment and Assessments of Learners and Teachers?

Competence is bound to context, yet seldom is environment explicitly considered in work-based assessments. This study explored faculty and residents' perspectives of the environment during internal medicine clinical teaching unit (CTU) rotations, the extent that each group accounts for environmental factors in assessments, and relationships between environmental factors and assessments. From July 2014 to June 2015, 212 residents and 54 faculty across 5 teaching hospitals at University of Toronto rated their CTU environment using a novel Practice Environment Rating Scale (PERS) matched by block and hospital. Faculty-PERS data were paired to In-Training Evaluation Reports (ITERs) of residents supervised during each block, and Resident-PERS data to Resident Assessment of Teaching Effectiveness (RATE) scores of the same faculty. Differences between perceptions and assessments were tested using repeated-measures MANOVAs, ANOVAs, and correlations. One-hundred sixty-four residents completed the PERS; residents rated the CTU environment more positively than faculty (3.91/5 vs. 3.29, P < .001). Residents were less likely to report considering environmental factors when assessing faculty (2.70/5) compared with faculty assessing residents (3.40, P < .0001), d = 1.2. Whereas Faculty-PERS ratings did not correlate with ITER scores, Resident-PERS ratings had weak to moderate correlations with RATE scores (overall r = 0.27, P = .001). Residents' perceptions of the environment had small but significant correlations with assessments of faculty. Faculty's perceptions did not affect assessments of residents, potentially because they reported accounting for environmental factors. Understanding the interplay between environment and assessment is essential to developing valid competency judgments.

Faculty Development- Is Some Better Than None?

This article was migrated. The article was marked as recommended. Introduction: With the advent of competency-based medical education there is an emphasis on formative workplace based assessment. The quality of these assessments is a concern for medical educators and their trainees. Faculty development (FD) strategies to improve assessment quality have resulted in some success. However, few faculty participate, and those who do are likely more motivated to improve, making it difficult to demonstrate a conclusive benefit. To address these weaknesses, we designed a FD initiative to improve the quality of completed in-training evaluation reports (ITERs). All faculty within a division participated. We hypothesized that clinical supervisors would improve their ITER quality based on feedback, regardless of their own motivation to do so, with a simple, point-in-time intervention. Methods: In this three-phase study, two independent raters used the Completed Clinical Evaluation Report Rating (CCERR) to assess the quality of ITERs completed by all faculty in the Division of Orthopedic Surgery at the University of Ottawa. In phase one, ITERs from the previous nine months were evaluated. In phase two, the participants were aware that their ITERs were being evaluated, but they did not receive feedback. In phase three, participants received regular feedback on their performance in the form of their mean CCERR scores. Mean CCERR scores from the different phases of the study were compared. Results: CCERR scores were similar for all three phases (one: 17.56 ± 1.02, two: 17.65 ± 0.96, three: 17.54 ± 0.75, p=0.98). Discussion and Conclusions: There was no evidence in our study that participants' improved their ITER quality despite being aware that they were being evaluated and/or receiving feedback. Potentially, this was related to a lack of motivation. Alternatively, the intensity and/or frequency of the feedback may have been inadequate to create change. These results raise concerns that some faculty development may not necessarily be better than none.

Correlation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes

Faculty evaluations, ABSITE scores, and operative case volumes often tell little about true resident performance. We developed an objective structured clinical examination called the Surgical X-Games (5 rooms, 15 minutes each, 12-15 tests total, different for each postgraduate [PGY] level). We hypothesized that performance in X-Games will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative cases volumes. PGY 2 to 5 GS residents (n = 35) were tested in a semiannual X-Games assessment using multiple simulation tasks: laparoscopic skills, bowel anastomosis, CT/CXR analysis, chest tube placement, etc. over 1 academic year. Resident scores were compared to their ABSITE, in-training evaluation reports, and operating room case numbers. Academic medical center. PGY-2, 3, 4, and 5 GS residents at Mayo Clinic in Rochester, MN. Results varied greatly within each class except for staff evaluations: in-training evaluation reports medians for PGY-2s were 5.3 (range: 5.0-6.0), PGY-3s 5.9 (5.5-6.3), PGY-4s 5.6 (5.0-6.0), and PGY-5s were 6.1 (5.6-6.9). Although ABSITE and operating room case volumes fluctated greatly with each PGY class, only X-Games scores (median: PGY-2 = 82, PGY-3 = 61, PGY-4 = 76, and PGY-5 = 60) correlated positively (p < 0.05) with operative case volume and negatively (p < 0.05) with staff evaluations. X-Games assessment generated wide differentiation of resident performance quickly, inexpensively, and objectively. Although "Minnesota-nice" surgical staff may feel all GS trainees are "above average," objective assessment tells us otherwise.

Difficulties in Residency: An Examination of Clinical Rotations and Competencies Where Family Medicine Residents Most Often Struggle.

Remediation in residency is expensive; however, most research has focused on general approaches to remediation, with minimal investigation into whether there are patterns to the competencies or rotations that are most difficult for residents. Acquiring this information may improve future physician training and potentially reduce the frequency of resource-intensive remediation. We aimed to determine the competencies and rotations most challenging for family medicine residents, as defined by the number of assessments with flags (one or more competencies indicated as less than satisfactory). A secondary data analysis of archived resident files from a large Canadian family medicine residency program was conducted. Residents from six cohorts were reviewed (N=393) and flags on the in-training evaluation reports (ITERs) and summative periodic progress reports were recorded and summarized with descriptive statistics. One hundred forty-one residents (36%) received at least one flag during training. Rotations where learners received the most flags were: internal medicine (average 1.52±4.82 flags), urban family medicine (average 1.48±4.18), and obstetrics (average 1.07±3.80). For residents having at least one flag, competencies causing most difficulty included: professionalism (21.4%), clinical decision making (17.8%), and teamwork and communication (15.5%). The file review identified coronary care unit, internal medicine, obstetrics, and general surgery as those rotations (adjusted for length) where family medicine residents most often struggled. Furthermore, deficient clinical knowledge was not one of the main reasons that residents are flagged. These findings may inform programs about where to target resident supports and resources.

Validity of In-Training Evaluation Report (ITER) in Saudi Pediatric Residency Training. Single center experience

This article was migrated. The article was not marked as recommended. Objective: The study aimed to examine validity evidence of in-training evaluation report (ITER) at King Abdulaziz University Hospital (KAUH)'s pediatric residency program. Predictive validity of ITERs, to assess learner acquisitions of required competencies were studied. We used ITER scores to identify trainees with unprofessional conduct. Methods: One-year data for 35 residents in pediatric residency training program at KAUH were reviewed. Data were extracted from 360 ITERs completed by faculty members and specialists. ITERs' reliability was evaluated using Cronbach's alpha. ITERs scores were correlated with final Objective Structured Clinical Examinations (OSCE) and written exams scores. Professionalism composite scores from ITERs were compared with scores of trainees reported for unprofessional attitude. Results: Specialists had high variability in scoring competencies with average standard deviations (SD) of 0.85 compared with 0.15 by faculty. Reliability of ITER scores using Cronbach's alpha was 0.895. Good correlation was observed between competencies ranging between 0.86 to 0.29. There was fair correlation between ITER composite scores and those of final OSCE and written exams. There was no correlation between professionalism scores and residents reported for unprofessional conduct. Conclusion ITER scores had good validity and reliability despite variability observed between faculty and specialist when scoring competencies. ITERs failed to detect trainees with unprofessional conduct.

Does faculty development influence the quality of in-training evaluation reports in pharmacy?

BackgroundIn-training evaluation reports (ITERs) of student workplace-based learning are completed by clinical supervisors across various health disciplines. However, outside of medicine, the quality of submitted workplace-based assessments is largely uninvestigated. This study assessed the quality of ITERs in pharmacy and whether clinical supervisors could be trained to complete higher quality reports.MethodsA random sample of ITERs submitted in a pharmacy program during 2013–2014 was evaluated. These ITERs served as a historical control (control group 1) for comparison with ITERs submitted in 2015–2016 by clinical supervisors who participated in an interactive faculty development workshop (intervention group) and those who did not (control group 2). Two trained independent raters scored the ITERs using a previously validated nine-item scale assessing report quality, the Completed Clinical Evaluation Report Rating (CCERR). The scoring scale for each item is anchored at 1 (“not at all”) and 5 (“exemplary”), with 3 categorized as “acceptable”.ResultsMean CCERR score for reports completed after the workshop (22.9 ± 3.39) did not significantly improve when compared to prospective control group 2 (22.7 ± 3.63, p = 0.84) and were worse than historical control group 1 (37.9 ± 8.21, p = 0.001). Mean item scores for individual CCERR items were below acceptable thresholds for 5 of the 9 domains in control group 1, including supervisor documented evidence of specific examples to clearly explain weaknesses and concrete recommendations for student improvement. Mean item scores for individual CCERR items were below acceptable thresholds for 6 and 7 of the 9 domains in control group 2 and the intervention group, respectively.ConclusionsThis study is the first using CCERR to evaluate ITER quality outside of medicine. Findings demonstrate low baseline CCERR scores in a pharmacy program not demonstrably changed by a faculty development workshop, but strategies are identified to augment future rater training.