Ottawa Surgical Competency Operating Room Research Articles

Cognitive load theory in action: e-learning modules improve performance in simulation-based education. A pilot study.

Learners participating in simulation-based education may experience cognitive overload with potential detrimental effects to learning and performance. Multiple strategies have been proposed to mitigate this detrimental response. However, these strategies have not fully considered the potential benefits of using online platforms, such as accessibility, cost-effectiveness, efficiency, and scalability. Addressing this gap in the literature, preparatory online modules were developed by applying concepts from cognitive load theory and simulation-based education. This study assessed whether using preparatory online modules to deliver weekly pre-briefing content could impact cognitive load and performance. The participants were first-year postgraduate medical trainees participating in a simulation-based resuscitation curriculum. Fifty-three trainees were allocated to receive preparatory online modules (online modules group, n = 27) or not (control group, n = 26) during the course component of a simulation-based resuscitation curriculum. Then, these trainees participated in a simulation-based objective structured clinical examination (OSCE). Sources of cognitive load (intrinsic, extraneous, and germane) were measured using a modified cognitive load questionnaire. Performance was assessed using the Ottawa Surgical Competency Operating Room Evaluation. Data were analyzed with descriptive statistics, and principal component analysis. During the course component, the online modules group was found to have higher intrinsic and germane cognitive load, and lower extraneous cognitive load compared to the control group. During the OSCE, the online modules group scored significantly higher in performance scores (p = 0.0077, d = 0.39, 95% confidence interval = 0.10;0.68) compared to the control group. Principal component analysis supported the results obtained with the modified cognitive load questionnaire. Trainees using preparatory online modules during the course component of a simulation-based resuscitation curriculum experienced cognitive load changes consistent with cognitive optimization. This may have contributed to their superior performance in the subsequent OSCE. Future research should explore the long-term impacts of online preparatory training and consider potential barriers to implementation in diverse healthcare environments.

Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

IntroductionAssessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance. Materials and methodsA web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments. ResultsThe Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority. ConclusionsThis novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

3-D Printed Fracture Models Improve Resident Performance and Clinical Outcomes in Operative Fracture Management

To determine if preoperative examination of patient additive manufactured (AM) fracture models can be used to improve resident operative competency and patient outcomes. Prospective cohort study. Seventeen matched pairs of fracture fixation surgeries (for a total of 34 surgeries) were performed. Residents first performed a set of baseline surgeries (n = 17) without AM fracture models. The residents then performed a second set of surgeries randomly assigned to include an AM model (n = 11) or to omit it (n = 6). Following each surgery, the attending surgeon evaluated the resident using an Ottawa Surgical Competency Operating Room Evaluation (O-Score). The authors also recorded clinical outcomes including operative time, blood loss, fluoroscopy duration, and patient reported outcome measurement information system (PROMIS) scores of pain and function at 6 months. Single-center academic level one trauma center. Twelve orthopaedic residents, between postgraduate year (PGY) 2 and 5, participated in this study. Residents significantly improved their O-Scores between the first and second surgery when they trained with AM models for the second surgery (p = 0.004, 2.43 ± 0.79 versus 3.73 ± 0.64). Similar improvements were not observed in the control group (p = 0.916, 2.69 ± 0.69 versus 2.77 ± 0.36). AM model training also significantly improved clinical outcomes, including surgery time (p = 0.006), fluoroscopy exposure time (p = 0.002), and patient reported functional outcomes (p = 0.0006). Conclusions: Training with AM fracture models improves the performance of orthopaedic surgery residents during fracture surgery.

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BackgroundVideos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.MethodsA multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird’s-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.Results and discussionThe scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio–video quality, and questionable feasibility for assessing readiness for practice.ConclusionVideo-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio–video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

The use of advanced robotic simulation labs to advance and assess senior resident robotic skills and operating room leadership competency: a pilot study.

General surgery has the fastest growing robotic operative volume in the United States, but most robotic curricula are focused on basic psychomotor skills. There are limited curricula focused on advanced robotic technical and related non-technical skills. We describe a novel pilot curriculum for robotic hiatal hernia repair developed for senior surgical residents to provide training and standardized assessment of higher-order robotic technical and leadership skills. Twelve senior residents, post-graduate year (PGY) 4 & 5, participated in a robotic hiatal hernia repair skills curriculum. Residents completed a pre- and post-survey on confidence and ability ratings on a 5-point Likert-type Scale, and a knowledge assessment. An informal faculty-led didactic was provided prior to the simulation. Residents were scored on two validated assessment tools: Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) and Global Ratings Scale of Operative Performance (GRS) by faculty proctors. Confidence in ability to independently complete a robotic hiatal hernia case increased from mean of 2.6 ± 0.8 to 3.3 ± 0.6 (p = 0.0007). Following the simulation, residents reported increased overall confidence and ability to operate independently with mean scores of 3.3 ± 0.8 and 3.8 ± 0.9, respectively. Mean O-SCORE and GRS scores were 3.6 (range 2 - 4) and 25.4 (range 12 - 31), respectively. Number of prior live robotic cases was strongly positively correlated to O-SCORE (R = 0.84, p = 0.0006) and GRS (R = 0.88, p = 0.0002). Our pilot study suggests live-operative robotic training is not sufficient alone for advanced robotic skill training. Simulations such as this can be used to (1) practice advanced robotic technical and relevant non-technical skills such as communication and operating room leadership in a low stake setting and (2) assess residents in a standardized environment to eventually evaluate robotic competency.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Validation of a Surgical Objective Structured Clinical Examination (S-OSCE) Using Convergent, Divergent, And Trainee-Based Assessments of Fidelity

Describe the validation of a surgical objective structured clinical examination (S-OSCE) for the purpose of competency assessment based on the Royal College of Canada's Can-MEDS framework. A surgical OSCE was developed to evaluate the management of common orthopedic surgical problems. The scores derived from this S-OSCE were compared to Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), a validated entrustability assessment, to establish convergent validity. The S-OSCE scores were compared to Orthopedic In-Training Examination (OITE) scores to evaluate divergent validity. Resident evaluations of the clinical encounter with a standardized patient and the operative procedure were scored on a 10-point Likert scale for fidelity. A tertiary level academic teaching hospital. 21 postgraduate year 2 to 5 trainees of a 5-year Canadian orthopedic residency program creating 160 operative case performances for review. There were 5 S-OSCE days, over a 4-year period (2016-2019) encompassing a variety of surgical procedures. Performance on the S-OSCE correlated strongly with the O-SCORE (Pearson correlation coefficient 0.88), and a linear regression analysis correlated moderately with year of training (R² = 0.5345). The Pearson correlation coefficient between the S-OSCE and OITE scores was 0.57. There was a significant increase in the average OITE score after the introduction of the surgical OSCE. Resident fidelity ratings were available from 16 residents encompassing 8 different surgical cases. The average score for the overall simulation (8.0±1.6) was significantly higher than the cadaveric surgical simulation (6.5 ± 0.8) (p < 0.001) CONCLUSIONS: The S-OSCE scores correlate strongly with an established form of assessment demonstrating convergent validity. The correlation between the S-OSCE and OITE scores was less, demonstrating divergent validity. Although residents rank the overall simulation highly, the fidelity of the cadaveric simulation may need improvement. Administration of a surgical OSCE can be used to evaluate preoperative and intraoperative decision making and complement other forms of assessment.

Reframing the O-SCORE as a Retrospective Supervision Scale Using Validity Theory.

Reframing the O-SCORE as a Retrospective Supervision Scale Using Validity Theory.

Exploring the use of rating scales with entrustment anchors in workplace-based assessment.

Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale. A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes. Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not. By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace.

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

The Development of a Simulated Umbilical Line Insertion Model and Curriculum in the Neonatal Intensive Care Unit.

BackgroundInsertion of an umbilical venous catheter (UVC) is a required skill for pediatric residents to learn and perform effectively. However, there is known variability in the ability of residents to perform this essential neonatal skill.ObjectiveThe objective of our study was to create a competency-based curriculum for umbilical vein catheter insertion using a human umbilical tissue simulated model, and to assess the feasibility of the curriculum on resident learners during their neonatology rotations.MethodsWe evaluated the curriculum by assessment of resident learning, reactions, and behaviours. Performance was assessed using the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE).ResultsA total of 14 residents were included for analysis. The majority were ‘senior’ residents (postgraduate year (PGY)-3 and PGY-4 n = 10; PGY-1 n =4), and they reported a wide range of previous experience with UVC insertion prior to this curriculum implementation. The residents’ reaction to the curriculum was overwhelmingly positive. All residents maintained or improved in their knowledge assessment. O-SCORE results showed improvement in UVC insertion before and after curriculum completion for both junior (2.5 +/- 0.71 to 4.5 +/- 0.41) and senior (3.55 +/- 0.42 to 4.95 +/- 0.15, p < 0.001) residents. The mean improvement in O-SCORE was greater for junior residents than senior residents.ConclusionThe results of this study demonstrate the feasibility and emerging impact of a competency-based curriculum using simulation for procedural skills.

Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution.

Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships. During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability. A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70). The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.

Difference in Resident Versus Attending Perspective of Competency and Autonomy During Arthroscopic Rotator Cuff Repairs.

Background:A noted deficiency in orthopaedic resident education is a lack of intraoperative autonomy; however, no studies exist evaluating this issue. The purpose of this study was to determine whether there is a difference between resident and attending perception of resident competency and autonomy during arthroscopic rotator cuff repairs and whether increased perceived competency leads to more autonomy.Methods:This study included 21 orthopaedic residents and 7 attendings from a single residency program. A survey was developed that included the previously validated Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) and implemented novel arthroscopic rotator cuff repair questions concerning 5 key procedural steps determined by attendings. The survey assessed resident and attending perception of percent opportunity given and percent completed of each step and whether the resident could complete the surgery independently. Paired comparisons were conducted using the Wilcoxon signed-rank test. Agreement between residents and attendings was calculated using a linear-weighted Gwet's AC2. A secondary analysis investigated resident perception of autonomy stratified by attending perception of competency using Kruskal-Wallis tests.Results:One hundred forty-two surveys were completed over one academic year. Residents reported a 4-point higher median O-SCORE (34) than attendings (30; p < 0.01; agreement = 0.63). Residents perceived less opportunity compared with attendings, with a median opportunity to complete each step of 54% vs. 70% (p < 0.01; agreement = 0.39). Residents also perceived lower percent completed of the key steps compared with attendings, with medians of 52% vs. 61% (p < 0.01; agreement = 0.37). Resident perceived opportunity increased with higher attending reported O-SCOREs (p < 0.01) and percent completion of the key steps (p < 0.01). No statistically significant increase in perceived opportunity was observed the with post-graduate year (PGY) level (p = 0.35).Discussion:This study demonstrated a significant difference in perception between residents and attendings regarding resident competency and autonomy during arthroscopic rotator cuff repairs. It also demonstrated that with increasing attending perception of competency, there was an increasing resident perception of autonomy, but there was no statistically significant difference of perceived autonomy based on the PGY-level.

Use of a Secure Web-Based Data Management Platform to Track Resident Operative Performance and Program Educational Quality Over Time

In surgery residency programs, Accreditation Council for Graduate Medical Education mandated performance assessment can include assessment in the operating room to demonstrate that necessary quality and autonomy goals are achieved by the conclusion of training. For the past 3 years, our institution has used The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) instrument to assess and track operative skills. Evaluation is accomplished in near real-time using a secure web-based platform for data management and analytics (Firefly). Simultaneous to access of the platform's case logging function, the O-SCORE instrument is delivered to faculty members for rapid completion, facilitating quality, and timeliness of feedback. We sought to demonstrate the platform's utility in detecting operative performance changes over time in response to focused educational interventions based on stored case log and O-SCORE data. Stored resident performance assessments for the most frequently performed laparoscopic procedures (cholecystectomy, appendectomy, inguinal hernia repair, ventral hernia repair) were examined for 3 successive academic years (2016-2019). During this time, 4 of 36 residents had received program-assigned supplemental simulation training to improve laparoscopic skills. O-SCORE data for these residents were extracted from peer data, which were used for comparisons. Assigned training consisted of a range of videoscopic and virtual reality skills drills with performance objectives. O-SCORE responses were converted to integers and autonomy scores for items pertaining to technical skill were compared before and after educational interventions (Student's t-tests). These scores were also compared to aggregate scores in the nonintervention group. Bayesian-modeled learning curves were used to characterize patterns of improvement over time. University of Massachusetts Medical School-Baystate Surgery Residency and Baystate Medical Center PARTICIPANTS: General surgery residents (n = 36) RESULTS: During the period of review, 3325 resident cases were identified meeting the case type criteria. As expected, overall autonomy increased with the number of cases performed. The 4 residents who had been assigned supplemental training (6-18 months) had preintervention score averages that were lower than that of the nonintervention group (2.25 ± 0.43 vs 3.57 ± 1.02; p < 0.0001). During the respective intervention periods, all 4 residents improved autonomy scores (increase to 3.40 ± 0.61; p < 0.0001). Similar improvements were observed for tissue handling, instrument handling, bimanual dexterity, visuospatial skill, and operative efficiency component skills. Postintervention scores were not significantly different compared to scores for the non-intervention group. Bayesian-modeled learning curves showed a similar pattern of postintervention performance improvement. The data management platform proved to be an effective tool to track responses to supplemental training that was deemed necessary to close defined skills gaps in laparoscopic surgery. This could be seen both in individual and in aggregated data. We were gratified that at the conclusion of the supplemental training, O-SCORE results for the intervention group were not different than those seen in the non-intervention group.

P042: Workplace-based assessment in emergency medicine: how do physicians use entrustment anchors?

Introduction: Competency based medical education (CBME) has triggered widespread utilization of workplace-based assessment (WBA) tools in postgraduate training programs. These WBAs predominately use rating scales with entrustment anchors, such as the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). However, little is known about the factors that influence a supervising physician's decision to assign a particular rating on scales using entrustment anchors. This study aimed to identify the factors that influence supervisors’ ratings of trainees using WBA tools with entrustment anchors at the time of assessment and to explore the experiences with and challenges of using entrustment anchors in the emergency department (ED). Methods: A convenience sample of full-time emergency medicine (EM) faculty were recruited from two sites within a single academic Canadian EM hospital system. Fifty semi-structured interviews were conducted with EM physicians within two hours of completing a WBA for an EM trainee. Interviews were audio-recorded, transcribed verbatim, and independently analyzed by two members of the research team. Themes were stratified by trainee level, rating and task. Results: Interviews involved 73% (27/37) of all EM staff and captured assessments completed on 83% (37/50) of EM trainees. The mean WBA rating of studied samples was 4.34 ± 0.77 (2 to 5), which was similar to the mean rating of all WBAs completed during the study period. Overall, six major factors were identified that influenced staff WBA ratings: amount of guidance required, perceived competence through discussion and questioning, trainee experience, clinical context, past experience working with the trainee, and perceived confidence. The majority of staff denied struggling to assign ratings. However, when they did struggle, it involved the interpretation of WBA anchors and their application to the clinical context in the ED. Conclusion: Several factors appear to be taken into account by clinical supervisors when they make decisions regarding the particular rating that they will assign a trainee on a WBA that uses entrustment anchors. Not all of these factors are specific to that particular clinical encounter. The results from this study further our understanding on the use of entrustment anchors within the ED and may facilitate faculty development regarding WBA completion as we move forward in CBME.

Education Management Platform Enables Delivery and Comparison of Multiple Evaluation Types

The purpose of this study was to determine whether an automated platform for evaluation selection and delivery would increase participation from surgical teaching faculty in submitting resident operative performance evaluations. We built a HIPAA-compliant, web-based platform to track resident operative assignments and to link embedded evaluation instruments to procedure type. The platform matched appropriate evaluations to surgeons' scheduled procedures, and delivered multiple evaluation types, including Ottawa Surgical Competency Operating Room Evaluation (O-Score) evaluations and Operative Performance Rating System (OPRS) evaluations. Prompts to complete evaluations were made through a system of automatic electronic notifications. We compared the time spent in the platform to achieve evaluation completion. As a metric for the platform's effect on faculty participation, we considered a task that would typically be infeasible without workflow optimization: the evaluator could choose to complete multiple, complementary evaluations for the same resident in the same case. For those cases with multiple evaluations, correlation was analyzed by Spearman rank test. Evaluation data were compared between PGY levels using repeated measures ANOVA. The study took place at 4 general surgery residency programs: The University of Massachusetts Medical School-Baystate, the University of Connecticut School or Medicine, the University of Iowa Carver College of Medicine, and Maimonides Medical Center. From March 2017 to February 2019, the study included 70 surgical teaching faculty and 101 general surgery residents. Faculty completed 1230 O-Score evaluations and 106 OPRS evaluations. Evaluations were completed quickly, with a median time of 36 ± 18 seconds for O-Score evaluations, and 53 ± 51 seconds for OPRS evaluations. 89% of O-Score and 55% of OPRS evaluations were completed without optional comments within one minute, and 99% of O-Score and 82% of OPRS evaluations were completed within 2 minutes. For cases eligible for both evaluation types, attendings completed both evaluations on 74 of 221 (33%) of these cases. These paired evaluations strongly correlated on resident performance (Spearman coefficient = 0.84, p < 0.00001). Both evaluation types stratified operative skill level by program year (p < 0.00001). Evaluation initiatives can be hampered by the challenge of making multiple surgical evaluation instruments available when needed for appropriate clinical situations, including specific case types. As a test of the optimized evaluation workflow, and to lay the groundwork for future data-driven design of evaluations, we tested the impact of simultaneously delivering 2 evaluation instruments via a secure web-based education platform. We measured the evaluation completion rates of faculty surgeon evaluators when rating resident operative performance, and how effectively the results of evaluation could be analyzed and compared, taking advantage of a highly integrated management of the evaluative information.

Picking the Right Tool for the Job: A Reliability Study of 4 Assessment Tools for Central Venous Catheter Insertion.

Determining procedural competence requires psychometrically sound assessment tools. A variety of instruments are available to determine procedural performance for central venous catheter (CVC) insertion, but it is not clear which ones should be used in the context of competency-based medical education. We compared several commonly used instruments to determine which should be preferentially used to assess competence in CVC insertion. Junior residents completing their first intensive care unit rotation between July 31, 2006, and March 9, 2007, were video-recorded performing CVC insertion on task trainer mannequins. Between June 1, 2016, and September 30, 2016, 3 experienced raters judged procedural competence on the historical video recordings of resident performance using 4 separate tools, including an itemized checklist, Objective Structured Assessment of Technical Skills (OSATS), a critical error assessment tool, and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). Generalizability theory (G-theory) was used to compare the performance characteristics among the tools. A decision study predicted the optimal testing environment using the tools. At the time of the original recording, 127 residents rotated through intensive care units at the University of Calgary, Alberta, Canada. Seventy-seven of them (61%) met inclusion criteria, and 55 of those residents (71%) agreed to participate. Results from the generalizability study (G-study) demonstrated that scores from O-SCORE and OSATS were the most dependable. Dependability could be maintained for O-SCORE and OSATS with 2 raters. Our results suggest that global rating scales, such as the OSATS or the O-SCORE tools, should be preferentially utilized for assessment of competence in CVC insertion.

Obstacles Affecting the Implementation of the O-SCORE for Assessment of Orthopedic Surgical Skills Competency

There is a need for meaningful and reliable measures of surgical competency in residency education. The goal of the current study is to incorporate the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) into the process of resident evaluation at our institution and to assess the feasibility and effectiveness of its use through a web-based platform. This is a feasibility study that prospectively assesses the implementation of a web-based O-SCORE at our institution. Over a 16-week period, 19 orthopedic surgery residents (PGY2-PGY5) participated in a quality improvement study, which involved collecting 2 feedback forms per week. Each form consisted of a resident form and a linked attending form. At the conclusion of the 16-week trial period, residents and faculty members were asked to complete a survey about their perceptions of the O-SCORE program. An academic medical center. The study included only residents in postgraduate training years (PGY) 2 through 5 (n = 20) and attendings (n = 37). During the 16-week study period, 608 resident surveys were requested for the 19 participating residents, of which 404 surveys (66.5%) were completed. Faculty completed 207 of 326 surveys for an overall compliance rate of 63.5%. The O-SCORE was able to significantly differentiate between all training years (p < 0.0001) with the exception of PGY3 residents when compared to PGY4 residents. Overall, residents and faculty found the program valuable and feasible. Resident and faculty perception of the value of the O-SCORE correlated with compliance rate of the O-SCORE surveys. This study demonstrates that implementation of an immediate feedback program utilizing an electronic platform is achievable and offers reproducible construct validity. However, issues affecting compliance among both residents and faculty physicians must temper optimism for the program and should be systematically addressed to allow for successful implementation.

Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score

ABSTRACTConstruct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency. Background: Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment. Approach: Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated. Results: Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72–1.00; p < .001; percentage agreement =90%–100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees. Conclusions: A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.

Development and implementation of a mobile version of the O-SCORE assessment tool and case log for competency-based assessment in urology residency training: An initial assessment of utilization and acceptance among residents and faculty.

In competency-based models of residency training, work-based assessments of residents' technical skills are essential both for providing formative feedback and for assessing surgical competence. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) is a previously validated paper-based evaluation tool created to assess a surgical trainees' operative competence. To address some of the barriers to assessment, we developed and implemented a mobile application that combines the O-SCORE with a surgical case log. A description of the development implementation process for the mobile O-SCORE and case log is provided. Following implementation, a survey was developed and administered electronically to all faculty and residents within the University of Ottawa's Division of Urology to assess user perceptions and utilization of the application. The survey was administered and data collected via Survey Monkey. The overall response rate was 94%. The majority of residents (94%) reported that it was easy to log cases with the application and 81% felt that it had a positive impact on their training; 75% of faculty were willing or very willing to complete evaluations when assigned and 66% felt that the application had a positive effect on the quality of feedback they provided. Overall, faculty and residents felt that our mobile O-SCORE application was user-friendly and valuable as both a surgical log and assessment tool. With surgical programs moving towards competency-based models of training and assessment, the O-SCORE mobile application represents a practical electronic surgical log and work-based assessment instrument that can be easily adopted into any surgical training program.