Operating Room Evaluation Research Articles

Wireless In-Ear Communication for Total Joint Arthroplasty: A Simulated Operating Room Evaluation

BackgroundEffective surgical team communication is a key component of patient safety. However, communication failure is common and has previously been identified as one of the leading causes of error. Nonetheless, there has been a lack of innovation in this domain. Contemporary communication technology may hamper loud noises frequently encountered during total joint arthroplasty surgery to allow the surgical team to communicate safely and effectively. The objective of this study was to assess whether the use of in-ear communication devices during simulated arthroplasty surgery could improve communication compared to standard of loud voice communication. MethodsThis was a randomized simulation study designed to evaluate differences in communication. Participants were recruited to engage in simulations of total joint arthroplasty with in-ear wireless communication devices and traditional methods. Data collected included participant demographics, sound level exposure, and communication effectiveness using a created list of phrases commonly involved in arthroplasty surgery. A System Usability Scale questionnaire and survey comparing participant experiences was given. ResultsFifteen participants were recruited. The median age was 29 years (standard deviation 2.9) and 66% were women. There were significant differences in the percent of correctly identified phrases (78.6% vs 44%), mean effectiveness (7.9/10 vs 4.9/10) and mean clarity (8/10 vs 4/10) in favor for the wireless in-ear communication simulation (P < .001 for all). Out of the 10 questions in the System Usability Scale questionnaire, 4 exhibited significant differences, favoring the use of in-ear communication, while the others showed no difference. The mean level of sound recorded was 86 dB in the in-ear device and 87.6 dB in the control group (P = .680). ConclusionsWireless in-ear communication appears to be safe and effective during total joint arthroplasty surgery which may promote efficiency and patient safety. Further, in-ear communication systems can dampen the loud and potentially damaging noise in orthopaedic procedures and may protect surgeons exposed to loud noises.

Cognitive load theory in action: e-learning modules improve performance in simulation-based education. A pilot study.

Learners participating in simulation-based education may experience cognitive overload with potential detrimental effects to learning and performance. Multiple strategies have been proposed to mitigate this detrimental response. However, these strategies have not fully considered the potential benefits of using online platforms, such as accessibility, cost-effectiveness, efficiency, and scalability. Addressing this gap in the literature, preparatory online modules were developed by applying concepts from cognitive load theory and simulation-based education. This study assessed whether using preparatory online modules to deliver weekly pre-briefing content could impact cognitive load and performance. The participants were first-year postgraduate medical trainees participating in a simulation-based resuscitation curriculum. Fifty-three trainees were allocated to receive preparatory online modules (online modules group, n = 27) or not (control group, n = 26) during the course component of a simulation-based resuscitation curriculum. Then, these trainees participated in a simulation-based objective structured clinical examination (OSCE). Sources of cognitive load (intrinsic, extraneous, and germane) were measured using a modified cognitive load questionnaire. Performance was assessed using the Ottawa Surgical Competency Operating Room Evaluation. Data were analyzed with descriptive statistics, and principal component analysis. During the course component, the online modules group was found to have higher intrinsic and germane cognitive load, and lower extraneous cognitive load compared to the control group. During the OSCE, the online modules group scored significantly higher in performance scores (p = 0.0077, d = 0.39, 95% confidence interval = 0.10;0.68) compared to the control group. Principal component analysis supported the results obtained with the modified cognitive load questionnaire. Trainees using preparatory online modules during the course component of a simulation-based resuscitation curriculum experienced cognitive load changes consistent with cognitive optimization. This may have contributed to their superior performance in the subsequent OSCE. Future research should explore the long-term impacts of online preparatory training and consider potential barriers to implementation in diverse healthcare environments.

Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

IntroductionAssessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance. Materials and methodsA web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments. ResultsThe Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority. ConclusionsThis novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

3-D Printed Fracture Models Improve Resident Performance and Clinical Outcomes in Operative Fracture Management

To determine if preoperative examination of patient additive manufactured (AM) fracture models can be used to improve resident operative competency and patient outcomes. Prospective cohort study. Seventeen matched pairs of fracture fixation surgeries (for a total of 34 surgeries) were performed. Residents first performed a set of baseline surgeries (n = 17) without AM fracture models. The residents then performed a second set of surgeries randomly assigned to include an AM model (n = 11) or to omit it (n = 6). Following each surgery, the attending surgeon evaluated the resident using an Ottawa Surgical Competency Operating Room Evaluation (O-Score). The authors also recorded clinical outcomes including operative time, blood loss, fluoroscopy duration, and patient reported outcome measurement information system (PROMIS) scores of pain and function at 6 months. Single-center academic level one trauma center. Twelve orthopaedic residents, between postgraduate year (PGY) 2 and 5, participated in this study. Residents significantly improved their O-Scores between the first and second surgery when they trained with AM models for the second surgery (p = 0.004, 2.43 ± 0.79 versus 3.73 ± 0.64). Similar improvements were not observed in the control group (p = 0.916, 2.69 ± 0.69 versus 2.77 ± 0.36). AM model training also significantly improved clinical outcomes, including surgery time (p = 0.006), fluoroscopy exposure time (p = 0.002), and patient reported functional outcomes (p = 0.0006). Conclusions: Training with AM fracture models improves the performance of orthopaedic surgery residents during fracture surgery.

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BackgroundVideos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.MethodsA multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird’s-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.Results and discussionThe scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio–video quality, and questionable feasibility for assessing readiness for practice.ConclusionVideo-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio–video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

Application of Participatory Teaching in Operating Room Nursing and Evaluation of Its Effectiveness

Application of Participatory Teaching in Operating Room Nursing and Evaluation of Its Effectiveness

The use of advanced robotic simulation labs to advance and assess senior resident robotic skills and operating room leadership competency: a pilot study.

General surgery has the fastest growing robotic operative volume in the United States, but most robotic curricula are focused on basic psychomotor skills. There are limited curricula focused on advanced robotic technical and related non-technical skills. We describe a novel pilot curriculum for robotic hiatal hernia repair developed for senior surgical residents to provide training and standardized assessment of higher-order robotic technical and leadership skills. Twelve senior residents, post-graduate year (PGY) 4 & 5, participated in a robotic hiatal hernia repair skills curriculum. Residents completed a pre- and post-survey on confidence and ability ratings on a 5-point Likert-type Scale, and a knowledge assessment. An informal faculty-led didactic was provided prior to the simulation. Residents were scored on two validated assessment tools: Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) and Global Ratings Scale of Operative Performance (GRS) by faculty proctors. Confidence in ability to independently complete a robotic hiatal hernia case increased from mean of 2.6 ± 0.8 to 3.3 ± 0.6 (p = 0.0007). Following the simulation, residents reported increased overall confidence and ability to operate independently with mean scores of 3.3 ± 0.8 and 3.8 ± 0.9, respectively. Mean O-SCORE and GRS scores were 3.6 (range 2 - 4) and 25.4 (range 12 - 31), respectively. Number of prior live robotic cases was strongly positively correlated to O-SCORE (R = 0.84, p = 0.0006) and GRS (R = 0.88, p = 0.0002). Our pilot study suggests live-operative robotic training is not sufficient alone for advanced robotic skill training. Simulations such as this can be used to (1) practice advanced robotic technical and relevant non-technical skills such as communication and operating room leadership in a low stake setting and (2) assess residents in a standardized environment to eventually evaluate robotic competency.

The Use of Formative Assessment in Postgraduate Urology Training: A Systematic Review.

Formative assessment is an essential component of surgical training. However, it is not usually a mandatory component in postgraduate curricula. The purpose of this study is to identify and evaluate how formative assessments are integrated into postgraduate urology training in programs across the globe. This study consisted of a systemic review to see how formative assessments are being implemented in various urology programs globally. A total of 427 articles were identified for the literature review. Of these, only 10 were included and critically appraised. These studies explored various techniques for exploration of formative assessments in urology training programs, which included established tools, such as portfolio reviews, and direct observations of procedure skills (DOPS); novel tools, including the Dutch urology practical skills (D-UPS) program and Ottawa surgical competency operating room evaluation (O-SCORE); and curricular models. Nine of the 10 articles favored their potential utility in formative assessments. Current literature involving formative assessments in postgraduate urology programs is scarce, and available resources have a high heterogeneity between them. More structured formative assessments need to be incorporated into surgical training programs, and affiliated training institutions should be encouraged to integrate them into their curricula.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Validation of a Surgical Objective Structured Clinical Examination (S-OSCE) Using Convergent, Divergent, And Trainee-Based Assessments of Fidelity

Describe the validation of a surgical objective structured clinical examination (S-OSCE) for the purpose of competency assessment based on the Royal College of Canada's Can-MEDS framework. A surgical OSCE was developed to evaluate the management of common orthopedic surgical problems. The scores derived from this S-OSCE were compared to Ottawa Surgical Competency Operating Room Evaluation (O-SCORE), a validated entrustability assessment, to establish convergent validity. The S-OSCE scores were compared to Orthopedic In-Training Examination (OITE) scores to evaluate divergent validity. Resident evaluations of the clinical encounter with a standardized patient and the operative procedure were scored on a 10-point Likert scale for fidelity. A tertiary level academic teaching hospital. 21 postgraduate year 2 to 5 trainees of a 5-year Canadian orthopedic residency program creating 160 operative case performances for review. There were 5 S-OSCE days, over a 4-year period (2016-2019) encompassing a variety of surgical procedures. Performance on the S-OSCE correlated strongly with the O-SCORE (Pearson correlation coefficient 0.88), and a linear regression analysis correlated moderately with year of training (R² = 0.5345). The Pearson correlation coefficient between the S-OSCE and OITE scores was 0.57. There was a significant increase in the average OITE score after the introduction of the surgical OSCE. Resident fidelity ratings were available from 16 residents encompassing 8 different surgical cases. The average score for the overall simulation (8.0±1.6) was significantly higher than the cadaveric surgical simulation (6.5 ± 0.8) (p < 0.001) CONCLUSIONS: The S-OSCE scores correlate strongly with an established form of assessment demonstrating convergent validity. The correlation between the S-OSCE and OITE scores was less, demonstrating divergent validity. Although residents rank the overall simulation highly, the fidelity of the cadaveric simulation may need improvement. Administration of a surgical OSCE can be used to evaluate preoperative and intraoperative decision making and complement other forms of assessment.

Reframing the O-SCORE as a Retrospective Supervision Scale Using Validity Theory.

Reframing the O-SCORE as a Retrospective Supervision Scale Using Validity Theory.

Exploring the use of rating scales with entrustment anchors in workplace-based assessment.

Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale. A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes. Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not. By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

Mentored undergraduate operating room teaching during the orthopedic trauma curriculum-No evidence of gender differences.

In many surgical disciplines, the interest among medical students to pursue asurgical career decreases during their medical studies. The same goes for students after graduation. The aim of our cohort study was to evaluate the operating room (OR) experiences of medical students during our curriculum. Over the course of one year 217 senior year medical students were included in our study. All of them took part in our training program for senior year medical students, which consisted of a1-week clinical rotation including visits to the OR. We developed aLikert-scaled questionnaire, which was evaluated anonymously; free text answers were also possible. Prior to the analysis of the sex and age differences, we confirmed that the evaluation scale provided acoherent measure of the OR evaluation. As afirst proxy, we conducted aseries of Spearman correlations which revealed high intercorrelations between all of the six items of the questionnaire, r(154) = 0.53 to r(154) = 0.94, all p < 0.001. These high intercorrelations transferred into avery high consistency of the six questions that evaluate the OR teaching; Cronbach's α = 0.95. There was no main effect of sex, F (1,146) = 2.19, p = 0.141. However, there was amain effect of age, F (2,146) = 3.75, p = 0.026, indicating that older participants evaluated the OR teaching more positively. Finally, there was no correlation between sex and age group, F (2,146) < 1, indicating that the effect of age on the evaluation score was equally pronounced for female and male participants. The aim of this study was to answer the question how mentored OR teaching during the orthopedic trauma curriculum is evaluated by medical students, and whether there are gender-specific differences. For this purpose we prospectively evaluated senior year medical students over aperiod of 12months during the orthopedic trauma curriculum with questionnaires. The medical students rated the mentored OR visits mainly positive. Previously published literature as well as our presented data indicate that the interest of medical students in starting asurgical career can only be increased if negative influencing factors are reduced. This includes especially positive communication with medical students and in daily professional interaction.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace.

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

The Development of a Simulated Umbilical Line Insertion Model and Curriculum in the Neonatal Intensive Care Unit.

BackgroundInsertion of an umbilical venous catheter (UVC) is a required skill for pediatric residents to learn and perform effectively. However, there is known variability in the ability of residents to perform this essential neonatal skill.ObjectiveThe objective of our study was to create a competency-based curriculum for umbilical vein catheter insertion using a human umbilical tissue simulated model, and to assess the feasibility of the curriculum on resident learners during their neonatology rotations.MethodsWe evaluated the curriculum by assessment of resident learning, reactions, and behaviours. Performance was assessed using the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE).ResultsA total of 14 residents were included for analysis. The majority were ‘senior’ residents (postgraduate year (PGY)-3 and PGY-4 n = 10; PGY-1 n =4), and they reported a wide range of previous experience with UVC insertion prior to this curriculum implementation. The residents’ reaction to the curriculum was overwhelmingly positive. All residents maintained or improved in their knowledge assessment. O-SCORE results showed improvement in UVC insertion before and after curriculum completion for both junior (2.5 +/- 0.71 to 4.5 +/- 0.41) and senior (3.55 +/- 0.42 to 4.95 +/- 0.15, p < 0.001) residents. The mean improvement in O-SCORE was greater for junior residents than senior residents.ConclusionThe results of this study demonstrate the feasibility and emerging impact of a competency-based curriculum using simulation for procedural skills.

Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution.

Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships. During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability. A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70). The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.

Difference in Resident Versus Attending Perspective of Competency and Autonomy During Arthroscopic Rotator Cuff Repairs.

Background:A noted deficiency in orthopaedic resident education is a lack of intraoperative autonomy; however, no studies exist evaluating this issue. The purpose of this study was to determine whether there is a difference between resident and attending perception of resident competency and autonomy during arthroscopic rotator cuff repairs and whether increased perceived competency leads to more autonomy.Methods:This study included 21 orthopaedic residents and 7 attendings from a single residency program. A survey was developed that included the previously validated Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) and implemented novel arthroscopic rotator cuff repair questions concerning 5 key procedural steps determined by attendings. The survey assessed resident and attending perception of percent opportunity given and percent completed of each step and whether the resident could complete the surgery independently. Paired comparisons were conducted using the Wilcoxon signed-rank test. Agreement between residents and attendings was calculated using a linear-weighted Gwet's AC2. A secondary analysis investigated resident perception of autonomy stratified by attending perception of competency using Kruskal-Wallis tests.Results:One hundred forty-two surveys were completed over one academic year. Residents reported a 4-point higher median O-SCORE (34) than attendings (30; p < 0.01; agreement = 0.63). Residents perceived less opportunity compared with attendings, with a median opportunity to complete each step of 54% vs. 70% (p < 0.01; agreement = 0.39). Residents also perceived lower percent completed of the key steps compared with attendings, with medians of 52% vs. 61% (p < 0.01; agreement = 0.37). Resident perceived opportunity increased with higher attending reported O-SCOREs (p < 0.01) and percent completion of the key steps (p < 0.01). No statistically significant increase in perceived opportunity was observed the with post-graduate year (PGY) level (p = 0.35).Discussion:This study demonstrated a significant difference in perception between residents and attendings regarding resident competency and autonomy during arthroscopic rotator cuff repairs. It also demonstrated that with increasing attending perception of competency, there was an increasing resident perception of autonomy, but there was no statistically significant difference of perceived autonomy based on the PGY-level.

Use of a Secure Web-Based Data Management Platform to Track Resident Operative Performance and Program Educational Quality Over Time

In surgery residency programs, Accreditation Council for Graduate Medical Education mandated performance assessment can include assessment in the operating room to demonstrate that necessary quality and autonomy goals are achieved by the conclusion of training. For the past 3 years, our institution has used The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) instrument to assess and track operative skills. Evaluation is accomplished in near real-time using a secure web-based platform for data management and analytics (Firefly). Simultaneous to access of the platform's case logging function, the O-SCORE instrument is delivered to faculty members for rapid completion, facilitating quality, and timeliness of feedback. We sought to demonstrate the platform's utility in detecting operative performance changes over time in response to focused educational interventions based on stored case log and O-SCORE data. Stored resident performance assessments for the most frequently performed laparoscopic procedures (cholecystectomy, appendectomy, inguinal hernia repair, ventral hernia repair) were examined for 3 successive academic years (2016-2019). During this time, 4 of 36 residents had received program-assigned supplemental simulation training to improve laparoscopic skills. O-SCORE data for these residents were extracted from peer data, which were used for comparisons. Assigned training consisted of a range of videoscopic and virtual reality skills drills with performance objectives. O-SCORE responses were converted to integers and autonomy scores for items pertaining to technical skill were compared before and after educational interventions (Student's t-tests). These scores were also compared to aggregate scores in the nonintervention group. Bayesian-modeled learning curves were used to characterize patterns of improvement over time. University of Massachusetts Medical School-Baystate Surgery Residency and Baystate Medical Center PARTICIPANTS: General surgery residents (n = 36) RESULTS: During the period of review, 3325 resident cases were identified meeting the case type criteria. As expected, overall autonomy increased with the number of cases performed. The 4 residents who had been assigned supplemental training (6-18 months) had preintervention score averages that were lower than that of the nonintervention group (2.25 ± 0.43 vs 3.57 ± 1.02; p < 0.0001). During the respective intervention periods, all 4 residents improved autonomy scores (increase to 3.40 ± 0.61; p < 0.0001). Similar improvements were observed for tissue handling, instrument handling, bimanual dexterity, visuospatial skill, and operative efficiency component skills. Postintervention scores were not significantly different compared to scores for the non-intervention group. Bayesian-modeled learning curves showed a similar pattern of postintervention performance improvement. The data management platform proved to be an effective tool to track responses to supplemental training that was deemed necessary to close defined skills gaps in laparoscopic surgery. This could be seen both in individual and in aggregated data. We were gratified that at the conclusion of the supplemental training, O-SCORE results for the intervention group were not different than those seen in the non-intervention group.

P042: Workplace-based assessment in emergency medicine: how do physicians use entrustment anchors?

Introduction: Competency based medical education (CBME) has triggered widespread utilization of workplace-based assessment (WBA) tools in postgraduate training programs. These WBAs predominately use rating scales with entrustment anchors, such as the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). However, little is known about the factors that influence a supervising physician's decision to assign a particular rating on scales using entrustment anchors. This study aimed to identify the factors that influence supervisors’ ratings of trainees using WBA tools with entrustment anchors at the time of assessment and to explore the experiences with and challenges of using entrustment anchors in the emergency department (ED). Methods: A convenience sample of full-time emergency medicine (EM) faculty were recruited from two sites within a single academic Canadian EM hospital system. Fifty semi-structured interviews were conducted with EM physicians within two hours of completing a WBA for an EM trainee. Interviews were audio-recorded, transcribed verbatim, and independently analyzed by two members of the research team. Themes were stratified by trainee level, rating and task. Results: Interviews involved 73% (27/37) of all EM staff and captured assessments completed on 83% (37/50) of EM trainees. The mean WBA rating of studied samples was 4.34 ± 0.77 (2 to 5), which was similar to the mean rating of all WBAs completed during the study period. Overall, six major factors were identified that influenced staff WBA ratings: amount of guidance required, perceived competence through discussion and questioning, trainee experience, clinical context, past experience working with the trainee, and perceived confidence. The majority of staff denied struggling to assign ratings. However, when they did struggle, it involved the interpretation of WBA anchors and their application to the clinical context in the ED. Conclusion: Several factors appear to be taken into account by clinical supervisors when they make decisions regarding the particular rating that they will assign a trainee on a WBA that uses entrustment anchors. Not all of these factors are specific to that particular clinical encounter. The results from this study further our understanding on the use of entrustment anchors within the ED and may facilitate faculty development regarding WBA completion as we move forward in CBME.

P188 QUALITY ASSURANCE OF SURGICAL INTERVENTION WITHIN RANDOMIZED CONTROLLED TRIALS FOR GASTRO-ESOPHAGEAL REFLUX DISEASE

Abstract Aim Randomized controlled trials provide level 1 evidence for surgical intervention in the treatment of gastro-esophageal reflux disease (GERD). This systematic review aimed to investigate the practice of surgical quality assurance in RCTs concerning surgical interventions in GERD. Background &amp; Methods A systematic literature search for articles was performed in Medline, Embase, Web of Science and Cochrane library for randomized controlled trials (RCT) evaluating surgical intervention in the management of GERD. Each RCT was assesses for surgical quality assurance (SQA) using an eight-point checklist in three main areas and given a SQA score out of 8. Clinical outcomes from each trial including reoperation rates were then compared against the SQA score. Results Thirty-one publications were included compromising 5803 patients. The assurance of surgical quality was poor with the randomized controlled trials included, ranging from 3% to 68% for the factors examined: Credentialing surgeon experience before entry into the trial by; 1. Case volume – 26% (8/31) 2. Operative reports – 6% (2/31) 3. Video assessment – 6% (2/31) 4. Live operating room evaluation – 3% (1/31) Pre-trial standardization by; 5. Education of surgeons – 13% (13%) 6. Consensus for standardization – 68% (21/31) Monitoring surgical performance during the trial by; 7. Video assessment – 10% (3/31) 8. Monitoring of data – 3% (1/31) RCTs with a high SQA score had a reduced average rate of reoperation (SQA score 4 to 8 vs. 0 to 3; 0% vs. 3.19%; P&lt;0.05). Credentialing surgeons by case volume before entry into the trial was also associated with a reduced rate of reoperation (0.04% vs. 3%; P&lt;0.05). Conclusion Surgical quality assurance has been largely lacking in randomized controlled trials for GERD previously. High surgical quality assurance was associated with reduced reoperation rates within RCTs, highlighting the need for good SQA within these types of trials. Future RCTs in benign UGI conditions with surgical interventions should ensure SQA in three main areas: (i) Credentialing surgeon experience before trial; to ensure they are past learning curve. (ii) Standardizing the surgical technique; to ensure less heterogeneity in operative performance. (iii) Monitoring performance during trial; to ensure standard of surgical quality is maintained within the trial.