Operative Performance Rating System Research Articles

Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial

BackgroundThe value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees. MethodsThis parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment. ResultsOf 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool. ConclusionThis pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.

Standardised Competency-Based Training of Medical Doctors and Associate Clinicians in Inguinal Repair with Mesh in Sierra Leone

IntroductionIn low-income settings, there is a high unmet need for hernia surgery, and most procedures are performed with tissue repair techniques. In preparation for a randomized clinical trial, medical doctors and associate clinicians received a short-course competency-based training on inguinal hernia repair with mesh under local anaesthesia. The aim of this study was to evaluate feasibility, safety and effectiveness of the training.MethodsAll trainees received a one-day theoretical module on mesh hernia repair under local anaesthesia followed by hands-on training. Performance was assessed using the American College of Surgeon’s Groin Hernia Operative Performance Rating System. Patients were followed up two weeks and one year after surgery. Outcomes of the patients operated on during the training trial were compared to the 229 trial patients operated on after the training.ResultsDuring three surgical camps, seven medical doctors and six associate clinicians were trained. In total, 129 patients were operated on as part of the training. Of the 13 trainees, 11 reached proficiency. Patients in the training group had more wound infections after two weeks (8.5% versus 3.1%; p = 0.041). There was no difference in recurrence and mortality after one year, and none of the deaths were attributed to the surgery.Discussion and conclusionMesh repair is the international standard for inguinal hernia repair worldwide. Nevertheless, this is not widely accessible in low-income settings. This study has demonstrated that short-course intensive hands-on training of MDs and ACs in mesh hernia repair is effective and safe.Trial Registration: International Clinical Trial Registry ISRCTN63478884.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy.

IntroductionSelf-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.Methods and proceduresUsing a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1–3] and senior trainees (PGY 4–5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.ResultsA total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.ConclusionsOPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00464-022-09466-6.

The use of a surgical boot camp combining anatomical education and surgical simulation for internship preparedness among senior medical students

BackgroundSenior medical students feel unprepared for surgical procedures and care for surgery patients when they begin their internship. This study sought to introduce and evaluate a surgical boot camp training for senior medical students.MethodsA 44-h surgical boot camp program of lectures on clinical practice simulation, anatomical dissections, and simulated operation on cadavers was designed, implemented, and evaluated during the 2018 to 2019 academic year. A self-administered questionnaire was used to assess students’ perceptions of the content, delivery, and self-confidence. The mini-Clinical Evaluation Exercise (mini-CEX) and the Operative Performance Rating System were used to assess skills essential to good clinical care and to facilitate feedback.ResultsOver 93% of the students were satisfied with the surgical boot camp, training equipment, and learning materials provided. After six sessions of training, 85.3% reported gaining self-confidence and performed better in some surgical procedures such as major gastrectomy. The mini-CEX scores suggested significant improvement in the students’ clinical skills, attitudes, and behaviors (P < 0.01). Ninety-eight percent of students felt that the anatomical knowledge taught met their needs. The scores of the Operative Performance Rating System suggested that the students’ surgical skills such as instruments handling, incising, treatment of surrounding tissues (blood vessels, nerves), and smoothness of the whole operation had increased significantly following the surgical boot camp (All P < 0.01).ConclusionThe surgical boot camp curriculum improved students’ satisfaction and confidence in core clinical practice competencies. Therefore, medical schools the world over should continue to seek ways to bridge the gaps between pre-clinical, clinical, and internship training.

IMPLEMENTATION OF THE EVALUATION TO IMPROVE RESIDENT SURGICAL SKILLS

Abstract INTRODUCTION A guided and structured learning process with an evaluation method is essential for resident training. A survey carried out in our department showed that the implementation of an objective evaluation of surgical skills is of great interest to residents. MATERIAL AND METHODS During 2020, a structured and objective evaluation of elemental procedures was implemented for first, third and fifth-year general surgery residents. The preoperative briefing is carried out to discuss the clinical case and plan the surgery. Then, the resident performs the surgical intervention, assisted by an expert surgeon who guides him according to his level and the difficulties encountered. A structured evaluation is carried out, using The American Board of Surgery model Operative Performance Rating System (OPRS). With the postoperative debriefing, the resident’s strengths and weaknesses are analyzed and self-reflected with the recording. RESULTS Eleven evaluations have been performed: 3 laparoscopic appendectomies (R1), 2 laparoscopic cholecystectomies (R3) and 3 laparoscopic sigmoidectomies (R5). In 3 of them, it was performed by 2 different evaluators (live and recorded surgery). Patient safety was not compromised. Participants indicated a high level of satisfaction and greater motivation in surgical learning. CONCLUSION Implementing this evaluation method in the current training programme of General and Digestive Surgery Residency is feasible and may be beneficial for the controlled development of their surgical skills.

A Competency-based Laparoscopic Cholecystectomy Curriculum Significantly Improves General Surgery Residents' Operative Performance and Decreases Skill Variability: Cohort Study.

To demonstrate the feasibility of implementing a CBE curriculum within a general surgery residency program and to evaluate its effectiveness in improving resident skill. Operative skill variability affects residents and practicing surgeons and directly impacts patient outcomes. CBE can decrease this variability by ensuring uniform skill acquisition. We implemented a CBE LC curriculum to improve resident performance and decrease skill variability. PGY-2 residents completed the curriculum during monthly rotations starting in July 2017. Once simulator proficiency was reached, residents performed elective LCs with a select group of faculty at 3 hospitals. Performance at curriculum completion was assessed using LC simulation metrics and intraoperative operative performance rating system scores and compared to both baseline and historical controls, comprised of rising PGY-3s, using a 2-sample Wilcoxon rank-sum test. PGY-2 group's performance variability was compared with PGY-3s using Levene robust test of equality of variances; P < 0.05 was considered significant. Twenty-one residents each performed 17.52 ± 4.15 consecutive LCs during the monthly rotation. Resident simulated and operative performance increased significantly with dedicated training and reached that of more experienced rising PGY-3s (n = 7) but with significantly decreased variability in performance ( P = 0.04). Completion of a CBE rotation led to significant improvements in PGY-2 residents' LC performance that reached that of PGY-3s and decreased performance variability. These results support wider implementation of CBE in resident training.

Education Management Platform Enables Delivery and Comparison of Multiple Evaluation Types

The purpose of this study was to determine whether an automated platform for evaluation selection and delivery would increase participation from surgical teaching faculty in submitting resident operative performance evaluations. We built a HIPAA-compliant, web-based platform to track resident operative assignments and to link embedded evaluation instruments to procedure type. The platform matched appropriate evaluations to surgeons' scheduled procedures, and delivered multiple evaluation types, including Ottawa Surgical Competency Operating Room Evaluation (O-Score) evaluations and Operative Performance Rating System (OPRS) evaluations. Prompts to complete evaluations were made through a system of automatic electronic notifications. We compared the time spent in the platform to achieve evaluation completion. As a metric for the platform's effect on faculty participation, we considered a task that would typically be infeasible without workflow optimization: the evaluator could choose to complete multiple, complementary evaluations for the same resident in the same case. For those cases with multiple evaluations, correlation was analyzed by Spearman rank test. Evaluation data were compared between PGY levels using repeated measures ANOVA. The study took place at 4 general surgery residency programs: The University of Massachusetts Medical School-Baystate, the University of Connecticut School or Medicine, the University of Iowa Carver College of Medicine, and Maimonides Medical Center. From March 2017 to February 2019, the study included 70 surgical teaching faculty and 101 general surgery residents. Faculty completed 1230 O-Score evaluations and 106 OPRS evaluations. Evaluations were completed quickly, with a median time of 36 ± 18 seconds for O-Score evaluations, and 53 ± 51 seconds for OPRS evaluations. 89% of O-Score and 55% of OPRS evaluations were completed without optional comments within one minute, and 99% of O-Score and 82% of OPRS evaluations were completed within 2 minutes. For cases eligible for both evaluation types, attendings completed both evaluations on 74 of 221 (33%) of these cases. These paired evaluations strongly correlated on resident performance (Spearman coefficient = 0.84, p < 0.00001). Both evaluation types stratified operative skill level by program year (p < 0.00001). Evaluation initiatives can be hampered by the challenge of making multiple surgical evaluation instruments available when needed for appropriate clinical situations, including specific case types. As a test of the optimized evaluation workflow, and to lay the groundwork for future data-driven design of evaluations, we tested the impact of simultaneously delivering 2 evaluation instruments via a secure web-based education platform. We measured the evaluation completion rates of faculty surgeon evaluators when rating resident operative performance, and how effectively the results of evaluation could be analyzed and compared, taking advantage of a highly integrated management of the evaluative information.

Original Homebuilt Off-the-Job Training System for Vascular Surgeons: System Analysis and Assessment.

Objective: We assessed the effectiveness and appropriateness of our original off-the-job training (Off JT) system using data acquired from recruited medical students and doctors.Materials and Methods: We presented our original homebuilt Off JT system, which is simple and inexpensive. In our unique system, we performed anastomosis at the bottom of a plastic pot, which mimics the actual open surgical procedure at a deep site. There were four evaluation points: (A) operating time, (B) performance of anastomosis by semi-automatically analyzing the image with the coefficient of variation (standard deviation/length) of the “bite” and the “pitch,” (C) scoring of the total surgical skill evaluated by the trainers according to the Operative Performance Rating System (OPRS), and (D) the relationship of these three factors (A, B, and C).Results: The procedural time and coefficient of variation of the bite and pitch decreased and the OPRS score increased after training. There was a strong correlation between procedural time, anastomotic performance, and OPRS score.Conclusion: The effectiveness of our original homebuilt system was shown by reduced procedural time, improved anastomotic quality, and increased OPRS score.

Establishing Validity Evidence for an Operative Performance Rating System for Plastic Surgery Residents

The aim of this study was to describe an operative performance rating system for plastic surgery residents and provide validity evidence for the instrument. Three plastic surgery residents (PGY levels 1, 5, and 6) from Southern Illinois University School of Medicine (SIUSOM) performed a carpal tunnel release with audio video recording. The 3 videos were reviewed by 8 expert hand surgeons and 3 SIUSOM faculty using the operative performance rating system instrument to assess resident operative performance. Validity evidence including content, internal structure, and relationship to other variables was collected. Inter-rater reliability was consistently fair to moderate (weighted Cohen's Kappa 0.44-0.84 for experts, 0.24-0.55 for SIUSOM raters), and all assessment items were highly correlated (Cronbach's alpha of 0.9867). Local SIUSOM faculty routinely demonstrated higher overall scores for PGY 1 and PGY 6 residents compared to expert raters. Although limited by small numbers, this pilot study suggests that potential bias based upon PGY year, identity, and performance history may exist and independent assessment by unbiased raters or comparison to national operative norms may be valuable. Our study provides baseline validity evidence for a resident operative performance assessment tool that can be integrated into practice in plastic surgery training programs.

Measuring Competence in Surgical Training through Assessment of Surgical Entrustable Professional Activities

To assess competency in surgical residents with bundled assessments using a surgical entrustable professional activity (SEPA) framework. A pilot study was conducted using a combination of validated assessment tools (multiple choice exam (MCE) questions from the Surgical Council on Resident Education [SCORE], the Clinical Assessment and Management Examination - Outpatient (CAMEO) form, the Virtual Surgical Patient (VSP) website, and a procedure-specific Operative Performance Rating System [OPRS]) to determine competency in surgical residents in the treatment of breast cancer and gallbladder disease, respectively. A large academic institution with a surgical training program in the Mid West of the United States. A total of 10 categorical surgical residents were invited to participate. Five completed the breast surgical EPA (SEPA) and 5 different residents completed the gallbladder SEPA. In terms of performance on the assessments, for the breast SEPA, scores did not appear to be related to PGY level, and residents' performance in general was the least strong on the MCE and the VSP case. The gallbladder SEPA showed a more expected pattern, distinguishing between junior and senior residents. As expected, all junior residents were required to remediate the OPRS assessment, while the senior residents passed. For the OPRS, senior level residents consistently were rated as "excellent" in terms of operative flow (5/5), while junior residents were all given a score of "good" (3/5). Assessing competence among surgical residents has been a discussion for several years. Varying methods of assessing competence have been proposed, but surgical competence is presently defined in a very general way through both the ACGME and American Board of Surgery (ABS). Using a SEPA format, as proposed, we could ensure specific understanding of each graduating resident's ability. These results show that the SEPA may be a valid tool for defining and capturing multiple areas of competence that are associated with different disease processes.

A cadaveric procedural anatomy simulation course improves video-based assessment of operative performance

BackgroundInadequate anatomic knowledge has been cited as a major contributor to declining surgical resident operative competence. We analyzed the impact of a comprehensive, procedurally oriented cadaveric procedural anatomy dissection laboratory on the operative performance of surgery residents, hypothesizing that trainees’ performance of surgical procedures would improve after such a dissection course. Materials and methodsMidlevel general surgery residents (n = 9) participated in an 8 wk, 16-h surgery faculty-led procedurally oriented cadaver simulation course. Both before and after completion of the course, residents participated in a practical examination, in which they were randomized to perform one of nine Surgical Council on Resident Education–designated “essential” procedures. The procedures were recorded using wearable video technology. Videos were deidentified before evaluation by six faculty raters blinded to examinee and whether performances occurred before or after an examinee had taken the course. Raters used the validated Operative Performance Rating System and Objective Structured Assessment of Technical Skill scales. ResultsAfter the course residents had higher procedure-specific scores (median, 4.0 versus 2.4, P < 0.0001), instrument-handling (4.0 versus 3.0, P = 0.006), respect for tissue (4.0 versus 3.0, P = 0.0004), time and motion (3.0 versus 2.0, P = 0.0007), operation flow (3.0 versus 2.0, P = 0.0005), procedural knowledge (4.0 versus 2.0, P = 0.0001), and overall performance scores (4.0 versus 2.0, P < 0.0001). Operative Performance Rating System and Objective Structured Assessment of Technical Skill scales averaged by number of items in each were also higher (3.2 versus 2.0, P = 0.0002 and 3.1 versus 2.2, P = 0.002, respectively). ConclusionsA cadaveric procedural anatomy simulation course covering a broad range of open general surgery procedures was associated with significant improvements in trainees’ operative performance.

Using GoPro to Give Video-Assisted Operative Feedback for Surgery Residents: A Feasibility and Utility Assessment

As an adjunct to simulation-based teaching, laparoscopic video-based surgical coaching has been an effective tool to augment surgical education. However, the wide use of video review in open surgery has been limited primarily due to technological and logistical challenges. The aims of our study were to (1) evaluate perceptions of general surgery (GS) residents on video-assisted operative instruction and (2) conduct a pilot study using a head-mounted GoPro in conjunction with the operative performance rating system to assess feasibility of providing video review to enhance operative feedback during open procedures. GS residents were anonymously surveyed to evaluate their perceptions of oral and written operative feedback and use of video-based operative resources. We then conducted a pilot study of 10 GS residents to assess the utility and feasibility of using a GoPro to record resident performance of an arteriovenous fistula creation with an attending surgeon. Categorical variables were analyzed using the chi-square test. Academic, tertiary medical center. GS residents and faculty. A total of 59 GS residents were anonymously surveyed (response rate = 65.5%). A total of 40% (n = 24) of residents reported that structured evaluations rarely or never provided meaningful feedback. When feedback was received, 55% (n = 32) residents reported that it was only rarely or sometimes in regard to their operative skills. There was no significant difference in surveyed responses among junior postgraduate year (PGY 1-2), senior (PGY 3-4), or chief residents (PGY-5). A total of 80% (n = 8) of residents found the use of GoPro video review very or extremely useful for education; they also deemed video review more useful for operative feedback than written or communicative feedback. An overwhelming majority (90%, n = 9) felt that video review would lead to improved technical skills, wanted to review the video with the attending surgeon for further feedback, and desired expansion of this tool to include additional procedures. Although there has been progress toward improving operative feedback, room for further improvement remains. The use of a head-mounted GoPro is a dynamic tool that provides high-quality video for operative review and has the potential to augment the training experience of GS residents. Future studies exploring a wide array of open procedures involving a greater number of trainees will be needed to further define the use of this resource.

The Educational Value and Feasibility of a Video-Capture System in the Surgical Training of OBGYN Residents

BACKGROUND: Many professions utilize video-capture technology for training and skill improvement. This study explores the utility, sustainability, and effectiveness of this technology as part of a resident surgical evaluation process. In order to optimize surgical training we utilized video-capture technology (SimCapture) as residents performed surgical procedures. Point-of care feedback using standard evaluation tools and video-review, using the same evaluation tools was also conducted. METHODS: We recorded a group of OBGYN residents performing one of four surgeries over 8 weeks. This project was considered IRB exempt. An ACGME milestone evaluation and OPRS (operative performance rating system) specific to the procedure was used to evaluate performance. The video was then evaluated by another faculty member, blinded to the resident's level of training, using the milestone and OPRS forms. Data were collected and analyzed regarding the value of the process as a tool to optimize surgical teaching. RESULTS: We hoped to assess approximately 25% of cases for our pilot project. Surgical evaluation and survey data regarding the educational value of the experience was collected. Data were assessed and compared using Spearman's Rho correlation statistical assessment using SPSS 16.0. DISCUSSION: Our pilot study offers benefit in other areas including surgical training, measuring surgical competence, and optimizing patient care. We hope to provide a video portfolio of resident surgical performance which can be used to display technical skills when pursuing future professional opportunities. This technology may also serve a role in the credentialing and remediation process.

A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills.

This study evaluates whether video-based coaching can enhance laparoscopic surgical skills performance. Many professions utilize coaching to improve performance. The sports industry employs video analysis to maximize improvement from every performance. Laparoscopic novices were baseline tested and then trained on a validated virtual reality (VR) laparoscopic cholecystectomy (LC) curriculum. After competence, subjects were randomized on a 1:1 ratio and each performed 5 VRLCs. After each LC, intervention group subjects received video-based coaching by a surgeon, utilizing an adaptation of the GROW (Goals, Reality, Options, Wrap-up) coaching model. Control subjects viewed online surgical lectures. All subjects then performed 2 porcine LCs. Performance was assessed by blinded video review using validated global rating scales. Twenty subjects were recruited. No significant differences were observed between groups in baseline performance and in VRLC1. For each subsequent repetition, intervention subjects significantly outperformed controls on all global rating scales. Interventions outperformed controls in porcine LC1 [Global Operative Assessment of Laparoscopic Skills: (20.5 vs 15.5; P = 0.011), Objective Structured Assessment of Technical Skills: (21.5vs 14.5; P = 0.001), and Operative Performance Rating System: (26 vs 19.5; P = 0.001)] and porcine LC2 [Global Operative Assessment of Laparoscopic Skills: (28 vs 17.5; P = 0.005), Objective Structured Assessment of Technical Skills: (30 vs 16.5; P < 0.001), and Operative Performance Rating System: (36 vs 21; P = 0.004)]. Intervention subjects took significantly longer than controls in porcine LC1 (2920 vs 2004 seconds; P = 0.009) and LC2 (2297 vs 1683; P = 0.003). Despite equivalent exposure to practical laparoscopic skills training, video-based coaching enhanced the quality of laparoscopic surgical performance on both VR and porcine LCs, although at the expense of increased time. Video-based coaching is a feasible method of maximizing performance enhancement from every clinical exposure.

Development of an Operative Performance Rating System for Plastic Surgery Residents

PURPOSE A standardized measure of operative performance is an essential component of the Patient Care Competency and is critical to the training of plastic surgery residents. The Operative Performance Rating System (OPRS) developed and validated by the Department of General Surgery at Southern Illinois University consists of procedure-specific evaluations for resident intraoperative performance. The OPRS provides an objective measure of procedure-specific resident performance that is not currently being assessed in plastic surgery training programs. The purpose of this study is to describe OPRS for plastic surgery residents and propose methodology for assessing the reliability, validity, and feasibility of this instrument. METHODS Ten procedure-specific rating instruments were developed for sentinel cases, each consisting of critical procedure-specific steps based on literature review and faculty focus group consensus. Sentinel cases were chosen based on review of the American Council of Graduate Medical Education Milestones and resident logs of the most commonly performed plastic surgery procedures, both at our institution and nationally. The degree of guidance required from the attending surgeon is recorded for each step. General operative performance competency is evaluated from validated items developed by the University of Toronto.1 All items use a 5-point Likert scale with behavioral anchors. The OPRS assessments will be incorporated into the internet-based resident management platform New Innovations. Sentinel procedures for evaluation will be identified on a weekly basis by the residency coordinator, based on resident operative assignments (postgraduate year 2–6) organized by the chief resident. OPRS assessment forms will be available electronically immediately following the procedures, with an e-mail reminder notification 24 hours later to help encourage compliance. In addition, resident self-assessment using the same OPRS will be conducted and correlated with faculty OPRS evaluations. Each OPRS assessment will be evaluated for internal consistency reliability and inter-item correlation. Inter-rater reliability will be measured by faculty assessment of videotaped sentinel procedures using the appropriate OPRS instrument. Performance variation based on resident PGY level will be analyzed using 1-way analysis of variance. Feasibility will also be determined based on attending and resident response rates and time to completion for the OPRS evaluations, as well as a short written survey to assess resident and attending satisfaction and obtain feedback following OPRS implementation. CONCLUSIONS A web-based OPRS provides timely and objective feedback to improve residents’ technical and decision-making skills, as demonstrated by the experiences of other surgical specialties.2 This instrument will provide both formative and summative resident feedback, encouraging faculty and residents to focus on demonstrated competencies and areas for improvement.3 Furthermore, resident operative performance can be monitored across time and residents, allowing program directors to have a long-term objective method of evaluating resident technical performance.3 A reliable and valid OPRS may provide a feasible method of intraoperative assessment that could be implemented across all plastic surgery training programs.

Development of an Operative Performance Rating System for Plastic Surgery Residents.

Development of an Operative Performance Rating System for Plastic Surgery Residents.

Technical Skills Assessment Toolbox

The purpose of this study was to create a technical skills assessment toolbox for 35 basic and advanced skills/procedures that comprise the American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) surgical skills curriculum and to provide a critical appraisal of the included tools, using contemporary framework of validity. Competency-based training has become the predominant model in surgical education and assessment of performance is an essential component. Assessment methods must produce valid results to accurately determine the level of competency. A search was performed, using PubMed and Google Scholar, to identify tools that have been developed for assessment of the targeted technical skills. A total of 23 assessment tools for the 35 ACS/APDS skills modules were identified. Some tools, such as Operative Performance Rating System (OSATS) and Objective Structured Assessment of Technical Skill (OPRS), have been tested for more than 1 procedure. Therefore, 30 modules had at least 1 assessment tool, with some common surgical procedures being addressed by several tools. Five modules had none. Only 3 studies used Messick's framework to design their validity studies. The remaining studies used an outdated framework on the basis of "types of validity." When analyzed using the contemporary framework, few of these studies demonstrated validity for content, internal structure, and relationship to other variables. This study provides an assessment toolbox for common surgical skills/procedures. Our review shows that few authors have used the contemporary unitary concept of validity for development of their assessment tools. As we progress toward competency-based training, future studies should provide evidence for various sources of validity using the contemporary framework.

An Operative Performance Rating System for Urology Residents

An operative performance rating system for urology residents was developed for 6 sentinel urological procedures. We tested the reliability, validity and feasibility of the operative performance rating system for urology residents. The operative performance rating system of each procedure contained a 3-point case difficulty scale, 4 to 6 procedure specific items, 3 general items and an overall performance item. A Likert scale of 1 to 5 was used for each item. A single video/audio record of each procedure was evaluated by the faculty. Single item interrater agreement was measured by comparing the observed variance and random measurement error variance. Resident operative performance evaluations were completed on line. Internal consistency reliability was measured using Cronbach α. Overall scale scores by resident training postgraduate year level were compared using 1-way ANOVA. Faculty evaluation of video/audio records showed an interrater agreement range of 0.71 to 0.92. Faculty evaluations of resident operative performance demonstrated an internal consistency reliability range of 0.91 to 0.95. Significant differences in overall scale scores between postgraduate year levels were noted for 3 of the 6 procedures (p ≤0.0016). An operative performance rating system for urology residents is feasible using an Internet based resident management system. Interrater agreement and internal consistency reliability meet threshold limits for checklist evaluation instruments. The operative performance rating system can discriminate among postgraduate year levels of resident training. A validated operative performance rating system can offer residents immediate, objective feedback on surgical performance and enable program directors to monitor progress in resident operative performance.

A Controlled Study to Determine Measurement Conditions Necessary for a Reliable and Valid Operative Performance Assessment

This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment. Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training. Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation. A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances. OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.

1498 AN OPERATIVE PERFORMANCE RATING SYSTEM FOR UROLOGY RESIDENTS

You have accessJournal of UrologyTechnology & Instruments: Surgical Education & Skills Assessment1 Apr 20111498 AN OPERATIVE PERFORMANCE RATING SYSTEM FOR UROLOGY RESIDENTS Aaron Benson and Thomas Tarter Aaron BensonAaron Benson Springfield, IL More articles by this author and Thomas TarterThomas Tarter Springfield, IL More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2011.02.1456AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES The best methods to evaluate medical procedures recommended by the American College of Graduate Medical Education are 1) simulations and models, and 2) checklists. Simulators are valuable adjuncts to resident training, but have not been uniformly applied. Checklists, including end-of-rotation evaluations presently applied, do not provide specific or timely feedback. Through a grant from the Society of University Chairpersons and Program Directors of the American Urological Association (AUA), an operative performance rating system (OPRS) has been developed to provide objective assessment of resident performance in the operating room. METHODS An OPRS was developed for six of the most commonly performed urology procedures at our institution. Procedure-specific performance items were developed using evidence based literature, and through faculty focus group discussion. To measure internal consistency, one of each sentinel procedure was video-recorded and assessed by all faculty using the appropriate OPRS. The OPRS instruments were incorporated into internet-based urology residency management software (New Innovations). After a resident performed a procedure, the responsible faculty received an e-mail invitation to complete the corresponding OPRS. For completed evaluations, internal consistency, individual performance item-to-overall score correlations, and PGY-level performance variation was calculated using one-way analysis of variation and Eta statistics. RESULTS Preliminary results for minimum completed evaluations in cystoscopy with ureteral stent placement (37) and prostate biopsy (23) demonstrate high Cronbach's alpha values (0.93 and 0.92, respectively). Significantly improving performance variation was shown for advancing resident levels (p=0.0006 and p=0.0002, respectively). Individual item-overall score correlations are also high (0.90 and 0.93, respectively). Three procedures have been video-recorded thus far and demonstrate high Cronbach's alpha values (0.90, 0.88, and 0.77, respectively). CONCLUSIONS An OPRS specifically designed for urologic surgery may provide a consistent means for immediate and specific feedback for urology residents. Our findings could lead to the development of a validated checklist instrument for measuring urology resident surgical proficiency that can be broadly applied across programs. © 2011 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 185Issue 4SApril 2011Page: e601 Advertisement Copyright & Permissions© 2011 by American Urological Association Education and Research, Inc.MetricsAuthor Information Aaron Benson Springfield, IL More articles by this author Thomas Tarter Springfield, IL More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...