Rating Scale Of Operative Performance Research Articles

Validation of Transvaginal Hysterectomy Surgical Model – Modification of the Flowerpot Model to Improve Vesicovaginal Plane Simulation

To establish face and construct validity for a novel variation of American College of Obstetrics and Gynecology "Flowerpot Model" for transvaginal hysterectomy (TVH) surgical simulation with improved vesicovaginal dissection during surgical education simulation. Cross-sectional face and construct validation study using the "Flowerpot Model." The vesicovaginal dissection plane was modified to include additional felt and balloon materials to simulate the bladder. Single academic center. Fourteen residents and fellows, postgraduate year (PGY) 2 to 6, subdivided into junior (n = 8) with ≤10 prior TVH surgeries and senior groups (n = 6) with >10 prior TVH surgeries performed. All subjects watched a brief introductory video and then were filmed simulating a TVH. For face validity, subjects completed an anatomic checklist and pre/post simulation satisfaction survey. For construct validation, 2 independent, blinded expert surgeons (M.A. and J.M.) graded films using the Global Rating Scale of Operative Performance (GRS). Primary outcome was mean GRS between groups. The junior group consisted of PGY 2 to 3 with ≤ 10 prior TVH, median 7.5 (interquartile range [IQR] 6.75) and senior group PGY 3 to 6 with >10 TVH, median 19 (IQR 10) (p <.01). Subjects were "satisfied" or "very satisfied" with bladder and anterior peritoneal fold simulation (92%) and found vesicovaginal dissection "realistic" (100%). GRS score was significantly different between groups (juniors, 19.5 [IQR 5] vs seniors, 28.5 [IQR 8.5]; p = .048). Intergrader correlation was high (ρ = 0.87, p <.01). Surgeon volume of prior TVH was not significantly correlated to average GRS score, ρ = 0.49 (p = .10). The model improved comfort and confidence scores in the junior group more than senior group (p = .04), but senior group still had higher post simulation confidence scores than the junior group (p = .02). Face and construct validity with the modified Flowerpot Model was demonstrated. This low fidelity model is capable of simulation of a TVH with a novel vesicovaginal dissection. Prior surgical experience was not correlated to GRS score or time to procedure completion.

Cerebrospinal Fluid Leak Repair Simulation Model: Face, Content, and Construct Validation.

Cerebrospinal fluid leak (CSFL) repair simulation models are scarce; however, these models are valuable tools for skull base reconstruction surgery training. This study aims to assess the face, content, and construct validity of a CSFL repair simulation model. Eight novices (residents- PGY3) and eight experts have performed skull base reconstruction in multiple sites in twelve human cadaveric heads in simulated surgical environment. The experts completed a post-study 21-item questionnaire to assess the face and content validity. The performances of the participants were recorded and scored by two independent investigators who were blinded to the participant's level. Global Rating Scale of Operative Performance (GRSOP) and a Specific Skull Base Reconstruction Checklist (SBRC) were used to score the performances. The responses from the expert group for the 21-item questionnaire were high for all items (4.13-4.88 out of 5). The internal consistency reliability of the questionnaire and the intraclass correlation, which was derived by Cronbach's Alpha, were 0.913 and 0.941 respectively. Differences in construct validity between the two groups were statistically significant for both the GRSOP and SBRS (P-value < 0.001). We demonstrated the face, content, and construct validity of the CSFL repair simulation model, which facilitates the acquisition of technical skills necessary for skull base reconstruction surgery. The model includes realistic features that make it useful in educational courses.

Concordance Between Expert and Nonexpert Ratings of Condensed Video-Based Trainee Operative Performance Assessment

We examined the impact of video editing and rater expertise in surgical resident evaluation on operative performance ratings of surgical trainees. Randomized independent review of intraoperative video. Operative video was captured at a single, tertiary hospital in Boston, MA. Six common general surgery procedures were video recorded of 6 attending-trainee dyads. Full-length and condensed versions (n = 12 videos) were then reviewed by 13 independent surgeon raters (5 evaluation experts, 8 nonexperts) using a crossed design. Trainee performance was rated using the Operative Performance Rating Scale, System for Improving and Measuring Procedural Learning (SIMPL) Performance scale, the Zwisch scale, and ten Cate scale. These ratings were then standardized before being compared using Bayesian mixed models with raters and videos treated as random effects. Editing had no effect on the Operative Performance Rating Scale Overall Performance (-0.10, p = 0.30), SIMPL Performance (0.13, p = 0.71), Zwisch (-0.12, p = 0.27), and ten Cate scale (-0.13, p = 0.29). Additionally, rater expertise (evaluation expert vs. nonexpert) had no effect on the same scales (-0.16 (p = 0.32), 0.18 (p = 0.74), 0.25 (p = 0.81), and 0.25 (p = 0.17). There is little difference in operative performance assessment scores when raters use condensed videos or when raters who are not experts in surgical resident evaluation are used. Future validation studies of operative performance assessment scales may be facilitated by using nonexpert surgeon raters viewing videos condensed using a standardized protocol.

The Role of Manual Dexterity and Cognitive Functioning in Enhancing Resident Operative Autonomy

Autonomy, both operative and nonoperative, is one of the most critical aspects of successful surgical training. Both surgeon and resident share the responsibility of achieving this goal. We hypothesize that operative autonomy is distinct and depends, for the most part, on the resident's manual dexterity, knowledge of, and preparation for the procedure. Over a period of 4 academic years, between July 2014 and June 2018, a total of 958 Global Rating Scale of Operative Performance evaluations were completed by 32 general and subspecialty faculty surgeons for 35 residents. Elective procedures were evaluated, including 165 (17.2%) by postgraduate year (PGY)1 residents, 253 (26.4%) by PGY2, 199 (20.8%) by PGY3, 147 (15.3%) by PGY4, and 194 (20.3%) by PGY5. The procedures evaluated were: 261 (27.2%) hernia repairs; 178 (18.6%) cholecystectomies; 102 (10.6%) colorectal and anal procedures; 73 (7.6%) vascular procedures; 56 (5.8%) thyroid and parathyroidectomies; 39 (4.1%) foregut (esophagus and stomach) procedures; 38 (4%) skin, soft tissue, and breast; 92 (10%) hepatopancreatic; 20 (2.1%) pediatric procedures; and 99 (10.3%) other procedures including amputations, cardiothoracic, and solid organs procedures. Each resident was scored from 1 to 5 (1 lowest, 5 highest) in each of the following categories of Global Rating Scale of Operative Performance: respect for tissue (RT), time and motion (T&M), instrument handling (IH), knowledge of the instrument (KI), flow of operation (FO) and resident's preparation for the procedure (RP). Resident operative autonomy (ROA) was assessed using the Zwisch scale, a 4-point scale describing faculty supervision behaviors associated with different degrees of resident autonomy (1: Show and Tell, 2: Active Help, 3: Passive Help, and 4: Supervision Only). Correlation and ordinal regression analyses were conducted to examine the relationship between ROA and manual dexterity (RT, T&M, IH, and FO), and cognitive functioning (knowledge of instruments and resident preparation). Results indicated a positive correlation between ROA and RT (r = 0.528, p < 0.001), T&M (r = 0.630, p < 0.001), IH (r = 0.597, p < 0.001), KI (r = 0.490, p < 0.001), FO (r = 0.637, p < 0.001), and RP (r = 0.525, p < 0.001). Additionally, there was a weak inverse correlation between ROA and the number of years the surgeon had been in practice (r = -0.127, p = 0.001). The significant predictors of resident autonomy found by the ordinal logistic regression include time and motion (p < 0.001), flow of operation (p < 0.001), and resident's preparation for the procedure (p < 0.001). Resident operative autonomy is a product of shared responsibility between the faculty and resident. However, residents' inherent and/or acquired skills and preparation for the operative procedures play a critical role. Residents should be advised to use available resources such as simulation to augment their skills preoperatively and to enhance their autonomy in the operating room.

Simulation Model for Retroperitoneal Dissection: Can Improve Surgical Skills for Complicated Open Laparotomy

Faculty Advisor: John Fischer, MD PURPOSE: To develop a low fidelity model for retroperitoneal dissection and identification of the ureter to improve resident's confidence and competency in the operating room. BACKGROUND: Work hour restrictions and few open laparotomies have created barriers to confident resident performance of retroperitoneal dissections. While simulation training models have been developed for laparoscopy, few have been created and validated for open laparotomies. METHODS: OB/GYN residents were divided into case (PGY3) and control (PGY4). A pretest questionnaire assessed operative experience. Both groups performed a videotaped low fidelity simulated retroperitoneal dissection that included identifying the course of the ureter. An instructional video was introduced and repeat simulation was performed by PGY3 trainees only. Three expert gynecologists independently scored performances using a checklist and global rating scale of operative performance (OSAT). PGY3 pretest/posttest scores were compared to PGY4 scores using the Wilcoxon rank sum test. Paired t-test was used for comparison of cases. P&lt;.05. RESULTS: Median pretest checklist score for PGY4 was significantly increased compared to PGY3 (7.3 vs. 6.3, P=.0314) however the median pretest OSAT score was not. A statistically significant increase in average PGY3 scores between pre vs. post final checklist scores and final OSAT scores was noted. A statistically significant increase in the median scores between PGY4 pretest (control) vs. PGY3 posttest (case) final checklist scores and final global scores (7.3 vs. 9.2, P=.0052) and (24.2 vs. 28.2, P=.0042) was noted. DISCUSSION: The development of a low fidelity model for retroperitoneal dissection and use of an instructional video may complement traditional learning for this core competency.

Preliminary Assessment of a Dry-Lab Model for Laparoscopic Percutaneous Inguinal Ring Suture Training.

Laparoscopic percutaneous inguinal ring suturing (PIRS) is a valuable technique to repair indirect inguinal hernias in children. The aim of the study was to show the preliminary results of training in an easy-to-build dry-lab model for laparoscopic PIRS technique. The dry-lab model consists of two surgical gloves (being the smaller inside the larger). The entry of the glove is tied with a rubber letting a 5-mm trocar within it (optic). Carbon dioxide is inflated through the trocar at 8 mmHg pressure. The four long fingers of the glove simulate four internal inguinal rings. Eighteen participants without significant laparoscopic experience were asked to visualize a video showing a laparoscopic PIRS performed in a 4-year-old girl with a right inguinal hernia followed by a video showing the same technique performed in our dry-lab model. They were then asked to repeat the procedure in each of the four fingers, and the procedures were recorded. Two surgeons using the task-specific checklists (TSCs), global rating scale of operative performance (GRS), and time for the whole procedure evaluated the performance blindly. We found significant improvement in median TSC (P < .05) and median GRS each time the procedure was repeated (P < .05). There was significant shortening each time the procedure was repeated (P < .05), except between the third and fourth finger (P = .068). Our dry-lab model might be a good option for starting laparoscopic PIRS training.

Centralized, capacity-building training of Lichtenstein hernioplasty in Brazil

BackgroundIn Brazil, access to healthcare varies widely by community. Options for repair of surgically correctable conditions, such as inguinal hernias, are limited. A training program was instituted to expand access to Lichtenstein hernioplasty. MethodsBetween September, 2014 and September, 2015, 3 orders of training series took place in São Paulo, Brazil. Participating surgeons received training and assessments from expert trainers using the Operative Performance Rating Scale (OPRS). Those who completed training successfully were invited to become trainers. OPRS scores were compared between training series. Outcomes were documented up to 6 months post-training. ResultsThe 3 orders of training series resulted in 45 surgeons trained and 213 hernias repaired. Eleven trainees subsequently became trainers. Mean post-training OPRS scores were 4.4 (scale of 5) and did not vary significantly between training series. The overall complication rate was 4.7%, with no hernia recurrences or reoperations at 6 months. ConclusionsCompetency-based training generates a regional network of surgeons proficient in Lichtenstein hernioplasty. Each training session progressively expands patient access to high quality operations in underserved communities in Brazil.

Global Outreach Using a Systematic, Competency-Based Training Paradigm for Inguinal Hernioplasty

Sustainable, capacity-building educational collaborations are essential to address the global burden of surgical disease. To assess an international, competency-based training paradigm for hernia surgery in underserved countries. In this prospective, observational study performed from November 1, 2013, through October 31, 2015, at 16 hospitals in Brazil, Ecuador, Haiti, Paraguay, and the Dominican Republic, surgeons completed initial training programs in hernia repair, underwent interval proficiency assessments, and were appointed regional trainers. Competency-based evaluations of technical proficiency were performed using the Operative Performance Rating Scale (OPRS). Maintenance of proficiency was evaluated by video assessments 6 months after training. Certified trainees received incentives to document independent surgical outcomes after training. An OPRS score of 3.0 (scale of 1 [poor] to 5 [excellent]) indicated proficiency. Secondary outcomes included initial vs final scores by country, scores among surgeons trained by the regional trainers (second-order trainees), interval scores 6 months after training, and postoperative complications. A total of 20 surgeon trainers, 81 local surgeons, and 364 patients (343 adult, 21 pediatric) participated in the study (mean [SD] age, 47.5 [16.3] years; age range, 16-83 years). All 81 surgeons successfully completed the program, and all 364 patients received successful operations. Mean (SD) OPRS scores improved from 4.06 (0.87) before the initial training program to 4.52 (0.57) after training (P < .001). No significant variation was found by country in final scores. On trainee certification, 20 became regional trainers. The mean (SD) OPRS score among 53 second-order trainees was 4.34 (0.68). After 6-month intervals, the mean (SD) OPRS score among participating surgeons was 4.34 (0.55). The overall operative complication rate during training series was 1.1%. Competency-based training helps address the global burden of surgical disease. The OPRS establishes an international standard of technical assessment. Additional studies of long-term surgeon trainer proficiency, community-specific quality initiatives, and expansion to other operations are warranted.

Laparoscopy Skills Simulation for the Obstetrics and Gynecology Resident.

Operating room experience alone is insufficient for surgical training. In the current era of graduate medical education, work hour restrictions and the expectations of patients and the public limit residents' ability to participate in surgical cases. We aimed to create a laparoscopy curriculum for gynecology residents that teaches the fundamentals of laparoscopy utilizing self-learning modules complemented by skill-focused and faculty-guided laboratories to maximize learning time outside of the operating room. The curriculum consists of five electronic modules on the basics of laparoscopy. Residents are expected to complete these on their own. In addition, quarterly simulation labs are taught by faculty and allow residents to practice, with guidance, key skills as derived from a previously published laparoscopic surgery skills curriculum. Residents are then evaluated using an objective structured assessment of technical skill comprised of the Global Rating Scale of Operative Performance and time metrics for each skill. We also evaluate resident confidence and satisfaction. In our initial year, residents' confidence regarding laparoscopy ability and satisfaction in training did demonstrate improvement. With concentrated practice, residents' observed skills also showed advancement. Furthermore, the implementation of the curriculum was feasible with reasonable cost and limited resident and faculty time. While simulation is well accepted as a modality for teaching surgery, its implementation is often limited due to concerns for time and money. Our experience illustrates that simulation is an effective teaching tool for residents without becoming a burden on a department.

Innovation in Pediatric Surgical Education for General Surgery Residents: A Mobile Web Resource

General surgery residents lack a standardized educational experience in pediatric surgery. We hypothesized that the development of a mobile educational interface would provide general surgery residents broader access to pediatric surgical education materials. We created an educational mobile website for general surgery residents rotating on pediatric surgery, which included a curriculum, multimedia resources, the Operative Performance Rating Scale (OPRS), and Twitter functionality. Residents were instructed to consult the curriculum. Residents and faculty posted media using the Twitter hashtag, #UCLAPedSurg, and following each surgical procedure reviewed performance via the OPRS. Site visits, Twitter posts, and OPRS submissions were quantified from September 2013 to July 2014. The pediatric surgery mobile website received 257 hits; 108 to the homepage, 107 to multimedia, 28 to the syllabus, and 19 to the OPRS. All eligible residents accessed the content. The Twitter hashtag, #UCLAPedSurg, was assigned to 20 posts; the overall audience reach was 85 individuals. Participants in the mobile OPRS included 11 general surgery residents and 4 pediatric surgery faculty. Pediatric surgical education resources and operative performance evaluations are effectively administered to general surgery residents via a structured mobile platform.

Wearable Technology for Global Surgical Teleproctoring

In underserved communities around the world, inguinal hernias represent a significant burden of surgically-treatable disease. With traditional models of international surgical assistance limited to mission trips, a standardized framework to strengthen local healthcare systems is lacking. We established a surgical education model using web-based tools and wearable technology to allow for long-term proctoring and assessment in a resource-poor setting. This is a feasibility study examining wearable technology and web-based performance rating tools for long-term proctoring in an international setting. Using the Lichtenstein inguinal hernia repair as the index surgical procedure, local surgeons in Paraguay and Brazil were trained in person by visiting international expert trainers using a formal, standardized teaching protocol. Surgeries were captured in real-time using Google Glass and transmitted wirelessly to an online video stream, permitting real-time observation and proctoring by mentoring surgeon experts in remote locations around the world. A system for ongoing remote evaluation and support by experienced surgeons was established using the Lichtenstein-specific Operative Performance Rating Scale. Data were collected from 4 sequential training operations for surgeons trained in both Paraguay and Brazil. With continuous internet connectivity, live streaming of the surgeries was successful. The Operative Performance Rating Scale was immediately used after each operation. Both surgeons demonstrated proficiency at the completion of the fourth case. A sustainable model for surgical training and proctoring to empower local surgeons in resource-poor locations and "train trainers" is feasible with wearable technology and web-based communication. Capacity building by maximizing use of local resources and expertise offers a long-term solution to reducing the global burden of surgically-treatable disease.

Validation of a Chicken Wing Training Model for Endoscopic Microsurgical Dissection as Shown by Cadaver Procedure

Objective: To determine if training with a chicken wing model improves performance of endoscopic endonasal surgery with microvascular dissection. Study Design: Randomized controlled trial. Methods: Medical students (MS), residents, and fellows (RF) were randomized into two groups: a control group that performed an endoscopic transantral internal maxillary artery dissection on anatomical specimen, and an interventional group that underwent microvascular dissection training on a chicken wing model before performing the anatomic dissection. The MS were divided randomly; the RF was matched based on previous endoscopic experience. Time to completion and quality of dissection using the Modified Global Rating Scale of Operative Performance were measured. Results: Average MS and RF time versus control scores were 41.14 and 26.44 minutes versus 74.40 and 45.31 minutes, respectively. Average MS and RF quality of intervention scores were 30.6 and 34.60 versus 24.00 and 13.60. Mann-Whitney U tests were significant for time and quality for MS (p = 0.032, 0.008), and RF, (p = 0.016, 0.032). Conclusions: The chicken wing model improves surgical performance in a surrogate model for actual endoscopic endonasal surgery. It is a cost-effective alternative to expensive human specimen materials and may improve operating efficiency.

Assessment of intraoperative judgment during gynecologic surgery using the Script Concordance Test

We sought to develop a valid, reliable assessment of intraoperative judgment by residents during gynecologic surgery based on Script Concordance Theory. This was a multicenter prospective study involving 5 obstetrics and gynecology residency programs. Surgeons from each site generated case scenarios based on common gynecologic procedures. Construct validity was evaluated by correlating scores to training level, in-service examinations, and surgical skill and experience using a Global Rating Scale of Operative Performance and case volumes. A final test that included 42 case scenarios was administered to 75 residents. Internal consistency (Cronbach alpha = 0.73) and test-retest reliability (Lin correlation coefficient = 0.76) were good. There were significant differences between test scores and training levels (P = .002) and test scores correlated with in-service examination scores (r = 0.38; P = .001). There was no association between test scores and total number of cases or technical skills. The Script Concordance Test appears to be a reliable, valid assessment tool for intraoperative decision-making during gynecologic surgery.

Establishing cutoff scores on assessments of surgical skills to determine surgical competence

The aim of this study was to establish minimum cutoff scores on intraoperative assessments of surgical skills to determine surgical competence for vaginal hysterectomy. Two surgical rating scales, the Global Rating Scale of Operative Performance and the Vaginal Surgical Skills Index, were used to evaluate trainees while performing vaginal hysterectomy. Cutoff scores were determined using the Modified Angoff method. Two hundred twelve evaluations were analyzed on 76 surgeries performed by 27 trainees. Trainees were considered minimally competent to perform vaginal hysterectomy if total absolute scores (95% confidence interval) on Global Rating Scale = 18 (16.5-20.3) and Vaginal Surgical Skills Index = 32 (27.7-35.5). On average, trainees met new cutoffs after performing 21 and 27 vaginal hysterectomies, respectively. With the new cutoffs applied to the same cohort of fourth-year obstetrics and gynecology trainees, all residents achieved competency in performing vaginal hysterectomy by the end of their gynecology rotations. Standard-setting methods using cutoff scores may be used to establish competence in vaginal surgery.

Laparoscopic skills assessment of senior obstetrics and gynecologic residents: a city wide experience

To establish feasibility and reliability of a laparoscopic skills assessment when administered to senior obstetrics and gynecologic residents trained at multiple programs. Multi-center, prospective, cohort study. An Objective Structured Assessment of Technical Skills (OSATS) containing 8 skills was administered to 27 senior obstetric and gynecology residents from 5 residency programs in Chicago, IL. The laparoscopic skills station evaluated competency and efficiency of laparoscopic suturing, intracorporeal and extracorporeal knot tying, and usage of endostitch in a specific case scenario. Residents were timed and assessed with 3 types of measures: task-specific checklist, global rating scale of operative performance, and pass/fail grade. Each resident was evaluated by one examiner blinded as to the postgraduate year and program. Two types of reliability were evaluated. Construct validity was assessed using one-way MANOVA with year of clinical training as the independent variable (4 or 5 years) and the station scores as the dependent variables. Ten PGY-3 and 17 PGY-4 residents participated in the exam. Eight of the residents had no experience with laparoscopic suturing. Global rating scale (GRS) of operative performance ranged from 6 to 25 with a maximum score of 30 possible. Those who had no experience in laparoscopic suturing scored the lowest with scores between 6 and 17. The mean score for the GRS was found to be 16.33 ± 1.18 (mean ± standard error of the mean). The overall pass rate was 50%. The mean score for the task-specific checklist was found to be 3.33 ± 0.41 with a maximum score of 7 possible. Mean time of skills completion was 9.97 ± 0.53 minutes. The overall inter-station reliability for the OSATS was 0.55. Controlling for year of training, overall pass rates were similar between programs. Feedback from residents and faculty was universally positive. Developing a multiple program OSATS affords the opportunity to achieve economies of scale and draw from multiple colleagues with a wide variety of experiences. It allows for resident assessment using a regional benchmark and can indicate areas of deficiency in training. Thirty percent of senior residents had no endoscopic suturing experience and 50 percent were unable to complete a single interrupted stitch in the space of 12 minutes. In contrast to our general surgery colleagues, laparoscopic suturing does not appear to be a priority skill among the gynecologic residencies sampled.