Global Evaluative Assessment Of Robotic Skills Score Research Articles

Validation of a Simulation Model for Robotic Myomectomy

Study ObjectiveSeveral simulation models have been evaluated for gynecologic procedures such as hysterectomy, but there are limited published data for myomectomy. This study aimed to assess the validity of a low-cost robotic myomectomy model for surgical simulation training. DesignProspective cohort simulation study. SettingSurgical simulation lab. ParticipantsTwelve obstetrics and gynecology residents and four fellowship-trained minimally invasive gynecologic surgeons were recruited for a 3:1 novice-to-expert ratio. InterventionsA robotic myomectomy simulation model was constructed using <$5 worth of materials: a foam cylinder, felt, a stress ball, bandage wrap, and multipurpose sealing wrap. Participants performed a simulation task involving two steps: fibroid enucleation and hysterotomy repair. Video recorded performances were timed and scored by two blinded reviewers using the validated Global Evaluative Assessment of Robotic Skills (GEARS) scale (5-25 points) and a modified GEARS scale (5-40 points), which adds three novel domains specific to robotic myomectomy. Performance was also scored using predefined task errors. Participants completed a post-task questionnaire assessing the model's realism and utility. Measurements and Main ResultsMedian task completion time was shorter for experts compared to novices (9.7 vs. 24.6 minutes, P=.001). Experts scored higher than novices on both the GEARS scale (median 23 vs. 12, P=.004) and modified GEARS scale (36 vs. 20, P=.004). Experts made fewer task errors than novices (median 15.5 vs. 37.5, P=.034). For interrater reliability of scoring, the intraclass correlation coefficient was calculated to be 0.91 for the GEARS assessment, 0.93 for the modified GEARS assessment, and 0.60 for task errors. Using the contrasting groups method, the passing mark for the simulation task was set to a minimum modified GEARS score of 28 and a maximum of 28 errors. Most participants agreed that the model was realistic (62.5%) and useful for training (93.8%). ConclusionWe have demonstrated evidence supporting the validity of a low-cost robotic myomectomy model. This simulation model and the performance assessments developed in this study provide further educational tools for robotic myomectomy training.

Validation of a multidisciplinary virtual reality (VR) robotic surgical curriculum.

The objective is to identify whether trainees demonstrate improvement in a standardized knot-tying task as assessed by Global Evaluative Assessment of Robotic Skills (GEARS) score after completion of a virtual reality (VR) robotic curriculum. An IRB-exempt prospective study was conducted with surgical trainees from August 2021 to February 2023. Participants initially performed a baseline robotic suturing task in which they were instructed to tie interrupted square knots in 10min. Participants then completed a virtual reality simulation curriculum involving 23 exercises until they achieved 90% proficiency on all tasks. Participants then repeated the suturing task. Pre- and post-curriculum suturing tasks were recorded, de-identified, and scored by expert graders using a GEARS score. Trainees from three academic centers were invited to participate. Medical students (MS1-MS3) and surgical residents from gynecology, urology, and general surgery were invited to participate. Twenty-five trainees completed the pre-curriculum suturing task, the VR curriculum, and the post-curriculum suturing task. Trainees demonstrated significant improvement in their post-test GEARS score by 2.43 points (p < 0.05) and were able to tie three additional knots within 10min after completion of the curriculum (p < 0.05). Trainees also demonstrated a faster time to complete first knot (114s improvement, p < 0.05) after completion of the curriculum. All participants agreed or strongly agreed that completion of the robotic curriculum helped them feel more comfortable using the robotic console, and improved their robotic surgical skills. Surgical trainees and medical students with limited prior robotic surgical experience demonstrated objective improvement after completion of a standardized VR curriculum.

Intentional enterotomies: validation of a novel robotic surgery training exercise.

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.

The Surgical Learning Curve: Does Robotic Technical Skill Explain Differences in Operative Performance?

Background: Prior studies on technical skills use small collections of videos for assessment. However, there is likely heterogeneity of performance among surgeons and likely improvement after training. If technical skill explains these differences, then it should vary among practicing surgeons and improve over time. Materials and Methods: Sleeve gastrectomy cases (n = 162) between July 2018 and January 2021 at one health system were included. Global evaluative assessment of robotic skills (GEARS) scores were assigned by crowdsourced evaluators. Videos were manually annotated. Analysis of variance was used to compare continuous variables between surgeons. Tamhane's post hoc test was used to define differences between surgeons with the eta-squared value for effect size. Linear regression was used for temporal changes. A P value <.05 was considered significant. Results: Variations in operative time discriminated between individuals (e.g., between 2 surgeons, means were 91 and 112 minutes, Tamhane's = 0.001). Overall, GEARS scores did not vary significantly (e.g., between those 2 surgeons, means were 20.32 and 20.6, Tamhane's = 0.151). Operative time and total GEARS score did not change over time (R2 = 0.0001-0.096). Subcomponent scores showed idiosyncratic temporal changes, although force sensitivity increased among all (R2 = 0.172-0.243). For a novice surgeon, phase-adjusted operative time (R2 = 0.24), but not overall GEARS scores (R2 = 0.04), improved over time. Conclusions: GEARS scores showed less variability and did not improve with time for a novice surgeon. Improved technical skill does not explain the learning curve of a novice surgeon or variation among surgeons. More work could define valid surrogate metrics for performance analysis.

Parallel, component training in robotic total mesorectal excision.

There has been widespread adoption of robotic total mesorectal excision (TME) for rectal cancer in recent years. There is now increasing interest in training robotic novice surgeons in robotic TME surgery using the principles of component-based learning. The aims of our study were to assess the feasibility of delivering a structured, parallel, component-based, training curriculum to surgical trainees and fellows. A prospective pilot study was undertaken between January 2021 and May 2021. A dedicated robotic training pathway was designed with two trainees trained in parallel per each robotic case based on prior experience, training grade and skill set. Component parts of each operation were allocated by the robotic trainer prior to the start of each case. Robotic proficiency was assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and the EARCS Global Assessment Score (GAS). Three trainees participated in this pilot study, performing a combined number of 52 TME resections. Key components of all 52 TME operations were performed by the trainees. GEARS scores improved throughout the study, with a mean overall baseline score of 17.3 (95% CI 15.1-1.4) compared to an overall final assessment mean score of 23.8 (95% CI 21.6-25.9), p = 0.003. The GAS component improved incrementally for all trainees at each candidate assessment (p < 0.001). Employing a parallel, component-based approach to training in robotic TME surgery is safe and feasible and can be used to train multiple trainees of differing grades simultaneously, whilst maintaining high-quality clinical outcomes.

Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery?

The Global Evaluative Assessment of Robotic Skills (GEARS) rubric provides a measure of skill in robotic surgery. We hypothesize surgery performed by more experienced operators will be associated with higher GEARS scores. Patients undergoing sleeve gastrectomy from 2016 to 2020 were analyzed. Three groups were defined by time in practice: less than 5, between 5 and 15, and more than 15years. Continuous variables were compared with ANOVA and multivariable regression was performed. Fourteen operators performing 154 cases were included. More experienced surgeons had higher GEARS scores and shorter operative times. On multivariable regression, operative time (P = 0.027), efficiency (P = .022), depth perception (P = 0.033), and bimanual dexterity (P = 0.047) were associated with experience. In our video-based assessment (VBA) model, operative time and several GEARS subcomponent scores were associated with surgical experience. Further studies should determine the association between these metrics and surgical outcomes.

The transferability of laparoscopic and open surgical skills to robotic surgery

BackgroundWithin the last decades, robotic surgery has gained popularity. Most robotic surgeons have changed their main surgical activity from open or laparoscopic without prior formal robotic training. With the current practice, it is of great interest to know whether there is a transfer of surgical skills. In visualization, motion scaling, and freedom of motion, robotic surgery resembles open surgery far more than laparoscopic surgery. Therefore, our hypothesis is that open-trained surgeons have more transfer of surgical skills to robotic surgery, compared to surgeons trained in laparoscopy.MethodsThirty-six surgically inexperienced medical students were randomized into three groups for intensive simulation training in an assigned modality: open surgery, laparoscopy, or robot-assisted laparoscopy. The training period was, for all study subjects, followed by performing a robot-assisted bowel anastomosis in a pig model. As surrogate markers of surgical quality, the anastomoses were tested for resistance to pressure, and video recordings of the procedure were evaluated by two blinded expert robotic surgeons, using a global rating scale of robotic operative performance (Global Evaluative Assessment of Robotic Skills (GEARS)).ResultsThe mean leak pressure of bowel anastomosis was 36.25 (7.62–64.89) mmHg in the laparoscopic training group and 69.01 (28.02–109.99) mmHg in the open surgery group, and the mean leak pressure for the robotic training group was 108.45 (74.96–141.94) mmHg. The same pattern was found with GEARS as surrogate markers of surgical quality. GEARS score was 15.71 (12.37–19.04) in the laparoscopic training group, 18.14 (14.70–21.58) in the open surgery group, and 22.04 (19.29–24.79) in the robotic training group. In comparison with the laparoscopic training group, the robotic training group had a statistically higher leak pressure (p = 0.0015) and GEARS score (p = 0.0023). No significant difference, for neither leak pressure nor GEARS, between the open and the robotic training group.ConclusionIn our study, training in open surgery was superior to training in laparoscopy when transitioning to robotic surgery in a simulation setting performed by surgically naive study subjects.

Intraoperative robotic-assisted low anterior rectal resection performance assessment using procedure-specific binary metrics and a global rating scale.

BackgroundThis study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure.MethodA prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated.ResultsTwenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 – Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019–0.008; GEARS scores, P = 0.029–0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15–41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58–64 per cent).ConclusionsPBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.

MP10-07 THE RELATIONSHIP OF TECHNICAL SKILLS AND COGNITIVE WORKLOAD TO ERRORS DURING ROBOTIC SURGICAL EXERCISES

You have accessJournal of UrologyCME1 May 2022MP10-07 THE RELATIONSHIP OF TECHNICAL SKILLS AND COGNITIVE WORKLOAD TO ERRORS DURING ROBOTIC SURGICAL EXERCISES Sidney I. Roberts, Steven Cen, Jessica H. Nguyen, Laura C. Perez, Luis G. Medina, Runzhuo Ma, Sandra P. Marshall, Rafal Kocielnik, Anima Anandkumar, and Andrew J. Hung Sidney I. RobertsSidney I. Roberts More articles by this author , Steven CenSteven Cen More articles by this author , Jessica H. NguyenJessica H. Nguyen More articles by this author , Laura C. PerezLaura C. Perez More articles by this author , Luis G. MedinaLuis G. Medina More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Sandra P. MarshallSandra P. Marshall More articles by this author , Rafal KocielnikRafal Kocielnik More articles by this author , Anima AnandkumarAnima Anandkumar More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000002532.07AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Surgical performance and skill in robotic surgery has been shown to impact patient outcomes and complication rates. Cognitive workload, or mental strain in the working memory, has been shown to differ between surgeon experience levels. Herein, we attempt to understand the relationship between surgeon technical skills, cognitive workload, and discrete errors committed during a simulated robotic dissection task. METHODS: Participant surgeons performed a robotic surgery dissection exercise (peeling a clementine, removing a single wedge). Participants were grouped based on surgical experience: novice (no prior surgical experience), intermediate (<100 robotic cases), and expert (≥100 cases). Technical skills were evaluated utilizing the validated Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The task was also evaluated for errors during active dissection or passive retraction. Cognitive workload was quantified as an Index of Cognitive Activity (ICA), derived from Task-Evoked-Pupillary-Response metrics; ICA ranged 0-1, with 1 representing maximum ICA. Generalized Estimating Equation (GEE) was used for all modelings to establish relationships between surgeon technical skills, cognitive workload and errors. RESULTS: Overall there were 22 patients: 7 novices, 9 intermediates, and 6 experts. We found a strong association between technical skills, as measured by multiple GEARS domains (depth perception, force sensitivity and robotic control), and passive errors - with higher GEARS scores associated with a lower relative risk of errors (all p <0.01). For novice surgeons, as average GEARS scores increased, the average estimated ICA decreased. In contrast, as average GEARS increased for expert surgeons, the average estimated ICA increased. When exhibiting optimal technical skill (maximal GEARS scores) novices and experts had a similar range of ICA scores (ICA 0.47 and 0.42, respectively) (Figure 1). CONCLUSIONS: This study found that there is an optimal cognitive workload level for surgeons of all experience levels during our robotic surgical exercise. Select technical skills were strong predictors of errors. Future research will explore whether an ideal cognitive workload range truly optimizes surgical performance and reduce surgical errors. Source of Funding: This study was supported in part by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K23EB026493 © 2022 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 207Issue Supplement 5May 2022Page: e148 Advertisement Copyright & Permissions© 2022 by American Urological Association Education and Research, Inc.MetricsAuthor Information Sidney I. Roberts More articles by this author Steven Cen More articles by this author Jessica H. Nguyen More articles by this author Laura C. Perez More articles by this author Luis G. Medina More articles by this author Runzhuo Ma More articles by this author Sandra P. Marshall More articles by this author Rafal Kocielnik More articles by this author Anima Anandkumar More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF DownloadLoading ...

The Relationship Between Technical Skills, Cognitive Workload, and Errors During Robotic Surgical Exercises.

Purpose: We attempt to understand the relationship between surgeon technical skills, cognitive workload, and errors during a simulated robotic dissection task. Materials and Methods: Participant surgeons performed a robotic surgery dissection exercise. Participants were grouped based on surgical experience. Technical skills were evaluated utilizing the validated Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The dissection task was evaluated for errors during active dissection or passive retraction maneuvers. We quantified cognitive workload of surgeon participants as an index of cognitive activity (ICA), derived from task-evoked pupillary response metrics; ICA ranged 0 to 1, with 1 representing maximum ICA. Generalized estimating equation (GEE) was used for all modelings to establish relationships between surgeon technical skills, cognitive workload, and errors. Results: We found a strong association between technical skills as measured by multiple GEARS domains (depth perception, force sensitivity, and robotic control) and passive errors, with higher GEARS scores associated with a lower relative risk of errors (all p < 0.01). For novice surgeons, as average GEARS scores increased, the average estimated ICA decreased. In contrast, as average GEARS increased for expert surgeons, the average estimated ICA increased. When exhibiting optimal technical skill (maximal GEARS scores), novices and experts reached a similar range of ICA scores (ICA: 0.47 and 0.42, respectively). Conclusions: This study found that there is an optimal cognitive workload level for surgeons of all experience levels during our robotic surgical exercise. Select technical skill domains were strong predictors of errors. Future research will explore whether an ideal cognitive workload range truly optimizes surgical training and reduces surgical errors.

Virtual Reality Simulation Has Weak Correlation with Overall Trainee Robot-Assisted Laparoscopic Hysterectomy Performance

Both simulator practice and intraoperative performance serve to inform surgical trainee training, but the skill transfer from simulation to the intraoperative setting remains unclear. This study evaluates the correlation between trainee performance on virtual reality simulation and (1) overall intraoperative performance during robotic-assisted laparoscopic hysterectomy (RALH) procedures and (2) suturing performance during vaginal cuff closure portion of the case. Retrospective subgroup analysis of randomized controlled trial. Academic hospital. Patients with RALH (N=29). Gynecological trainees (N=21) performed simulation tasks using the da Vinci skills simulator on the day of surgery before performing RALH. Attending surgeons assessed participants' intraoperative performance using Global Evaluative Assessment of Robotic Skills (GEARS). Performance of the vaginal cuff closure step was subsequently assessed using GEARS scoring of anonymized videos. Spearman's correlation was used to quantify the relationship between simulation and intraoperative performances. Trainees achieved a median intraoperative GEARS score of 18.5/30 (interquartile range: 17-22) and a median total simulator score of 84.4/100 (interquartile range: 78.1-87.5). More advanced residents exhibited worse overall simulator performance (median score 86.6/100 compared with 78.8/100, p=.03) and similar intraoperative GEARS scores during overall RALH and vaginal cuff closure compared with less experienced trainees. Total simulation performance score was negatively correlated with GEARS Bimanual Dexterity (ρ=-0.46, p=.02) and Force Sensitivity subscores (ρ=-0.39, p=.05). There was no correlation between total GEARS intraoperative vaginal cuff closure scores and overall simulation performances; however, total Tubes simulation score was correlated with higher GEARS Force Sensitivity subscore (ρ=0.73, p=.05). In this study, there was limited correlation between simulation score metrics and trainees' overall intraoperative performance. Furthermore, we identified that GEARS scores could not distinguish between similar trainee skill levels. These findings underscore the need to develop intraoperative assessment tools that can better discriminate different but similar skill levels.

Assessment of the training program for Versius, a new innovative robotic system for use in minimal access surgery

ObjectivesThe Versius surgical system has been developed for use in robot-assisted minimal access surgery (MAS). This study aimed to evaluate the effectiveness of the Versius training program.DesignA 3.5-day program following...

Validation of a Simulation Model for Robotic Sacrocolpopexy.

We sought to validate a simulation model for robotic sacrocolpopexy (RSCP) that includes multiple steps: presacral dissection/mesh attachment, vaginal mesh attachment, and peritoneal closure. An RSCP training model was developed. Female pelvic medicine and reconstructive surgery (FPMRS) experts and current FPMRS fellows were videotaped using the model; sessions were timed and scored using the Global Evaluative Assessment of Robotic Skills (GEARS) by 3 surgeon reviewers masked to participants' identities. Construct validity was measured by comparing performance on the model between experts and trainees. Interrater reliability was determined by calculating intraclass correlation coefficients for total GEARS scores. Face validity was assessed by a postprocedure questionnaire. Experts included 9 board-certified FPMRS physicians experienced in RSCP; trainees were 17 fellows. Experts practiced at 7 different institutions in the United States, and the majority (5/7) taught fellows. Trainees were from 7 institutions and in various years of training: postgraduate year (PGY) 5 (n = 6), PGY 6 (n = 5), and PGY 7 (n = 6). Experts' performances were rated significantly higher for total GEARS scores and for relevant domains of the GEARS scale. Intraclass correlation coefficient for the 3 reviewer pairs (0.96-0.99) indicated high interrater reliability. All participants "agreed/strongly agreed" that the model closely approximated live RSCP surgery and was useful for teaching and learning the procedure, indicating high face validity. This novel, multistep simulation model demonstrated construct validity and high interrater reliability. Face validity was also established. Consequently, this RSCP model could be used for surgical training and assessment of these discrete surgical skill steps.

Comparison of Robotic Surgery Video Quality Between YouTube and Curated Sources Using GEARS Criteria

ObjectiveTo compare the quality of robotic prostatectomy surgical videos on the popular website YouTube with more curated, professional sources using the Global Evaluative Assessment of Robotic Skills (GEARS) criteria. MethodsA search was performed on YouTube for robotic prostatectomy. Results were sorted by views and the first ten that met inclusion criteria were selected for review. To represent curated sources five robotic prostatectomy videos were selected from the DaVinci Surgery Community (DVS) video repository and the AUA Surgical Video Library in order of publishing from present to past. Videos were edited to be deidentified. The videos were reviewed blindly in parallel and graded using the GEARS criteria. Concordance among reviewers was measured using Chronbach's alpha. Comparisons between groups were made using student t-test. ResultsThere was a high level of reliability of overall GEARS scores between reviewers for each video (α = 0.843). There was no significant difference between overall GEARS scores between the YouTube videos (mean 24.8, SDEV 1.85) and the AUA group (mean 24.3, SDEV 6.18) (P = 0.78). YouTube videos scored higher than the DVS videos (mean 22.1, SDEV 2.34) (P 0.03). ConclusionDespite concerns about the quality of surgical videos on YouTube for education, the most viewed surgical videos for robot assisted laparoscopic prostatectomy score as well or better than more curated sources using the GEARS criteria. This may represent selection via crowd sourcing of the best videos amongst a much larger overall quantity.

Quantifying the “Assistant Effect” in Robotic-Assisted Radical Prostatectomy (RARP): Measures of Technical Performance

Surgeons are reliant on the bedside assistant during robotic surgeries. Using a modified global rating scale (GRS), we aim to assess the association between an assistant's technical skill on surgeon performance in Robotic-Assisted Radical Prostatectomy (RARP). Prospective, intraoperative video from RARP cases at three centers were collected. Baseline demographic and RARP-experience data were collected from participating surgeons and trainees. The dissection of the prostatic pedicle and neurovascular bundle step (NVB) was analyzed. Expert analysts scored the console surgeon performance using the Global Evaluative Assessment of Robotic Skills (GEARS), and the bedside assistant performance using a modified Objective Structured Assessment of Technical Skills (aOSATS). The primary outcome is the association between console surgeon performance, as measured by GEARS, and assistant skill, as measured by aOSATS. Spearman's rho correlations were used to test the relationship between assistant and surgeon technical performance, and a multivariable linear regression model was created to test this association while controlling for patient factors. 92 RARP cases were available for the analysis, comprising 14 console surgeons and 22 different bedside assistants. In only 5 (5.4%) cases, the neurovascular bundle step was completed by a trainee, and in 13 (14.1%) of cases, a staff-level surgeon acted as the bedside assistant. aOSATS score was significantly associated with robotic console experience (P=0.011), and prior laparoscopic experience (P<0.001). Assistant aOSATS score showed a weak but significant correlation with surgeon GEARS score during the neurovascular bundle step (spearman's rho=0.248, P=0.028). On linear regression, aOSATS remained a significant predictor of console surgeon performance (P=0.016), after controlling for patient age and BMI, prostate volume, tumor stage, and presence of nerve-sparing. This is the first study to assess the association between assistant technical skill and surgeon performance in RARP. Additionally, we have provided validity evidence for a modified OSATS global rating scale for training and assessing bedside assistant performance.

Ex Vivo Porcine Model for Robot-Assisted Partial Nephrectomy Simulation at a High-Volume Tertiary Center: Resident Perception and Validation Assessment Using the Global Evaluative Assessment of Robotic Skills Tool.

Introduction: With increased demands on surgeon productivity and outcomes, residency robotics training increasingly relies on simulations. The objective of this study is to assess the validity and effectiveness of an ex vivo porcine training model as a useful tool to improve surgical skill and confidence with robot-assisted partial nephrectomy (RAPN) among urology residents. Methods: A 2.5 cm circular area of ex vivo porcine kidneys was marked as the area of the tumor. Tumor excision and renorrhaphy was performed by trainees using a da Vinci Si robot. All residents ranging from postgraduate year (PGY) 2 to 5 participated in four training sessions during the 2017 to 2018 academic year. Each session was videorecorded and scored using the global evaluative assessment of robotic skills (GEARS) by faculty members. Results: Twelve residents completed the program. Initial mean GEARS score was 16.7 and improved by +1.4 with each subsequent session (p = 0.008). Initial mean excision, renorrhaphy, and total times were 8.2, 13.9, and 22.1 minutes, which improved by 1.6, 2.0, and 3.6 minutes, respectively (all p < 0.001). Residents' confidence at performing RAPN and robotic surgery increased after completing the courses (p = 0.012 and p < 0.001, respectively). Overall, residents rated that this program has greatly contributed to their skill (4/5) and confidence (4.1/5) in robotic surgery. Conclusions: An ex vivo porcine simulation model for RAPN and robotic surgery provides measurable improvement in GEARS score and reduction in procedural time, although significant differences for all PGY levels need to be confirmed with larger study participation. Adoption of this simulation in a urology residency curriculum may improve residents' skill and confidence in robotic surgery.

The influence of reviewer's occupation on the skill assessment of urethrovesical anastomosis in robot-assisted radical prostatectomy.

In recent years, crowd-sourced assessments have been reported as a timesaving, cost-efficient, and practicable method of surgical skill evaluation. However, the differences in the assessment of surgical skills by the individual reviewers cannot be further examined in terms of characteristics of the reviewers because they are usually anonymously and randomly selected. This study aimed to reveal the effects of reviewers' occupations on their assessment of a surgeon's skill. In total, 42 urologists, 19 paramedics, 73 medical students, and 28 non-medical personnel used the Global Evaluative Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool to assess the surgical skill of surgeons in nine edited video clips of complete urethrovesical anastomosis during a robot-assisted radical prostatectomy. The total GEARS scores of the four groups of reviewers were compared, and the similarities and the differences between the ratings of the urologists group and those of the other three groups were subsequently investigated. The rankings of video clips in the order of GEARS scores were very similar in each group, and a strong positive correlation (R2 values >0.8) was observed between the scores assigned by the urologists group and those assigned by the other three groups. Our findings indicate that the crude evaluation of robot-assisted urethrovesical anastomosis is not affected by the reviewers' occupations. Non-medical personnel may be able to provide a rudimentary screening evaluation of surgical skill.

A Novel Resident Robotic Curriculum Using Self-Selection for Advanced Training and a Single Web-Based Tool for Feedback and Case-Tracking

Study Objective Describe a novel two-phase robotic curriculum facilitated by a web-based feedback and case-tracking tool, which allows residents to self-select into advanced training. Design Descriptive. Setting Academic Hospital. Patients or Participants OBGYN residents and faculty. Interventions Phase I (Basic), required of all residents, includes online training modules, assessment, and robotic dry lab. Phase II (Advanced) console training is elective. Residents complete 10 SIM drills to proficiency before performing live surgery. A web-based tool is used for surgical feedback. Resident online assessments, SIM lab reports, feedback, and case-logs were reviewed (7/2018-6/2019). Attendings and residents completed a beta-tested satisfaction survey. Measurements and Main Results All 24 residents completed Phase I training. Phase II was initiated by 12 residents and 10/12(83.3%) completed all 10 SIM drills. A median of 10(3,26) attempts were required to complete each drill with a mean time of 5.1hrs to complete all drills. 128 web-based post-surgical feedback entries were completed. Residents performed as bedside assistants (75%,n=96), console surgeon (5.5%,n=7), or both (19.5%,n=25). Most common tracked procedure was hysterectomy 111/193(57.5%). Console surgeons performed 32 cases with a mean console time of 34.6±19.5min. Most common procedure steps included: colpotomy/cuff closure 31.1%, uterine artery 75% and utero-ovarian vessel seal/transection 84.4%(n=32). 28/32(87.5%) Global Evaluative Assessment of Robotic Skills (GEARS) assessments were completed. Mean GEARS score 20.6±3.7(n=28), communication (4.2±0.8,n=61), workload management (3.9±0.9,n=54), and team skills (4.3±0.8,n=60). Residents completing >50% of case were assessed as “apprentice” 38.5% or “competent” 23.1%(n=13). 27/44(61.4%), satisfaction survey response rate. After curriculum change, attendings considered residents prepared as bedside assistants and console surgeons (100%), whereas only 16.7% were prepared before (n=6). Respondents were satisfied with the curriculum (95.0%), found the web-based tool helpful to give/receive feedback (90.0%), and recommended continued use (90.0%) (n=19). Conclusion This novel robotic curriculum allows residents to self-select into advanced training. A web-based tool improves individualized feedback and case-tracking alleviates many challenges of graduated robotic training.

Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery.

Operative time has been traditionally used as a proxy for surgical skill and is commonly utilized to measure the learning curve, assuming that faster operations indicate a more skilled surgeon. The Global Evaluative Assessment of Robotic Skills (GEARS) rubric is a validated Likert scale for evaluating technical skill. We hypothesize that operative time will not correlate with the GEARS score. Patients undergoing elective robotic sleeve gastrectomy at a single bariatric center of excellence hospital from January 2019 to March 2020 were captured in a prospectively maintained database. For step-specific scoring, videos were broken down into three steps: ligation of short gastric vessels, gastric transection, and oversewing the staple line. Overall and step-specific GEARS scores were assigned by crowd-sourced evaluators. Correlation between operative time and GEARS score was assessed with linear regression and calculation of the R2 statistic. Sixty-eight patients were included in the study, with a mean operative time of 112 ± 27.4min. The mean GEARS score was 20.1 ± 0.81. Mean scores for the GEARS subcomponents were: bimanual dexterity 4.06 ± 0.17; depth perception 3.96 ± 0.24; efficiency 3.82 ± 0.19; force sensitivity 4.06 ± 0.20; robotic control 4.16 ± 0.21. Operative time and overall score showed no correlation (R2 = 0.0146, p = 0.326). Step-specific times and scores showed weak correlation for gastric transection (R2 = 0.0737, p = 0.028) and no correlation for ligation of short gastric vessels (R2 = 0.0262, p = 0.209) or oversewing the staple line (R2 = 0.0142, p = 0.344). Operative time and crowd-sourced GEARS score were not correlated. Operative time and GEARS scores measure different performance characteristics, and future studies should consider using both a validated skills assessment tool and operative time for a more complete evaluation of skill.

Robotic dismembered pyeloplasty surgical simulation using a 3D-printed silicone-based model: development, face validation and crowdsourced learning outcomes assessment.

Ureteropelvic junction obstruction (UPJO) is an uncommonly encountered pathology, posing a challenge for resident training. We describe the development and face validation of a robotic pyeloplasty simulation using a 3D-printed silicone-based model of UPJO for surgical training, in combination with crowdsourced scoring to objectively assess performance and learning outcomes. The organs were created using 3D modeling software and printed using a silicone-based material by Lazarus 3D, LLC. They were secured in a laparoscopic box trainer and the robotic system was docked. Eight residents and three faculty each performed two robotic-assisted right dismembered pyeloplasties on separate occaisions. Face validity was evaluated on a 5-point Likert scale. Crowd-Sourced Assessment of Technical Skills (C-SATS Inc.) scored surgical performance using the Global Evaluative Assessment of Robotic Skills (GEARS) criteria, based on video review of each simulation. All participants completed the simulation twice with fully patent anastomoses. Average time to complete the first and second trials was 44.4min and 43.2min, respectively. The average GEARS score was 17.1 and 17.6 for the first and second trials respectively. Participants improved on average in all 5 GEARS categories, with significant improvement in depth perception (p = 0.006). The model received mean scores (out of 5) of 4.36 for aesthetics, 4.18 for overall feel, 3.55 for realism, 4.72 for usability, and 4.72 for suturability. Residents had a significant increase in confidence between initial and final surveys on a 5-point Likert Scale: 1.63 vs. 2.38 (p = 0.03). Using 3D-printed silicone-based models, participants completed robotic-assisted dismembered pyeloplasties for training and skill acquisition. We demonstrated face validity of the simulation, which was also found to improve participant speed and significantly improve resident confidence. Crowdsourced assessment demonstrated significant improvement in depth perception.