Assessment Of Robotic Skills Research Articles

Fellow perceptions of robotic surgery preparedness for fellowship.

Many surgical fellowship programs incorporate robotic surgery, yet not all residency programs offer robotic training for residents. Given the variability of pre-fellowship robotic exposure, the goal of this study was to explore whether standardized robotic skills assessment would be useful to fellows at the start of their training. In partnership with the Fellowship Council, we surveyed current fellows of accredited Thoracic, Colorectal, HPB, and Advanced GI/MIS/Bariatrics/Foregut Fellowship programs. Main outcome measurements included amount of robotic exposure during residency, fellows' robotic skills at the start of fellowship, and fellows' attitudes towards a standardized skills assessment. In total, 78 fellows completed the survey. Mean percentage of cases performed robotically during fellowship was 38.1% (SD ± 32.25%). From our respondents, 46% had no robotic curriculum during residency and 86.9% felt a standardized robotic curriculum during residency would have been beneficial. When asked if they started fellowship with adequate robotic skills to operate autonomously, 31% strongly agreed, but 24.4% strongly disagreed. The majority of fellows reported their fellowship program did not conduct an assessment of their robotic skills (71.5%), or provide a specific robotic curriculum (75.6%). On the other hand, 73.3% felt a formal proficiency assessment at the start of fellowship would be helpful for individualized support and training. Given the significant variability in resident exposure to robotic surgery, a standardized robotic curriculum during residency would likely be beneficial. Additionally, a robotic skills assessment at the start of fellowship could help provide a tailored training experience for fellows interested in this skillset.

Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill.

Surgical skill evaluation that relies on subjective scoring of surgical videos can be time-consuming and inconsistent across raters. We demonstrate differentiated opportunities for objective evaluation to improve surgeon training and performance. Subjective evaluation was performed using the Global evaluative assessment of robotic skills (GEARS) from both expert and crowd raters; whereas, objective evaluation used objective performance indicators (OPIs) derived from da Vinci surgical systems. Classifiers were trained for each evaluation method to distinguish between surgical expertise levels. This study includes one clinical task from a case series of robotic-assisted sleeve gastrectomy procedures performed by a single surgeon, and two training tasks performed by novice and expert surgeons, i.e., surgeons with no experience in robotic-assisted surgery (RAS) and those with more than 500 RAS procedures. When comparing expert and novice skill levels, OPI-based classifier showed significantly higher accuracy than GEARS-based classifier on the more complex dissection task (OPI 0.93 ± 0.08 vs. GEARS 0.67 ± 0.18; 95% CI, 0.16-0.37; p = 0.02), but no significant difference was shown on the simpler suturing task. For the single-surgeon case series, both classifiers performed well when differentiating between early and late group cases with smaller group sizes and larger intervals between groups (OPI 0.9 ± 0.08; GEARS 0.87 ± 0.12; 95% CI, 0.02-0.04; p = 0.67). When increasing the group size to include more cases, thereby having smaller intervals between groups, OPIs demonstrated significantly higher accuracy (OPI 0.97 ± 0.06; GEARS 0.76 ± 0.07; 95% CI, 0.12-0.28; p = 0.004) in differentiating between the early/late cases. Objective methods for skill evaluation in RAS outperform subjective methods when (1) differentiating expertise in a technically challenging training task, and (2) identifying more granular differences along early versus late phases of a surgeon learning curve within a clinical task. Objective methods offer an opportunity for more accessible and scalable skill evaluation in RAS.

Development and evaluation of a societal core robotic surgery accreditation curriculum for the UK

Standardised proficiency-based progression is the cornerstone of safe robotic skills acquisition, however, is currently lacking within surgical training curricula. Expert consensuses have defined a modular pathway to accredit surgeons. This study aimed to address the lack of a formal, pre-clinical core robotic skills, proficiency-based accreditation curriculum in the UK. Novice robotic participants underwent a four-day pre-clinical core robotic skills curriculum incorporating multimodal assessment. Modifiable-Global Evaluative Assessment of Robotic Skills (M-GEARS), VR-automated performance metrics (APMs) and Objective Clinical Human Reliability Analysis (OCHRA) error methodology assessed performance at the beginning and end of training. Messick’s validity concept and a curriculum evaluation model were utilised. Feedback was collated. Proficiency-based progression, benchmarking, tool validity and reliability was assessed through comparative and correlational statistical methods. Forty-seven participants were recruited. Objective assessment of VR and dry models across M-GEARS, APMs and OCHRA demonstrated significant improvements in technical skill (p < 0.001). Concurrent validity between assessment tools demonstrated strong correlation in dry and VR tasks (r = 0.64–0.92, p < 0.001). OCHRA Inter-rater reliability was excellent (r = 0.93, p < 0.001 and 81% matched error events). A benchmark was established with M-GEARS and for the curriculum at 80%. Thirty (63.82%) participants passed. Feedback was 5/5 stars on average, with 100% recommendation. Curriculum evaluation fulfilled all five domains of Messick’s validity. Core robotic surgical skills training can be objectively evaluated and benchmarked to provide accreditation in basic robotic skills. A strategy is necessary to enrol standardised curricula into national surgical training at an early stage to ensure patient safety.

Differences in coaching in single- versus dual-console robotic cases: a mixed-methods study.

A growing importance has been placed on development of trainee robotic surgical skills through simulation curricula and intraoperative experience. However, few studies have examined how console case type impacts learning outcomes.We sought to evaluate how intraoperative coaching and resident autonomy differ based on the use of a single- versus dual-console robot. Robotic single- and dual-console cases from February to September 2023 at a single institution were included. Faculty and trainees wore microphones to capture audio during the case. Pre/post surveys were administered, which included metrics on faculty coaching based on the Wisconsin Surgical Coaching Rubric (WiSCoR) and on trainee technical performance based on the Global Evaluative Assessment of Robotic Skills (GEARS). Statistical analysis of survey data was performed using SPSS. Audio from cases was coded by 2 researchers with a deductive approach using WiSCoR as a framework. Data were collected for 7 (38.9%) single and 11 (61.1%) dual-console cases across 9 case types from 4 surgical specialties. Chi-square analysis demonstrated no significant difference in percentage of case trainee spent in the operating surgeon role based on trainee level or console case type. Independent t-tests showed no significant difference in trainee autonomy, trainee performance, or faculty coaching scores based on console case type. Trainees rated faculty highest in WiSCoR Domains 1 (sharing responsibility) and 3 (providing constructive feedback). Qualitative analysis showed that for single-console cases, Domain 4 (goal setting) was most represented (34.0% of comments), while for dual-console cases, Domain 1 was most represented (37.0% of comments). Qualitative analysis highlights that despite similar survey-based faculty ratings across domains, coaching on self-reflection (Domain 2) is infrequently done, highlighting an opportunity for improvement in this area of coaching during robotic surgery.

Association of Crowd-Sourced Assessment of Technical Skills and Outcomes of Robotic-assisted Radical Prostatectomy

ObjectiveTo investigate if use of the Crowd-Sourced Assessment of Technical Skills (CSATSTM) platform and video peer review with constructive feedback is associated with improvement in technical skill and patient outcomes for robotic-assisted laparoscopic prostatectomy (RALP). MethodsFive fellowship-trained urologists voluntarily submitted RALP cases for CSATSTM Global Evaluative Assessment of Robotic Skills (GEARS) scoring and expert narrative review between April 15, 2022 - April 30, 2023. Surgeon-selected and randomly-selected cases were reviewed. Surgeons underwent local peer review of videos with constructive feedback. Change in GEARS scores and frequency of post-operative outcomes over the 12-month periods before and during the study were analyzed in logistic regression models. Bias was assessed with sensitivity analysis comparing surgeon-selected to randomly-selected cases. ResultsGEARS scores for randomly-selected versus surgeon-selected cases did not differ significantly. Overall GEARS score correlated positively with annual surgical RALP volume (r=0.39, p=0.003) and negatively with years in practice (r=-0.34, p=0.01). After adjusting for confounders, there was no significant improvement in overall GEARS Score (0.01±0.02/month, p=0.48); but likelihood of sepsis (Odds Ratio 0.07, 95% CI 0.01-1.00, p=0.05) and pelvic fluid collection (Odds Ratio 0.09, 95% CI 0.01-0.99, p=0.049) were significantly decreased during the intervention period (n=165) compared to the prior 12 months (n=144). No outcome increased in likelihood (p>0.05). ConclusionsIntegration of CSATSTM and local video peer review is associated with significant improvement in patient outcomes after RALP, despite no significant change in surgeons’ GEARS scores. This is the first study demonstrating improvement in patient RALP outcomes after implementation of such a paradigm in practicing surgeons.

Validation of a Simulation Model for Robotic Myomectomy

Study ObjectiveSeveral simulation models have been evaluated for gynecologic procedures such as hysterectomy, but there are limited published data for myomectomy. This study aimed to assess the validity of a low-cost robotic myomectomy model for surgical simulation training. DesignProspective cohort simulation study. SettingSurgical simulation lab. ParticipantsTwelve obstetrics and gynecology residents and four fellowship-trained minimally invasive gynecologic surgeons were recruited for a 3:1 novice-to-expert ratio. InterventionsA robotic myomectomy simulation model was constructed using <$5 worth of materials: a foam cylinder, felt, a stress ball, bandage wrap, and multipurpose sealing wrap. Participants performed a simulation task involving two steps: fibroid enucleation and hysterotomy repair. Video recorded performances were timed and scored by two blinded reviewers using the validated Global Evaluative Assessment of Robotic Skills (GEARS) scale (5-25 points) and a modified GEARS scale (5-40 points), which adds three novel domains specific to robotic myomectomy. Performance was also scored using predefined task errors. Participants completed a post-task questionnaire assessing the model's realism and utility. Measurements and Main ResultsMedian task completion time was shorter for experts compared to novices (9.7 vs. 24.6 minutes, P=.001). Experts scored higher than novices on both the GEARS scale (median 23 vs. 12, P=.004) and modified GEARS scale (36 vs. 20, P=.004). Experts made fewer task errors than novices (median 15.5 vs. 37.5, P=.034). For interrater reliability of scoring, the intraclass correlation coefficient was calculated to be 0.91 for the GEARS assessment, 0.93 for the modified GEARS assessment, and 0.60 for task errors. Using the contrasting groups method, the passing mark for the simulation task was set to a minimum modified GEARS score of 28 and a maximum of 28 errors. Most participants agreed that the model was realistic (62.5%) and useful for training (93.8%). ConclusionWe have demonstrated evidence supporting the validity of a low-cost robotic myomectomy model. This simulation model and the performance assessments developed in this study provide further educational tools for robotic myomectomy training.

IDEAL-D Phase 0 Evaluation of the Avatera System in Robot-Assisted Prostate, Bladder and Renal Surgery.

Purpose: To evaluate the utilization of novel Avatera system in urological operations according to the IDEAL-D framework recommendations for high-risk invasive surgical devices. Materials and Methods: Three surgeons attempted to perform 23 upper and lower urinary tract operations on human cadavers and in live porcine models using the Avatera system. Total operative time and the duration of the substeps were evaluated. Surgical performance was assessed with the Global Evaluative Assessment of Robotic Skills (GEARS) score. Suturing was rated using the technical checklist for the assessment of suturing in robotic surgery. Attending surgeons rated their satisfaction with the Avatera system on a scale of 1-5. Results and Limitation: Seventeen out of 18 operations performed on cadavers were completed, while one pyeloplasty was discontinued. All five operations performed in porcine models were completed. Although 1 pig was euthanized on the fifth postoperative day, its symptoms were unrelated to surgery. Mean GEARS and Suturing scores in the upper urinary tract were 29 ± 0.7 and 29.5 ± 0.95, respectively, and in the lower urinary 28.5 ± 1.2 and 29.5 ± 0.5, respectively. Surgeons' satisfaction was high or very high for all procedures. Conclusions: The Avatera system was associated with good surgical performance and high surgeons' satisfaction rates. All urological procedures performed were shown to be feasible, with comparable risks to other robot-assisted surgery systems.

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills.

Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment. Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed. OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001). OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.

Simulation-based assessment of robotic cardiac surgery skills: An international multicenter, cross-specialty trial

ObjectiveThis study aimed to investigate the validity of simulation-based assessment of robotic-assisted cardiac surgery skills using a wet lab model, focusing on the use of a time-based score (TBS) and modified Global Evaluative Assessment of Robotic Skills (mGEARS) score. MethodsWe tested 3 wet lab tasks (atrial closure, mitral annular stitches, and internal thoracic artery [ITA] dissection) with both experienced robotic cardiac surgeons and novices from multiple European centers. The tasks were assessed using 2 tools: TBS and mGEARS score. Reliability, internal consistency, and the ability to discriminate between different levels of competence were evaluated. ResultsThe results demonstrated a high internal consistency for all 3 tasks using mGEARS assessment tool. The mGEARS score and TBS could reliably discriminate between different levels of competence for the atrial closure and mitral stitches tasks but not for the ITA harvesting task. A generalizability study also revealed that it was feasible to assess competency of the atrial closure and mitral stitches tasks using mGEARS but not the ITA dissection task. Pass/fail scores were established for each task using both TBS and mGEARS assessment tools. ConclusionsThe study provides sufficient evidence for using TBS and mGEARS scores in evaluating robotic-assisted cardiac surgery skills in wet lab settings for intracardiac tasks. Combining both assessment tools enhances the evaluation of proficiency in robotic cardiac surgery, paving the way for standardized, evidence-based preclinical training and credentialing. Clinical trial registry numberNCT05043064.

Surgical skill level classification model development using EEG and eye-gaze data and machine learning algorithms

The aim of this study was to develop machine learning classification models using electroencephalogram (EEG) and eye-gaze features to predict the level of surgical expertise in robot-assisted surgery (RAS). EEG and eye-gaze data were recorded from 11 participants who performed cystectomy, hysterectomy, and nephrectomy using the da Vinci robot. Skill level was evaluated by an expert RAS surgeon using the modified Global Evaluative Assessment of Robotic Skills (GEARS) tool, and data from three subtasks were extracted to classify skill levels using three classification models—multinomial logistic regression (MLR), random forest (RF), and gradient boosting (GB). The GB algorithm was used with a combination of EEG and eye-gaze data to classify skill levels, and differences between the models were tested using two-sample t tests. The GB model using EEG features showed the best performance for blunt dissection (83% accuracy), retraction (85% accuracy), and burn dissection (81% accuracy). The combination of EEG and eye-gaze features using the GB algorithm improved the accuracy of skill level classification to 88% for blunt dissection, 93% for retraction, and 86% for burn dissection. The implementation of objective skill classification models in clinical settings may enhance the RAS surgical training process by providing objective feedback about performance to surgeons and their teachers.

Comparison of Global Evaluative Assessment of Robotic Surgery with objective performance indicators for the assessment of skill during robotic-assisted thoracic surgery

BackgroundThe Global Evaluative Assessment of Robotic Skills is a popular but ultimately subjective assessment tool in robotic-assisted surgery. An alternative approach is to record system or console events or calculate instrument kinematics to derive objective performance indicators. The aim of this study was to compare these 2 approaches and correlate the Global Evaluative Assessment of Robotic Skills with different types of objective performance indicators during robotic-assisted lobectomy. MethodsVideo, system event, and kinematic data were recorded from the robotic surgical system during left upper lobectomy on a standardized perfused and pulsatile ex vivo porcine heart–lung model. Videos were segmented into steps, and the superior vein dissection was graded independently by 2 blinded expert surgeons with Global Evaluative Assessment of Robotic Skills. Objective performance indicators representing categories for energy use, event data, movement, smoothness, time, and wrist articulation were calculated for the same task and compared to Global Evaluative Assessment of Robotic Skills scores. ResultsVideo and data from 51 cases were analyzed (44 fellows, 7 attendings). Global Evaluative Assessment of Robotic Skills scores were significantly higher for attendings (P < .05), but there was a significant difference in raters’ scores of 31.4% (defined as >20% difference in total score). The interclass correlation was 0.44 for 1 rater and 0.61 for 2 raters. Objective performance indicators correlated with Global Evaluative Assessment of Robotic Skills to varying degrees. The most highly correlated Global Evaluative Assessment of Robotic Skills domain was efficiency. Instrument movement and smoothness were highly correlated among objective performance indicator categories. Of individual objective performance indicators, right-hand median jerk, an objective performance indicator of change of acceleration, had the highest correlation coefficient (0.55). ConclusionThere was a relatively poor overall correlation between the Global Evaluative Assessment of Robotic Skills and objective performance indicators. However, both appear strongly correlated for certain metrics such as efficiency and smoothness. Objective performance indicators may be a potentially more quantitative and granular approach to assessing skill, given that they can be calculated mathematically and automatically without subjective interpretation.

Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial.

To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity. GEARS-Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations. In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves. The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments. Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.

THTP3.4 Performances measures in Robotic-Assisted Surgery- A Systematic Review

Abstract Background Robotic-Assisted Surgery (RAS) has seen a global rise in the past 30 years. Despite the increased adoption, there is no current standardised training curricula or measure of performance. The aim of this systematic review was to define the RAS metrics used to assess technical performance across the surgical specialities. Methods Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published 31st January 2020 – 31st January 2022. All randomised controlled trials (RCTs) and observational studies were considered. Review articles and systematic reviews were excluded. The papers’ quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. Results The initial search yielded 1189 papers of which 74 were eligible. The majority were published in urology (45% n=33). 27 unique metrics were identified and categorised as “task based”, “procedural based”, “cognitive assessments” and “global assessments”. Global assessments were the most common category of assessment (n=12); the most frequently used was GEARs (Global Evaluative Assessment of Robotic Skills). Only 4 metrics (automated, Proficiency Based Progression (PBP), pupillary measures and time to completion) used quantitative measures to assess performance whilst the remaining relied upon Likert scales thereby creating variability amongst users. Conclusion There is wide variation in tools used to assess performance in RAS. The majority of tools are subjective which increases the risk of bias amongst users and therefore accuracy in the measure of performance. A validated, objective global assessment tool that utilises quantitative measures is required.

Validation of a multidisciplinary virtual reality (VR) robotic surgical curriculum.

The objective is to identify whether trainees demonstrate improvement in a standardized knot-tying task as assessed by Global Evaluative Assessment of Robotic Skills (GEARS) score after completion of a virtual reality (VR) robotic curriculum. An IRB-exempt prospective study was conducted with surgical trainees from August 2021 to February 2023. Participants initially performed a baseline robotic suturing task in which they were instructed to tie interrupted square knots in 10min. Participants then completed a virtual reality simulation curriculum involving 23 exercises until they achieved 90% proficiency on all tasks. Participants then repeated the suturing task. Pre- and post-curriculum suturing tasks were recorded, de-identified, and scored by expert graders using a GEARS score. Trainees from three academic centers were invited to participate. Medical students (MS1-MS3) and surgical residents from gynecology, urology, and general surgery were invited to participate. Twenty-five trainees completed the pre-curriculum suturing task, the VR curriculum, and the post-curriculum suturing task. Trainees demonstrated significant improvement in their post-test GEARS score by 2.43 points (p < 0.05) and were able to tie three additional knots within 10min after completion of the curriculum (p < 0.05). Trainees also demonstrated a faster time to complete first knot (114s improvement, p < 0.05) after completion of the curriculum. All participants agreed or strongly agreed that completion of the robotic curriculum helped them feel more comfortable using the robotic console, and improved their robotic surgical skills. Surgical trainees and medical students with limited prior robotic surgical experience demonstrated objective improvement after completion of a standardized VR curriculum.

Developing surgical skill level classification model using visual metrics and a gradient boosting algorithm.

Assessment of surgical skills is crucial for improving training standards and ensuring the quality of primary care. This study aimed to develop a gradient boosting classification model (GBM) to classify surgical expertise into inexperienced, competent, and experienced levels in robot-assisted surgery (RAS) using visual metrics. Eye gaze data were recorded from 11 participants performing four subtasks; blunt dissection, retraction, cold dissection, and hot dissection using live pigs and the da Vinci robot. Eye gaze data were used to extract the visual metrics. One expert RAS surgeon evaluated each participant's performance and expertise level using the modified Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The extracted visual metrics were used to classify surgical skill levels and to evaluate individual GEARS metrics. Analysis of Variance (ANOVA) was used to test the differences for each feature across skill levels. Classification accuracies for blunt dissection, retraction, cold dissection, and burn dissection were 95%, 96%, 96%, and 96%, respectively. The time to complete only the retraction was significantly different among the 3 skill levels (p-value = 0.04). Performance was significantly different for 3 categories of surgical skill level for all subtasks (p-values<0.01). The extracted visual metrics were strongly associated with GEARS metrics (R2>0.7 for GEARS metrics evaluation models). Machine learning (ML) algorithms trained by visual metrics of RAS surgeons can classify surgical skill levels and evaluate GEARS measures. The time to complete a surgical subtask may not be considered a stand-alone factor for skill level assessment.

Intentional enterotomies: validation of a novel robotic surgery training exercise.

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.

Telesurgery and telesurgical support using a double-surgeon cockpit system allowing manipulation from two locations.

Although several studies on telesurgery have been reported globally, a clinically applicable technique has not yet been developed. As part of a telesurgical study series conducted by the Japan Surgical Society, this study describes the first application of a double-surgeon cockpit system to telesurgery. Surgeon cockpits were installed at a local site and a remote site 140km away. Three healthy pigs weighing between 26 and 29kg were selected for surgery. Non-specialized surgeons performed emergency hemostasis, cholecystectomy, and renal vein ligation with remote assistance using the double-surgeon cockpits and specialized surgeons performed actual telesurgery. Additionally, the impact of adding internet protocol security (IPsec) encryption to the internet protocol-virtual private network (IP-VPN) line on communication was evaluated to address clinical security concerns. The average time required for remote emergency hemostasis with the double-surgeon cockpit system was 10.64s. A non-specialized surgeon could safely perform cholecystectomy or renal vein ligation with remote assistance. Global Evaluative Assessment of Robotic Skills and System Usability Scale scores were higher for telesurgical support-assisted surgery by a non-specialized surgeon using the double-surgeon cockpits than for telesurgery performed by a specialized surgeon without the double-cockpit system. Adding IPsec encryption to the IP-VPN did not have a significant impact on communication. Telesurgical support through our double-surgeon cockpit system is feasible as first step toward clinical telesurgery.

MP26-09 EYE MOVEMENT BEHAVIOR ASSOCIATES WITH EXPERTISE LEVEL IN ROBOT-ASSISTED SURGERY

You have accessJournal of UrologyCME1 Apr 2023MP26-09 EYE MOVEMENT BEHAVIOR ASSOCIATES WITH EXPERTISE LEVEL IN ROBOT-ASSISTED SURGERY Mehdi Seilanian Toussi, Saeed Shadpour, Kristopher Attwood, Qian Liu, Camille Gutierrez, james L. Mohler, and Somayeh B. Shafiei Mehdi Seilanian ToussiMehdi Seilanian Toussi More articles by this author , Saeed ShadpourSaeed Shadpour More articles by this author , Kristopher AttwoodKristopher Attwood More articles by this author , Qian LiuQian Liu More articles by this author , Camille GutierrezCamille Gutierrez More articles by this author , james L. Mohlerjames L. Mohler More articles by this author , and Somayeh B. ShafieiSomayeh B. Shafiei More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003254.09AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: An objective method for assessing robot-assisted surgery (RAS) skills that can be used in clinical settings has not been established. This study aimed to compare visual metrics at three different skill levels—inexperienced, competent, or experienced— to determine whether eye gaze data differed among RAS surgeons. METHODS: Eleven right-handed participants who performed coagulation subtasks (monopolar or bipolar cautery) during hysterectomy, cystectomy, or nephrectomy on live pigs with the Da Vinci robot had their eye gaze data captured using TobiiPro2 eyeglasses. Visual metrics were extracted using eye gaze data (Table 1). An expert RAS surgeon evaluated each participant's performance and level of expertise after watching operation videos using the modified Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. Each metric was compared across the three skill levels—novice, competent, or experienced—using ANOVA. RESULTS: When participants performed coagulation using the dominant hand, the entropy of the right eye’s pupil diameter differed among skill levels (p-value = 0.01). When this subtask was completed with the non-dominant hand, the rates of change of the gaze directions of the left eye in the horizontal and vertical dimensions and the right eye in the vertical dimension differed (p-values = 2x10-6, 6x10-6, and 6x10-4, respectively) across skill levels. Table 2 shows the significance level for the difference in visual metrics between skill level pairs. CONCLUSIONS: Visual metrics differed significantly across RAS skill levels. The ease and efficacy of recording and evaluating visual data without interfering with the surgeon's work suggest a new objective surgical skill evaluation method that could be applied to clinical settings. Source of Funding: National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number R01EB029398, the Alliance Foundation of Roswell Park Comprehensive Cancer Center, and the National Cancer Institute (NCI) grant P30CA016056 involving the use of Roswell Park Comprehensive Cancer Center’s Laboratory Animal Shared Resource and Applied Technology Laboratory for Advanced Surgery © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e355 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Mehdi Seilanian Toussi More articles by this author Saeed Shadpour More articles by this author Kristopher Attwood More articles by this author Qian Liu More articles by this author Camille Gutierrez More articles by this author james L. Mohler More articles by this author Somayeh B. Shafiei More articles by this author Expand All Advertisement PDF downloadLoading ...

The Surgical Learning Curve: Does Robotic Technical Skill Explain Differences in Operative Performance?

Background: Prior studies on technical skills use small collections of videos for assessment. However, there is likely heterogeneity of performance among surgeons and likely improvement after training. If technical skill explains these differences, then it should vary among practicing surgeons and improve over time. Materials and Methods: Sleeve gastrectomy cases (n = 162) between July 2018 and January 2021 at one health system were included. Global evaluative assessment of robotic skills (GEARS) scores were assigned by crowdsourced evaluators. Videos were manually annotated. Analysis of variance was used to compare continuous variables between surgeons. Tamhane's post hoc test was used to define differences between surgeons with the eta-squared value for effect size. Linear regression was used for temporal changes. A P value <.05 was considered significant. Results: Variations in operative time discriminated between individuals (e.g., between 2 surgeons, means were 91 and 112 minutes, Tamhane's = 0.001). Overall, GEARS scores did not vary significantly (e.g., between those 2 surgeons, means were 20.32 and 20.6, Tamhane's = 0.151). Operative time and total GEARS score did not change over time (R2 = 0.0001-0.096). Subcomponent scores showed idiosyncratic temporal changes, although force sensitivity increased among all (R2 = 0.172-0.243). For a novice surgeon, phase-adjusted operative time (R2 = 0.24), but not overall GEARS scores (R2 = 0.04), improved over time. Conclusions: GEARS scores showed less variability and did not improve with time for a novice surgeon. Improved technical skill does not explain the learning curve of a novice surgeon or variation among surgeons. More work could define valid surrogate metrics for performance analysis.

Value of the Crowd-Sourced Assessment of Technical Skills (C-SATS) Platform in Surgical Procedures: A Systematic Review of Evidence

Background: Crowd-Sourced Assessment of Technical Skills (C-SATS) is a surgical data management and learning platform that leverages the knowledge of large expert surgeon and lay groups to assess the technique and technical skills of surgeons in a highly efficient manner. The aim of this systematic review was to summarize published literature on the performance of C-SATS as compared to expert evaluations and assess its use as a training and validation tool in minimally invasive surgery (MIS). Methods: A comprehensive literature search was performed per PRISMA guidelines using the Medline, Embase, and Google Scholar databases on published studies that evaluated the use of C-SATS following MIS, such as laparoscopic or robotic-assisted surgery. Results: A total of 21 reports were included in the review. Twelve studies comparing crowd-sourcing evaluations against expert opinion indicated overall excellent or good correlation with Global Operative Assessment of Laparoscopic Skills (GOALS), Global Evaluative Assessment of Robotic Skills (GEARS), and Robotic Objective Structured Assessment of Technical Skills (R-OSATS) scores, with correlation coefficients (Pearson or Spearman) ranging from 0.69 to 0.95 and reliability index (Cronbach’s alpha) from 0.63 to 0.93 across different specialties and surgical approaches. When using C-SATS to assess performance and validation, assessments positively correlated with traditional methods of time and error-based scoring and global rating scale. Conclusions: Based on the current published literature, the C-SATS platform has been shown to efficiently provide crowd-sourced evaluations that correlate favorably with expert evaluation across a range of surgical specialties and approaches. Use of crowdsourcing has uniformly yielded accurate evaluations of surgeons’ technical skills in a markedly shorter time than expert reviews. C-SATS may be a cost-effective complement or alternative to traditional models of evaluating surgical proficiency. Future studies are needed to determine whether the use of C-SATS will lead to improved surgical performance and patient outcomes.