Automated Performance Metrics Research Articles

Preliminary validation of the virtual bariatric endoscopic simulator

Background and AimsObesity is a global health concern. Bariatric surgery offers reliably effective and durable weight loss and improvements of other comorbid conditions. However, the accessibility of bariatric surgery remains limited. Minimally invasive techniques, including endoscopic sleeve gastroplasty (ESG), have emerged to bridge this gap. To effectively complete the ESG procedure, one requires skill in multiple complex interventional endoscopic maneuvers. This requisite expertise poses challenges for training in this burgeoning field. MethodsWe designed the Virtual Bariatric Endoscopic (ViBE) simulator software to mimic the ESG procedure accurately. It features a detailed simulation of an endoscope equipped with an endoscopic suturing system and a high-resolution stomach, enhancing the visualization of the procedural details. Furthermore, it incorporates performance metrics utilizing a reverse scoring system to evaluate users’ proficiency in tasks such as argon plasma coagulation (APC) marking, suturing, and cinching. To validate the simulator, we conducted a study involving experts and novices at the Indiana University School of Medicine, where participants engaged with the simulation environment in a series of training tasks. ResultsTwelve participants, comprising five experts and seven novices, participated in the study. They were asked to complete a post-questionnaire featuring seven items, rating each on a Likert scale. The APC task realism received the highest score, averaging 3.83. The usefulness of improving endoscopic technical skills averaged 3.08, with the realism of cinching the knot and suturing tasks receiving scores of 3.17 and 3.25, respectively, suggesting a generally positive reception. Automated performance metrics indicated that, on average, experts outperformed novices by 10.83 points. ConclusionThe ViBE simulation strives to replicate the steps of the ESG within a virtual environment. Our primary objective in developing this simulator is to enhance the learning curve for endoscopic suturing and ESG techniques, thereby safely extending these skills to a broader patient base.

Automated performance metrics and surgical gestures: two methods for assessment of technical skills in robotic surgery

The objective of this study is to compare automated performance metrics (APM) and surgical gestures for technical skills assessment during simulated robot-assisted radical prostatectomy (RARP). Ten novices and six experienced RARP surgeons performed simulated RARPs on the RobotiX Mentor (Surgical Science, Sweden). Simulator APM were automatically recorded, and surgical videos were manually annotated with five types of surgical gestures. The consequences of the pass/fail levels, which were based on contrasting groups’ methods, were compared for APM and surgical gestures. Intra-class correlation coefficient (ICC) analysis and a Bland–Altman plot were used to explore the correlation between APM and surgical gestures. Pass/fail levels for both APM and surgical gesture could fully distinguish between the skill levels of the surgeons with a specificity and sensitivity of 100%. The overall ICC (one-way, random) was 0.70 (95% CI: 0.34–0.88), showing moderate agreement between the methods. The Bland–Altman plot showed a high agreement between the two methods for assessing experienced surgeons but disagreed on the novice surgeons’ skill level. APM and surgical gestures could both fully distinguish between novices and experienced surgeons in a simulated setting. Both methods of analyzing technical skills have their advantages and disadvantages and, as of now, those are only to a limited extent available in the clinical setting. The development of assessment methods in a simulated setting enables testing before implementing it in a clinical setting.

Vision-Based Force Estimation for Minimally Invasive Telesurgery Through Contact Detection and Local Stiffness Models

In minimally invasive telesurgery, obtaining accurate force information is difficult due to the complexities of in-vivo end effector force sensing. This constrains development and implementation of haptic feedback and force-based automated performance metrics, respectively. Vision-based force sensing approaches using deep learning are a promising alternative to intrinsic end effector force sensing. However, they have limited ability to generalize to novel scenarios, and require learning on high-quality force sensor training data that can be difficult to obtain. To address these challenges, this paper presents a novel vision-based contact-conditional approach for force estimation in telesurgical environments. Our method leverages supervised learning with human labels and end effector position data to train deep neural networks. Predictions from these trained models are optionally combined with robot joint torque information to estimate forces indirectly from visual data. We benchmark our method against ground truth force sensor data and demonstrate generality by fine-tuning to novel surgical scenarios in a data-efficient manner. Our methods demonstrated greater than 90% accuracy on contact detection and less than 10% force prediction error. These results suggest potential usefulness of contact-conditional force estimation for sensory substitution haptic feedback and tissue handling skill evaluation in clinical settings.

Artificial intelligence and robotic surgical education

There are numerous barriers in robotic surgical training, including reliance on observational learning, low-quality feedback, and inconsistent assessment. Artificial intelligence (AI) offers potential solutions to these central problems in robotic surgical education and may allow for more efficient and efficacious training. Three key areas in which AI has particular relevance to robotic surgical education are video labeling, feedback, and assessment. Video labeling refers to the automated designation of prespecified categories to operative videos. Numerous prior studies have applied AI for video labeling, particularly for retrospective educational review after an operation. Video labeling allows learners and their instructors to rapidly identify critical parts of an operative video. We recommend incorporating AI-based video labeling into robotic surgical education where available. AI also offers a mechanism by which reliable feedback can be provided in robotic surgery. Feedback through AI harnesses automated performance metrics (APMs) and natural language processing (NLP) to provide actionable and descriptive plans to learners while reducing faculty assessment burden. We recommend combining supervised AI-generated, APM-based feedback with expert-based feedback to allow surgeons and trainees to reflect on metrics like bimanual dexterity and efficiency. Finally, summative assessment by AI could allow for automated appraisal of surgeons or surgical trainees. However, AI-based assessment remains limited by concerns around bias and opaque processes. Several studies have applied computer vision to compare AI-based assessment with expert-completed rating scales, though such work remains investigational. At this time, we recommend against the use of AI for summative assessment pending additional validity evidence. Overall, AI offers solutions and promising future directions by which to address multiple educational challenges in robotic surgery. Through advances in video labeling, feedback, and assessment, AI has demonstrated ways by which to increase the efficiency and efficacy of robotic surgical education.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review.

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity. A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis. A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes. Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Automated performance metrics, learning curve and robotic colorectal surgery.

The aim of this study was to evaluate the usefulness of Automated Performance Metrics (APMs) in assessing the learning curve. A retrospective review of 85 consecutive patients who underwent total robotic colorectal surgery at a single institution between August 2020 and October 2022 was performed. Patient demographics, operation type, and APMs were collected and analysed. Cumulative summation technique (CUSUM) was used to construct learning curves of surgeon console time (SCT), use of the fourth arm, clutch activation, instrument off screen (number and duration), and cut electrocautery activation. Two phases with 50 and 35 cases were identified from the CUSUM graph for SCT. The SCT was significantly different between the two phases (176 and 251min, p<0.002). After adjustment for SCT, the APMs were not significantly different between the two phases. Most APMs do not offer additional learning curve information when compared with SCT analysis alone.

Robotic Surgery: A Comprehensive Review of the Literature and Current Trends.

Robotic surgery (RS) is an evolution of minimally invasive surgery that combines medical science, robotics, and engineering. The first robots approved by the Food and Drug Administration (FDA) were the Da Vinci Surgical System and the ZEUS Robotic Surgical System, which have been improving over time. Through the decades, the equipment applied to RS had undergone a wide transformation as a response to the development of new techniques and facilities for the assembly and implementation of the own. RS has revolutionized the field of urology, enabling surgeons to perform complex procedures with greater precision and accuracy, and many other surgical specialties such as gynecology, general surgery, otolaryngology, cardiothoracic surgery, and neurosurgery.Several benefits, such as a better approach to the surgical site, a three-dimensional image that improves depth perception, and smaller scars, enhance range of motion, allowing the surgeon to conduct more complicated surgical operations, and reduced postoperative complications have made robotic-assisted surgery an increasingly popular approach. However, some points like the cost of surgical procedures, equipment-instrument, and maintenance are important aspects to consider. Machine learning will likely have a role to play in surgical training shortly through "automated performance metrics," where algorithms observe and "learn" individual surgeons' techniques, assess performance, and anticipate surgical outcomes with the potential to individualize surgical training and aid decision-making in real time.

Development of an Automated Composite Ureteroscopic Efficiency Score Through Simulated Ureteroscopic Skills Assessment.

Introduction: Flexible ureteroscopy (fURS) is the most common procedure for treatment of urolithiasis. We previously utilized kinematic evaluations of simulated fURS to demonstrate that certain body movements are associated with efficient ureteroscopic manipulation for complex tasks. In this study, we incorporated computer vision to create an efficiency score using the ureteroscope travel distance (DIST), task time (TIME), spectral arc length (SPARC), and percentage of purposeful wall collisions (COLL). The goal is a simulation-based system that can abstract these automated performance metrics (APMs) to differentiate between novice and expert ureteroscope handling. Methods: A ureteroscopic simulation box was used. Body kinematics, task time, and ureteroscopic movements were analyzed using a motion capture system and video camera. Optical flow computer vision was used to track the ureteroscope. DIST, TIME, and SPARC were automatically calculated. Wall collisions were automatically captured and independently judged by two authors; an algorithm was developed to automatically determine the COLL variable. A mixed-effects model was used to aggregate these variables and distinguish between surgeons' first and final task attempts. Normalized values of these metrics were added to create a composite ureteroscopic efficiency score (CUES). Results: Twelve urologists completed the simulated tasks. The COLL assessment algorithm determined beneficial wall collisions with an accuracy of 77%. Normalized values of TIME, DIST, SPARC, and COLL were combined to create a composite ureteroscopic efficiency score (CUES). Compared with the first attempt, both the second and third attempts showed statistically significant improvements in CUES. The ROC-AUC score reached 0.86, suggesting excellent discrimination between attempts. There was also a statistically significant difference in CUES when comparing resident and attending performance. Conclusions: APMs can be abstracted using computer vision and artificial intelligence; an aggregate composite score (CUES) may be a promising method for evaluation of ureteroscopic efficiency.

Use of surgical video-based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study.

Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery. Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined-instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions-on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials. In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001). Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

Machine Learning to Delineate Surgeon and Clinical Factors That Anticipate Positive Surgical Margins After Robot-Assisted Radical Prostatectomy.

Purpose: Automated performance metrics (APMs), derived from instrument kinematic and systems events data during robotic surgery, are validated objective measures of surgeon performance. Our previous studies showed that APMs are strong outcome predictors of urinary continence after robot-assisted radical prostatectomy (RARP). We now use machine learning to investigate how surgeon performance (i.e., APMs) and clinical factors can predict positive surgical margins (PSMs) after RARP. Methods: We prospectively collected data of patients undergoing RARP at our institution from 2016 to 2019. Random Forest model predicted PSMs based on 15 clinical factors and 38 APMs from 11 standardized RARP steps. Out-of-bag Gini impurity index determined the top 10 variables of importance (VOI). APMs in the top 10 VOI were assessed for confounding effects by extracapsular extension (ECE) and pathologic T (pT) through Poisson regression with Generalized Estimating Equation. Results: 55/236 (23.3%) cases had PSMs. Of the 55 cases with PSMs, 9 (16.4%) were pT2 and 46 (83.6%), pT3. The full model, including clinical factors and APMs, achieved area under the curve (AUC) 0.74. When assessing clinical factors or APMs alone, the model achieved AUC 0.72 and 0.64, respectively. The strongest PSM predictors were ECE and pT stage, followed by APMs in specific steps. After adjusting for ECE and pT stage, most APMs remained as independent predictors of PSM. Conclusion: Using machine learning methods, we found that the strongest predictors of PSMs after RARP are nonmodifiable, disease-driven factors (ECE and pT). While APMs provide minimal additional insight into when PSMs may occur, they are nonetheless capable of independently predicting PSMs based on objective measures of surgeon performance.

Tailored Feedback Based on Clinically Relevant Performance Metrics Expedites the Acquisition of Robotic Suturing Skills—An Unblinded Pilot Randomized Controlled Trial

Previously, we identified 8 objective suturing performance metrics highly predictive of urinary continence recovery after robotic-assisted radical prostatectomy. Here, we aimed to test the feasibility of providing tailored feedback based upon these clinically relevant metrics and explore the impact on the acquisition of robotic suturing skills. Training surgeons were recruited and randomized to a feedback group or a control group. Both groups completed a baseline, midterm and final dry laboratory vesicourethral anastomosis (VUA) and underwent 4 intervening training sessions each, consisting of 3 suturing exercises. Eight performance metrics were recorded during each exercise: 4 automated performance metrics (derived from kinematic and system events data of the da Vinci® Robotic System) representing efficiency and console manipulation competency, and 4 suturing technical skill scores. The feedback group received tailored feedback (a visual diagram+verbal instructions+video examples) based on these metrics after each session. Generalized linear mixed model was used to compare metric improvement (Δ) from baseline to the midterm and final VUA. Twenty-three participants were randomized to the feedback group (11) or the control group (12). Demographic data and baseline VUA metrics were comparable between groups. The feedback group showed greater improvement than the control group in aggregate suturing scores at midterm (mean Δ feedback group 4.5 vs Δ control group 1.1) and final VUA (Δ feedback group 5.3 vs Δ control group 4.9). The feedback group also showed greater improvement in the majority of the included metrics at midterm and final VUA. Tailored feedback based on specific, clinically relevant performance metrics is feasible and may expedite the acquisition of robotic suturing skills.

Developing basic robotic skills using virtual reality simulation and automated assessment tools: a multidisciplinary robotic virtual reality-based curriculum using the Da Vinci Skills Simulator and tracking progress with the Intuitive Learning platform.

This study sought to develop basic robotic surgical skills among surgical trainees across multiple specialties using a VR-based curriculum and provided objective, on-demand, automated assessments using the Intuitive Learning platform. This curriculum was developed using the Da Vinci Skills Simulator and included 24 exercises. A pre-test and post-test were required for completion of the curriculum. Scores > 90 on individual exercises and the post-test were required for successful completion. The Intuitive learning platform provided automated performance metrics and tracked trainee progression. The curriculum was implemented and data collected over a 12-month period. 21 trainees completed the entire curriculum. Post-test scores were significantly higher than pre-test scores and trainees reported improvement in their robotic skills after curriculum completion. A comparison based on training level revealed that junior residents had significantly lower number of attempts per exercise, fewer penalties, and higher completion scores when compared to senior residents and fellows. Individual exercise analysis demonstrated that exercises, such as 'Three-Arm Relay' and 'Ring Rollercoaster', required the longest time and most attempts to achieve a passing score. The 'Energy Pedals' and 'Knot Tying' skills were the least-utilized skills addressed in the curriculum. Virtual reality-based curriculums using the Intuitive Learning platform can be standardized across multiple specialties allowing for the development of basic robotic skills, shared interdisciplinary surgical education, and provides powerful objective and automated performance metrics of trainees.

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy

Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon's technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)-objective measures related to instrument movements-can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes. A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query "robot-assisted surgery OR robotic surgery AND performance metrics" between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed. The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies. APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required.

The Experienced Surgeon and New Tricks—It’s Time for Full Adoption and Support of Automated Performance Metrics and Databases

The Experienced Surgeon and New Tricks—It’s Time for Full Adoption and Support of Automated Performance Metrics and Databases

PD57-07 AUTOMATED PERFORMANCE METRICS THAT ANTICIPATE EARLY AND LATE URINARY CONTINENCE RECOVERY AFTER ROBOT-ASSISTED RADICAL PROSTATECTOMY

You have accessJournal of UrologyProstate Cancer: Localized: Surgical Therapy IV (PD57)1 Sep 2021PD57-07 AUTOMATED PERFORMANCE METRICS THAT ANTICIPATE EARLY AND LATE URINARY CONTINENCE RECOVERY AFTER ROBOT-ASSISTED RADICAL PROSTATECTOMY Runzhuo Ma, Ryan S. Lee, Sharath S. Reddy, Jessica H. Nguyen, Steven Cen, Katarina Urbanova, Joern H. Witt, Inderbir S. Gill, Christian Wagner, and Andrew J. Hung Runzhuo MaRunzhuo Ma More articles by this author , Ryan S. LeeRyan S. Lee More articles by this author , Sharath S. ReddySharath S. Reddy More articles by this author , Jessica H. NguyenJessica H. Nguyen More articles by this author , Steven CenSteven Cen More articles by this author , Katarina UrbanovaKatarina Urbanova More articles by this author , Joern H. WittJoern H. Witt More articles by this author , Inderbir S. GillInderbir S. Gill More articles by this author , Christian WagnerChristian Wagner More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000002091.07AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Automated performance metrics (APMs), objective metrics derived from kinematic/system events data and captured by a systems data recorder (Intuitive Surgical) during surgery, are a validated surgical assessment tool. We previously showed APMs from certain steps of robot-assisted radical prostatectomy (RARP) to be predictors of early urinary continence recovery. Having expanded our cohort to include two high-volume centers with longer follow-up, we now explore how APMs may be associated with both early and late urinary continence recovery after RARP. METHODS: Non-teaching RARP cases from 2016-2019 from two centers were included in this study. Non-teaching cases were defined as cases in which an expert surgeon (>100 prior robotic surgery cases) performed all seven major steps, including anterior/posterior bladder neck dissection (BND), left/right neurovascular bundle (NVB) preservation, apical dissection, and posterior/anterior vesicourethral anastomosis. The outcome of interest was 6-month (early) and 12-month (late) continence recovery status. Continence was defined as using 0 or 1 safety pad per day. Decision tree (CART©) was used to predict outcomes. APMs from the seven major steps (38 x 7), 9 patient factors, and 5 treatment factors were utilized in the model. Gini impurity was used as splitting criteria and a 10-fold cross-validation was conducted. Out-of-bag Gini impurity index was used for feature ranking. RESULTS: 110 non-teaching RARP cases from 22 expert surgeons were included in this study. At 6 and 12 months, 59.1% (65/110) and 74.5% (82/110) of cases recovered continence, respectively. In the training phase of the model, the 6- and 12-month continence recovery prediction achieved 71.8% and 65.5% accuracy, respectively; in the testing phase, the accuracies slightly decreased to 67.3% and 61.8%. At both time points, the top 10 features came from APMs; none were patient factors. For 6-month continence status prediction, all top 10 features came from anterior BND. For 12-month continence status prediction, 6/10 came from anterior BND, 3/10 came from left NVB, and 1/10 came from posterior BND. CONCLUSIONS: Surgical performance, as measured by APMs, was the strongest predicter of both early and late continence recovery after RARP. Bladder neck dissection was the most important surgical performance step for early continence recovery in this non-teaching case cohort, while late continence recovery was affected by performance in more varied steps. Source of Funding: This study was funded in part by an Intuitive Surgical Clinical Grant; Intuitive Surgical provided the systems data recorder. Research reported in this publication was also supported in part by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K23EB026493. © 2021 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 206Issue Supplement 3September 2021Page: e1012-e1012 Advertisement Copyright & Permissions© 2021 by American Urological Association Education and Research, Inc.MetricsAuthor Information Runzhuo Ma More articles by this author Ryan S. Lee More articles by this author Sharath S. Reddy More articles by this author Jessica H. Nguyen More articles by this author Steven Cen More articles by this author Katarina Urbanova More articles by this author Joern H. Witt More articles by this author Inderbir S. Gill More articles by this author Christian Wagner More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement Loading ...

MP15-09 MACHINE LEARNING AND AUTOMATED PERFORMANCE METRICS TO PREDICT POSITIVE SURGICAL MARGINS AFTER ROBOT-ASSISTED RADICAL PROSTATECTOMY

INTRODUCTION AND OBJECTIVE:Automated performance metrics (APMs), derived from instrument kinematic/systems events data during surgery and captured by a systems data recorder (Intuitive Surgical), a...

A systematic review of virtual reality for the assessment of technical skills in neurosurgery.

Virtual reality (VR) and augmented reality (AR) systems are increasingly available to neurosurgeons. These systems may provide opportunities for technical rehearsal and assessments of surgeon performance. The assessment of neurosurgeon skill in VR and AR environments and the validity of VR and AR feedback has not been systematically reviewed. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was conducted through MEDLINE and PubMed. Studies published in English between January 1990 and February 2021 describing the use of VR or AR to quantify surgical technical performance of neurosurgeons without the use of human raters were included. The types and categories of automated performance metrics (APMs) from each of these studies were recorded. Thirty-three VR studies were included in the review; no AR studies met inclusion criteria. VR APMs were categorized as either distance to target, force, kinematics, time, blood loss, or volume of resection. Distance and time were the most well-studied APM domains, although all domains were effective at differentiating surgeon experience levels. Distance was successfully used to track improvements with practice. Examining volume of resection demonstrated that attending surgeons removed less simulated tumor but preserved more normal tissue than trainees. More recently, APMs have been used in machine learning algorithms to predict level of training with a high degree of accuracy. Key limitations to enhanced-reality systems include limited AR usage for automated surgical assessment and lack of external and longitudinal validation of VR systems. VR has been used to assess surgeon performance across a wide spectrum of domains. The VR environment can be used to quantify surgeon performance, assess surgeon proficiency, and track training progression. AR systems have not yet been used to provide metrics for surgeon performance assessment despite potential for intraoperative integration. VR-based APMs may be especially useful for metrics that are difficult to assess intraoperatively, including blood loss and extent of resection.

Virtual Reality vs Dry Laboratory Models: Comparing Automated Performance Metrics and Cognitive Workload During Robotic Simulation Training.

Background: This study compares surgical performance during analogous vesico-urethral anastomosis (VUA) tasks in two robotic training environments, virtual reality (VR) and dry laboratory (DL), to investigate transferability of skill assessment across the two platforms. Utilizing computer-generated performance metrics and pupillary data, we evaluated the two environments to distinguish surgical expertise and ultimately whether performance in the VR simulation correlates with performance in live robotic surgery in the DL. Materials and Methods: Experts (≥300 cases) and trainees (<300 cases) performed analogous VUAs during VR and DL sessions on a da Vinci robotic console following an Institutional Review Board (IRB) approved protocol (HS-16-00318). Twenty-two metrics were generated in each environment (kinematic metrics, tissue metrics, and biometrics). The DL included 18 previously validated automated performance metrics (APMs) (kinematics and event metrics) captured by an Intuitive system data recorder. In both settings, Tobii Pro Glasses 2 recorded the task-evoked pupillary response (reported as Index of Cognitive Activity [ICA]) to indicate cognitive workload, analyzed by EyeTracking cognitive workload software. Pearson correlation, Mann-Whitney, and independent t-tests were used for the comparative analyses. Results: Our study included six experts (median caseload 1300 [interquartile range 400-3000]) and 11 trainees (25 [0-250]). A total of 8/9 metrics directly comparable between VR and DL showed significant positive correlation (r ≥ 0.554, p ≤ 0.032); 5/22 VR metrics distinguished expertise, including task time (p = 0.031), clutch usage (p = 0.040), unnecessary needle piercing (p = 0.026), and suspected injury to the endopelvic fascia (p = 0.040). This contrasts with 14/22 APMs in DL (p ≤ 0.038), including linear velocities of all three instruments (p ≤ 0.038) and dominant-hand instrument wrist articulation (p = 0.013). Trainees experienced higher cognitive workload (ICA) in both environments when compared with experts (p < 0.036). Conclusions: Most performance metrics between VR and DL exhibited moderate to strong correlations, showing transferability of skills across the platforms. Comparing training environments, APMs during DL tasks are better able to distinguish expertise than VR-generated metrics.

Development and validation of an objective scoring tool to evaluate surgical dissection: Dissection Assessment for Robotic Technique (DART).

Evaluation of surgical competency has important implications for training new surgeons, accreditation, and improving patient outcomes. A method to specifically evaluate dissection performance does not yet exist. This project aimed to design a tool to assess surgical dissection quality. Delphi method was used to validate structure and content of the dissection evaluation. A multi-institutional and multi-disciplinary panel of 14 expert surgeons systematically evaluated each element of the dissection tool. Ten blinded reviewers evaluated 46 de-identified videos of pelvic lymph node and seminal vesicle dissections during the robot-assisted radical prostatectomy. Inter-rater variability was calculated using prevalence-adjusted and bias-adjusted kappa. The area under the curve from receiver operating characteristic curve was used to assess discrimination power for overall DART scores as well as domains in discriminating trainees (≤100 robotic cases) from experts (>100). Four rounds of Delphi method achieved language and content validity in 27/28 elements. Use of 3- or 5-point scale remained contested; thus, both scales were evaluated during validation. The 3-point scale showed improved kappa for each domain. Experts demonstrated significantly greater total scores on both scales (3-point, p< 0.001; 5-point, p< 0.001). The ability to distinguish experience was equivalent for total score on both scales (3-point AUC= 0.92, CI 0.82-1.00, 5-point AUC= 0.92, CI 0.83-1.00). We present the development and validation of Dissection Assessment for Robotic Technique (DART), an objective and reproducible 3-point surgical assessment to evaluate tissue dissection. DART can effectively differentiate levels of surgeon experience and can be used in multiple surgical steps.

Erratum: An Objective Assessment of Performance During Robotic Partial Nephrectomy: Validation and Correlation of Automated Performance Metrics With Intraoperative Outcomes.

Erratum: An Objective Assessment of Performance During Robotic Partial Nephrectomy: Validation and Correlation of Automated Performance Metrics With Intraoperative Outcomes.