Automated Performance Metrics Research Articles

MP01-05 HOW ACCURATE IS THE CROWD: DETERMINING THE DISCRIMINATORY CAPABILITY OF EXPERT AND CROWDSOURCED GEARS EVALUATIONS FOR THE ROBOTIC ASSISTED RADICAL PROSTATECTOMY

You have accessJournal of UrologySurgical Technology & Simulation: Training & Skills Assessment I1 Apr 2018MP01-05 HOW ACCURATE IS THE CROWD: DETERMINING THE DISCRIMINATORY CAPABILITY OF EXPERT AND CROWDSOURCED GEARS EVALUATIONS FOR THE ROBOTIC ASSISTED RADICAL PROSTATECTOMY Paul Oh, Jian Chen, Anthony Jarc, Micha Titus, and Andrew Hung Paul OhPaul Oh More articles by this author , Jian ChenJian Chen More articles by this author , Anthony JarcAnthony Jarc More articles by this author , Micha TitusMicha Titus More articles by this author , and Andrew HungAndrew Hung More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2018.02.111AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Evaluation of surgical video by experts to assess robotic performance is expensive and time-consuming compared to crowdsourcing from the general population. This study evaluates experts′ and the Crowd′s discriminatory ability in differentiating with Global Evaluative Assessment of Robotic Skills (GEARS) scores, one′s clinical outcomes and automated performance metrics. METHODS Automated performance metrics (instrument motion tracking and system events) of 25 anterior vesico-urethral anastomoses of robotic-assisted radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Video footage was graded with GEARS by 4 experts (averaged) and the Crowd (C-SATS) composed of 807 crowdworkers. GEARS scores (individual domains and total) were stratified into quartiles (Top 25% = Q1, Middle 50% = Q2, Bottom 25% = Q3), and each quartile′s outcomes and metrics were compared using ANOVA with post-hoc Tukey analysis to determine the discriminatory strength of GEARS. RESULTS Individual GEARS domains: The Crowd′s strongest discriminatory ability was in Efficiency (5 differences in automated performance metrics, p<0.04) and Bimanual Dexterity (BD; 4 metrics, p<0.05), while it was the least effective in Robotic Control (RC; 1 metric, p=0.04) and Depth Perception (DP; 0 metrics). Averaged Expert scores had similar discriminatory abilities in Efficiency and BD, but they also determined differences in RC (1 outcome, 2 metrics; p<0.03) and DP (1 outcome, 1 metric; p<0.04). Estimated blood loss (EBL) was the one outcome differentiated by Averaged Experts in Efficiency, RC, and DP (p<0.05). Total GEARS (Figure): The Crowd′s Total GEARS quartiles detected differences in 4 metrics (p<0.043) compared to 1 metric (p=0.013) by the Averaged Experts. The Crowd and Averaged Experts consistently detected differences in velocity of dominant arm in all GEARS domains. CONCLUSIONS The Crowd′s individual GEARS scores can determine differences in surgical maneuvers, but not to the extent of experts, who can also detect differences in EBL. However, the Crowd is more discriminatory than experts using Total GEARS. Considering the feasibility of assessing numerous surgeries, the Crowd maintains a strong role as an evaluator when using Total GEARS scores. © 2018FiguresReferencesRelatedDetails Volume 199Issue 4SApril 2018Page: e3 Advertisement Copyright & Permissions© 2018MetricsAuthor Information Paul Oh More articles by this author Jian Chen More articles by this author Anthony Jarc More articles by this author Micha Titus More articles by this author Andrew Hung More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

PD38-03 AUTOMATED PERFORMANCE METRICS DURING ROBOTIC-ASSISTED RADICAL PROSTATECTOMY CAN DIFFERENTIATE CLINICAL OUTCOME

You have accessJournal of UrologyProstate Cancer: Localized: Surgical Therapy V1 Apr 2018PD38-03 AUTOMATED PERFORMANCE METRICS DURING ROBOTIC-ASSISTED RADICAL PROSTATECTOMY CAN DIFFERENTIATE CLINICAL OUTCOME Andrew Hung, Paul Oh, Jian Chen, Nathan Cheng, Michael Lin-Brande, Micha Titus, Niero Rajarubendra, and Inderbir Gill Andrew HungAndrew Hung More articles by this author , Paul OhPaul Oh More articles by this author , Jian ChenJian Chen More articles by this author , Nathan ChengNathan Cheng More articles by this author , Michael Lin-BrandeMichael Lin-Brande More articles by this author , Micha TitusMicha Titus More articles by this author , Niero RajarubendraNiero Rajarubendra More articles by this author , and Inderbir GillInderbir Gill More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2018.02.1750AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES Surgical skill is associated with post-operative outcomes of patients. Our study utilized automated performance metrics (APMs) obtained by a novel recorder ″dVLogger″ during robotic-assisted prostatectomies (RRP) to assess contemporary skill and to see if such determination could distinguish clinical outcomes in historical cases. METHODS Automated performance metrics (instrument motion tracking metrics and system events) from 17 single-surgeon cases were recorded from 8 faculty surgeons (median: 2 cases/surgeon, range: 1-4). Based on a previous study at our institution where we showed that faster, more frequent camera usage correlated with greater surgeon experience, cases were stratified into two groups based on endoscopic camera velocity and frequency of camera manipulation during the whole RRP. Four surgeons were placed in Group 1 (optimal camera skill) and four surgeons in Group 2 (sub-optimal camera skill). Historical clinical data of these 8 surgeons from January to December 2016 was obtained from our institute′s IRB-approved database. Kruskal-Wallis and Chi-square test were used to determine differences in clinical data between groups. RESULTS 196 total historical cases were divided into Group 1 (n=125) and Group 2 (n=71) based on the surgeon′s contemporary performance using APMs (Table). There were no significant differences in patient demographics between groups (age, BMI, PSA, Gleason score, ASA). Group 1 was found to have shorter surgery time (224 vs. 245 minutes, p<0.01), decreased intra-abdominal drain duration (1 vs. 7 days, p<0.01), decreased Foley catheter duration (7 vs. 9 days, p<0.01), and less estimated blood loss (p=0.01). No differences were seen in positive margin rate, medical/surgical complications, or 6-month post-operative PSA, continence status, or erectile function (p>0.05). CONCLUSIONS A previous landmark bariatric study showed that surgical skill is associated with fewer post-operative complications, readmissions, and re-operations. Our analysis shows for the RRP, skilled surgeons with more efficient APMs experience superior intra- and post-operative outcomes, further highlighting the importance of robotic skill on patient outcomes in urologic surgery. © 2018FiguresReferencesRelatedDetails Volume 199Issue 4SApril 2018Page: e736-e737 Advertisement Copyright & Permissions© 2018MetricsAuthor Information Andrew Hung More articles by this author Paul Oh More articles by this author Jian Chen More articles by this author Nathan Cheng More articles by this author Michael Lin-Brande More articles by this author Micha Titus More articles by this author Niero Rajarubendra More articles by this author Inderbir Gill More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes

Surgical performance is critical for clinical outcomes. We present a novel machine learning (ML) method of processing automated performance metrics (APMs) to evaluate surgical performance and predict clinical outcomes after robot-assisted radical prostatectomy (RARP). We trained three ML algorithms utilizing APMs directly from robot system data (training material) and hospital length of stay (LOS; training label) (≤2 days and >2 days) from 78 RARP cases, and selected the algorithm with the best performance. The selected algorithm categorized the cases as "Predicted as expected LOS (pExp-LOS)" and "Predicted as extended LOS (pExt-LOS)." We compared postoperative outcomes of the two groups (Kruskal-Wallis/Fisher's exact tests). The algorithm then predicted individual clinical outcomes, which we compared with actual outcomes (Spearman's correlation/Fisher's exact tests). Finally, we identified five most relevant APMs adopted by the algorithm during predicting. The "Random Forest-50" (RF-50) algorithm had the best performance, reaching 87.2% accuracy in predicting LOS (73 cases as "pExp-LOS" and 5 cases as "pExt-LOS"). The "pExp-LOS" cases outperformed the "pExt-LOS" cases in surgery time (3.7 hours vs 4.6 hours, p = 0.007), LOS (2 days vs 4 days, p = 0.02), and Foley duration (9 days vs 14 days, p = 0.02). Patient outcomes predicted by the algorithm had significant association with the "ground truth" in surgery time (p < 0.001, r = 0.73), LOS (p = 0.05, r = 0.52), and Foley duration (p < 0.001, r = 0.45). The five most relevant APMs, adopted by the RF-50 algorithm in predicting, were largely related to camera manipulation. To our knowledge, ours is the first study to show that APMs and ML algorithms may help assess surgical RARP performance and predict clinical outcomes. With further accrual of clinical data (oncologic and functional data), this process will become increasingly relevant and valuable in surgical assessment and training.

Global Evaluative Assessment of Robotic Skills: Validation of a Clinical Assessment Tool to Measure Robotic Surgical Skills

We developed and validated a standardized assessment tool for robotic surgical skills, and report its usefulness, reliability and construct validity in a clinical setting. The Global Evaluative Assessment of Robotic Skills is a tool developed by deconstructing the fundamental elements of robotic surgical procedures in consultation with expert robotic surgeons. Surgical performance was assessed during robot-assisted laparoscopic prostatectomy on a 5-point anchored Likert scale across 6 domains. An overall performance score was derived by summing the ratings in each domain. Expert surgeons and postgraduate year 4 to 6 urology residents were evaluated to determine construct validity. Assessments were completed by the attending surgeon, a trained observer and the operator. A total of 29 evaluations of 25 trainees and 4 attending surgeons were completed. Experts scored significantly higher on the assessment than novice operators (p = 0.004). Postgraduate year 4 and 5 residents scored significantly lower than the expert group (p <0.05) while no difference was observed between mean performance scores of postgraduate year 6 trainees and attending surgeons (p >0.05). The internal consistency of the assessment tool was excellent (Cronbach's α = 0.90 to 0.93). The overall assessment score ICC among raters was 0.80 (95% CI 0.65-0.90). The Global Evaluative Assessment of Robotic Skills is simple to administer and able to differentiate levels of robotic surgical expertise. This standardized assessment tool shows excellent consistency, reliability and validity. Further study is warranted to evaluate its usefulness for surgical education and the establishment of competency in robotic surgery.

The R.E.N.A.L. Nephrometry Score: A Comprehensive Standardized System for Quantitating Renal Tumor Size, Location and Depth

Treatment decisions for renal malignancies depend largely on qualitative data, including a description of tumor anatomy and the experience of the treating surgeon. Currently characterization of renal tumor anatomical elements is descriptive and lacks standardization. Surgical decision making and data set comparisons would be significantly enhanced by a consistent, reproducible system that quantitates the pertinent characteristics of localized renal lesions. We have developed and propose a standardized nephrometry scoring system (R.E.N.A.L. Nephrometry Score) to quantify the anatomical characteristics of renal masses on computerized tomography/magnetic resonance imaging. The nephrometry score is based on 5 critical and reproducible anatomical features of solid renal masses. Of the 5 components 4 are scored on a 1, 2 or 3-point scale with the 5th indicating the anterior or posterior location of the mass relative to the coronal plane of the kidney. We applied the R.E.N.A.L. Nephrometry Score to 50 consecutive masses resected at Fox Chase Cancer Center. The R.E.N.A.L. Nephrometry Score consists of (R)adius (tumor size as maximal diameter), (E)xophytic/endophytic properties of the tumor, (N)earness of tumor deepest portion to the collecting system or sinus, (A)nterior (a)/posterior (p) descriptor and the (L)ocation relative to the polar line. The suffix h (hilar) is assigned to tumors that abut the main renal artery or vein. The nephrometry scoring system accurately classified the complexity of 50 consecutive tumors undergoing excision at our institution. Standardized reporting of renal tumor size, location and depth is essential for decision making and effective comparisons. The R.E.N.A.L. Nephrometry Score is a reproducible standardized classification system that quantitates the salient anatomy of renal masses. This novel approach for the systematic characterization of renal tumors provides a tool for meaningful comparisons of renal masses in clinical practice and in the urological literature.