Comparison of Global Evaluative Assessment of Robotic Surgery with objective performance indicators for the assessment of skill during robotic-assisted thoracic surgery

Abstract

BackgroundThe Global Evaluative Assessment of Robotic Skills is a popular but ultimately subjective assessment tool in robotic-assisted surgery. An alternative approach is to record system or console events or calculate instrument kinematics to derive objective performance indicators. The aim of this study was to compare these 2 approaches and correlate the Global Evaluative Assessment of Robotic Skills with different types of objective performance indicators during robotic-assisted lobectomy. MethodsVideo, system event, and kinematic data were recorded from the robotic surgical system during left upper lobectomy on a standardized perfused and pulsatile ex vivo porcine heart–lung model. Videos were segmented into steps, and the superior vein dissection was graded independently by 2 blinded expert surgeons with Global Evaluative Assessment of Robotic Skills. Objective performance indicators representing categories for energy use, event data, movement, smoothness, time, and wrist articulation were calculated for the same task and compared to Global Evaluative Assessment of Robotic Skills scores. ResultsVideo and data from 51 cases were analyzed (44 fellows, 7 attendings). Global Evaluative Assessment of Robotic Skills scores were significantly higher for attendings (P < .05), but there was a significant difference in raters’ scores of 31.4% (defined as >20% difference in total score). The interclass correlation was 0.44 for 1 rater and 0.61 for 2 raters. Objective performance indicators correlated with Global Evaluative Assessment of Robotic Skills to varying degrees. The most highly correlated Global Evaluative Assessment of Robotic Skills domain was efficiency. Instrument movement and smoothness were highly correlated among objective performance indicator categories. Of individual objective performance indicators, right-hand median jerk, an objective performance indicator of change of acceleration, had the highest correlation coefficient (0.55). ConclusionThere was a relatively poor overall correlation between the Global Evaluative Assessment of Robotic Skills and objective performance indicators. However, both appear strongly correlated for certain metrics such as efficiency and smoothness. Objective performance indicators may be a potentially more quantitative and granular approach to assessing skill, given that they can be calculated mathematically and automatically without subjective interpretation.