Assessment of Robotic Console Skills (ARCS): construct validity of a novel global rating scale for technical skills in robotically assisted surgery.

Abstract

Skill assessment during robotically assisted surgery remains challenging. While the popularity of the Global Evaluative Assessment of Robotics Skills (GEARS) has grown, its lack of discrimination between independent console skills limits its usefulness. The purpose of this study was to evaluate construct validity and interrater reliability of a novel assessment designed to overcome this limitation. We created the Assessment of Robotic Console Skills (ARCS), a global rating scale with six console skill domains. Fifteen volunteers who were console surgeons for 0 ("novice"), 1-100 ("intermediate"), or >100 ("experienced") robotically assisted procedures performed three standardized tasks. Three blinded raters scored the task videos using ARCS, with a 5-point Likert scale for each skill domain. Scores were analyzed for evidence of construct validity and interrater reliability. Group demographics were indistinguishable except for the number of robotically assisted procedures performed (p=0.001). The mean scores of experienced subjects exceeded those of novices in dexterity (3.8>1.4, p<0.001), field of view (4.1>1.8, p<0.001), instrument visualization (3.9>2.2, p<0.001), manipulator workspace (3.6>1.9, p=0.001), and force sensitivity (4.3>2.6, p<0.001). The mean scores of intermediate subjects exceeded those of novices in dexterity (2.8>1.4, p=0.002), field of view (2.8>1.8, p=0.021), instrument visualization (3.2>2.2, p=0.045), manipulator workspace (3.1>1.9, p=0.004), and force sensitivity (3.7>2.6, p=0.033). The mean scores of experienced subjects exceeded those of intermediates in dexterity (3.8>2.8, p=0.003), field of view (4.1>2.8, p<0.001), and instrument visualization (3.9>3.2, p=0.044). Rater agreement in each domain demonstrated statistically significant concordance (p<0.05). We present strong evidence for construct validity and interrater reliability of ARCS. Our study shows that learning curves for some console skills plateau faster than others. Therefore, ARCS may be more useful than GEARS to evaluate distinct console skills. Future studies will examine why some domains did not adequately differentiate between subjects and applications for intraoperative use.