Validation of Video-based Skill Assessment in Carotid Artery Stenting

Abstract

Objectives: To develop weighted error-based, generic and procedure-specific rating scales, to validate these scales for video-based assessment during virtual carotid artery stent (CAS) procedures and correlate them with simulator-derived metrics. Methods: A questionnaire was developed to assess the technique during live CAS procedures. Errors were rated from 1 (unimportant) to 5 (life-threatening) by 28 highly experienced CAS (>50 CAS) physicians. Virtual CAS procedure was performed by 21 interventionalists with varied CAS experience. Fluoroscopy screen and hand movements were video-taped, and simulator-derived metrics recorded. Experienced CAS practitioners then rated the video-taped performances using weighted error, generic and procedure-specific rating scales. Results: Of the 23 errors assessed, 12 were regarded as moderate (score 3), six serious (score 4) and four life-threatening (score 5). The generic rating scale was able to detect significant differences in performance between inexperienced and experienced CAS operators (score 25 vs. 32 respectively, P < 0.01). All scoring systems demonstrated good inter-rater reliability (α = 0.61–0.87). Significant correlations were observed between simulator-derived and video-based scores: weighted error-based score (r: 0.76, P < 0.01), generic (r: 0.62, P < 0.01) and procedure-specific (r: 0.76, P < 0.01) rating scales. Conclusions: The generic endovascular rating scale differentiated between levels of CAS experience among skilled interventionalists and correlated to simulator-based error scoring.