Validity Evidence for Vascular Skills Assessment: The Feasibility of Fundamentals of Vascular Surgery in General Surgery Residency

Abstract

As the Fundamentals of Laparoscopic Surgery (FLS) and Fundamentals of Endoscopic Surgery (FES) have been used for general surgery assessment, the Fundamentals of Vascular Surgery (FVS) has recently been developed to evaluate core operative skills for vascular trainees. This study examines the 3 year implementation of FVS for general surgery residents and it gathers validity evidence using Messick's framework. We hypothesized that the curriculum and assessment tool enhance general surgery resident training and assessment. This is a retrospective review of FVS assessments of residents using descriptive and multivariate analyses. This study was conducted at an academic institution, where simulation-based teaching sessions occur in coordination between the general surgery and the integrated vascular surgery residency programs. Seventeen general surgery residents were assessed in FVS skills by an expert rater from 2018 to 2020. Overall, 86 assessments were completed. Assessment focuses on 3 open vascular skills (End-to-Side Anastomosis, Patch Angioplasty and Clockface Suturing). Response Process: 7 items comprise a graded rating for a skills score. Additionally, a global summary score is designated. Internal Structure: The assessment tool has a Cronbach's alpha of 0.87, demonstrating good internal consistency. Addition of the second rater correlated with Cohen's kappa -0.69 (p < 0.001), indicating poor interrater reliability. Relationships to other variables: The most significant improvement occurred in total scores between PGY2s (17.4 ± 2.37) and PGY4s (23.2 ± 3.00), p < 0.001, indicating adequate level discernment. The validity evidence of FVS assessment in this study supports its use in general surgery residency at a time when opportunities for open vascular skills assessment may be decreasing due to case availability and shifting paradigms. Further study into quality rater training is needed to optimize national implementation of FVS and ensure consistency in grading.