Evaluation of a Web-Based Surgical Training Approach and Insights from Distal Bypass Competition 2021 Using a Simulator Kit

Abstract

ObjectiveThe study aimed to assess a novel web-based surgical training approach among young vascular surgeons and validate web-based evaluation methods for assessing anastomosis skills through the online Distal Bypass Competition (DBC). MethodsThe Objective Structured Assessment of Technical Skill (OSATS) was utilized to assess anastomosis proficiency using a simulator among 42 participants (17 junior and 25 senior surgeons) in the DBC 2021. Initial evaluation of basic anastomosis skills was conducted via recorded videos, comparing OSATS scores with instructor-derived impression-based scores (Comprehensive Evaluation Score: CES) during the nomination round. Real-time evaluations conducted over Zoom focused on assessing technical and composition scores of the anastomoses to determine finalists. The ice-skating method addresses potential variability among judges by excluding the highest and lowest scores for each procedural element from the final evaluation. This method, demonstrated in real-time, provides trainees with valuable feedback on which movements are highly rated, enhancing the educational impact. Evaluation criteria included precision in needlework, external appearance, handling of the intima, and internal morphology of the anastomosis. Furthermore, voluntary survey questionnaires were administered to gather participant feedback on their experience with DBC 2021. ResultsThe OSATS was associated with a lower coefficient of variation (CV) than was the CES (0.19 ± 0.07 vs. 0.24 ± 0.09, P=0.01) with regard to the evaluation of basic anastomotic skills in the nomination round. Interestingly, there was no significant difference in CV between PES based on OSATS and CES in the senior class (0.16 ± 0.06 vs. 0.22 ± 0.09, P=0.13), but a significant difference was particularly evident in the junior class (0.12 ± 0.07 vs. 0.27 ± 09, P=0.03). Lower scores were associated with higher CVs in both classes, but the OSATS improved the CV among participants who had inadequate anastomosis skills. In real-time evaluation, composition scores exhibited higher CVs compared to technical scores, even among high-scoring participants. Notably, the internal morphology of the anastomosis had the largest CV in the composition scores. The ice-skating method reduced the CV as compared to the original data evaluated using the OSATS (0.15 ± 0.04 vs. 0.09 ± 0.02 for the technical score, P<0.01 and 0.18 ± 1.00 vs. 0.07 ± 0.05 for the composition score, P=0.03); accordingly, the ice-skating method was more conducive to objective, real-time evaluation. The results of the post-DBC survey indicated that 89% of participants reported increased training time, and half of these participants indicated that their confidence in their clinical skills related to this procedure increased due to the process of training for the DBC. ConclusionsThe OSATS yielded more detailed evaluations of anastomosis skill with lower variability compared to the instructor’s impression-based evaluation in this web-based training. Real-time evaluation methods, such as the ice-skating method, have proven to be valuable tools for standardized assessment of web-based surgical education.