Ongoing Evaluation of Video-Based Assessment of Proctors’ Scoring of the Fundamentals of Laparoscopic Surgery Manual Skills Examination

Abstract

In continued efforts to maintain standards required for high-stakes examination administration, Society of American Gastrointestinal and Endoscopic Surgeons' Fundamentals of Laparoscopic Surgery (FLS) program requires all prospective test proctors to complete the New FLS Proctor Workshop. As the FLS proctor pool and training evolves, the influence of experience on performance is not clear. In the previous work, we proposed that asynchronous web-based performances might be used to train proctors. In this study, we assessed the accuracy of proctors' ratings from asynchronous web-based performances by comparing the sensitivity and the specificity of ratings from proctors having varied experience levels. A sample of 42 (26% of all registered proctors) FLS proctors (recently trained novice, n = 15; intermediate, n = 20; and expert, n = 7) viewed 2 videotaped, laparoscopic-view FLS performances via the web. The first performance (error) contained 1 predetermined critical performance error in each of the 5 tasks (5 total errors), whereas the second performance (no error) contained no critical errors. For both the performances, participants noted time to complete each of the 5 tasks and any critical errors they observed (dichotomously scored). Participants also completed a demographic section that captured years of experience as a proctor. Using the Kruskal-Wallis test, we compared new trainee, intermediate, and expert proctors' recorded task times for both the performances. The sensitivity and the specificity values were also independently calculated using ratings from the error and the no error performances and then compared using the same test. There were no differences in recorded times across proctor groups for any of the tasks; p = 0.21 and 0.94. Rating sensitivity was 79% (novice), 75% (intermediate), and 83% (expert), with no significant differences across experience levels; p > 0.46. Following removal of the peg transfer task that had particularly low sensitivity, the sensitivity for the remaining 4 tasks increased [88% and 92%]. The specificity was 93% (novice), 86% (intermediate), and 100% (expert); p = 0.046. Comments from less-experienced proctors indicated that their focus on noncritical performance issues, such as "loop not on black line" and "air knot," were consistent with decreased specificity. Favorable results of this pilot study suggest that web-based FLS performances may be used for assessing FLS proctors' rating quality with adequate sensitivity and specificity. Decreased rating specificity from intermediate proctors indicates that these proctors would benefit from increased testing frequencies, updated training on current FLS proctoring standards, and ongoing performance review.