Operative Assessment Of Laparoscopic Skills Research Articles

Video-Based Coaching for Laparoscopic Salpingectomy: A Randomized Controlled Trial.

To evaluate the effect of video-based coaching on the gynecology resident performance of laparoscopic salpingectomy. PGY-1 and PGY-2 residents were randomized before their gynecology rotations to standard gynecology curriculum (control group) or standard curriculum plus two video-coaching sessions by a fellowship-trained minimally invasive gynecologic surgeon (VBC group). Residents were video recorded intraoperatively performing three unilateral laparoscopic salpingectomies. Participants in the VBC group were coached between the procedures. The primary outcome was the improvement in modified GOALS (Global Operative Assessment of Laparoscopic Skills) and OSA-LS (Objective Structured Assessment of Laparoscopic Salpingectomy) scores, compared with baseline, in the VBC and control groups, with videos independently graded by three blinded minimally invasive gynecologic surgeons. A minimum sample size of 18 participants (nine per group) was needed to achieve 90% power to detect a difference of 5.0 points. From October 2021 to December 2022, 28 PGY-1 and PGY-2 residents completed the study with 14 participants per group. Baseline characteristics were similar between groups. In the VBC group, modified GOALS scores significantly improved by 3.0 points from video 1 to video 2 (P=.04) and by 3.2 points from video 1 to video 3 (P=.02). Modified OSA-LS scores also increased significantly in the VBC group, by 6.1 points from video 1 to video 3 (P=.02). In the control group, modified GOALS and OSA-LS scores improved from baseline but were not significant (P=.2, P=.5). Video-based coaching also enhanced resident comfort and confidence in performing laparoscopic surgery. Video-based coaching improves resident performance of laparoscopic salpingectomy and can be used as an adjunct to resident surgical training. ClinicalTrials.gov, NCT05103449.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study

BackgroundThe learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees.MethodsThis randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores.ResultsThe participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001).ConclusionsStructured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

Decoding surgical skill: an objective and efficient algorithm for surgical skill classification based on surgical gesture features -experimental studies.

Various surgical skills lead to differences in patient outcomes and identifying poorly skilled surgeons with constructive feedback contributes to surgical quality improvement. The aim of the study was to develop an algorithm for evaluating surgical skills in laparoscopic cholecystectomy based on the features of elementary functional surgical gestures (Surgestures). Seventy-five laparoscopic cholecystectomy videos were collected from 33 surgeons in five hospitals. The phase of mobilization hepatocystic triangle and gallbladder dissection from the liver bed of each video were annotated with 14 Surgestures. The videos were grouped into competent and incompetent based on the quantiles of modified global operative assessment of laparoscopic skills (mGOALS). Surgeon-related information, clinical data, and intraoperative events were analyzed. Sixty-three Surgesture features were extracted to develop the surgical skill classification algorithm. The area under the receiver operating characteristic curve of the classification and the top features were evaluated. Correlation analysis revealed that most perioperative factors had no significant correlation with mGOALS scores. The incompetent group has a higher probability of cholecystic vascular injury compared to the competent group (30.8 vs 6.1%, P =0.004). The competent group demonstrated fewer inefficient Surgestures, lower shift frequency, and a larger dissection-exposure ratio of Surgestures during the procedure. The area under the receiver operating characteristic curve of the classification algorithm achieved 0.866. Different Surgesture features contributed variably to overall performance and specific skill items. The computer algorithm accurately classified surgeons with different skill levels using objective Surgesture features, adding insight into designing automatic laparoscopic surgical skill assessment tools with technical feedback.

Keep it simple: a crosshair on the screen during proctored laparoscopic surgeries.

Inside the operating room, experts use verbal instructions to guide surgical novices through laparoscopic procedures. In this study, we evaluated the use of a crosshair attached to the video monitor, as a hands-free pointing tool to simplify instructions during operation. Ten surgical novices performed two elective laparoscopic cholecystectomies within a week of each other, randomized such that one was performed with and the other without using the crosshair. Directly after operation, questionnaires were completed by the novices and the consultant surgeons. Measures including the comprehensibility of instructions, subjective feeling of safety during preparation, time delays due to different instruction options, and disruptive influence while instructors used the crosshair. Differences in operative performance were evaluated based on the global operative assessment of laparoscopic skills (GOALS) scores. When the crosshair was used, surgical novices had a better understanding of which anatomical structure should be shown (p = 0.028). Operating time (p = 0.222) and feeling of confidence during preparation did not differ with versus without crosshair use (p = 0.081). All participants stated that the crosshair did not negatively affect the field of vision. In terms of the median GOALS score, the operative performance was improved when the crosshair was used compared with verbal instructions only (median 15, IQR (11; 21) vs. median 12, IQR (5; 19), p < 0.001). The crosshair is a simple, inexpensive, and widely available method to improve communication between instructors and novices in everyday training.

Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial

BackgroundThe value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees. MethodsThis parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment. ResultsOf 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool. ConclusionThis pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.

Using video-based assessment (VBA) to document fellow improvement in safely completing the jejunojejunostomy portion of laparoscopic Roux-en-Y gastric bypass (RYGB) surgery.

Surgical assessment instruments are used for formative and summative trainee evaluations. To characterize the features of existing instruments and a novel 12-item objective, procedure-specific assessment tool for Roux-en-Y Gastric Bypass (RYGB-OPSA), we evaluated the progress of a single surgical fellow over 17 consecutive surgeries. Seventeen consecutive RYGB videos completed between 8/2021 and 1/2022 by an academic hospital surgical fellow were de-identified and assessed by four board-certified bariatric surgeons using Global Operative Assessment of Laparoscopic Skills (GOALS), General Assessment of Surgical Skill (GASS), and RYGB-OPSA which includes the reflection of transverse colon, identification of ligament of Treitz, biliopancreatic and Roux limbs orientation, jejunal division point selection, stapler use, mesentery division, bleeding control, jejunojejunostomy (JJ) anastomotic site selection, apposition of JJ anastomotic site, JJ creation, common enterotomy closure of JJ, and integrity of anastomosis. The GASS measured economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, and achievement of hemostasis. RYGB-OPSA and GASS items were scored "poor-unsafe," "acceptable-safe," or "good-safe." Change in performance was measured by linear trendline slope. Over the course of 17 procedures, significant improvement was demonstrated by three GOALS items, GOALS overall score, GASS bimanual dexterity, and three RYGB-OPSA tasks: JJ creation, jejunal division point selection, and stapler use. Achievement of hemostasis declined but never rated "poor-unsafe." Overall RYGB-OPSA and GOALS trendlines documented significant increase across the 17 procedures. This examination of a bariatric surgery fellow's operative training experience as measured by three surgical assessment instruments demonstrated anticipated improvements in general skills and safe completion of procedure-specific tasks. Effective surgical assessment instruments have enough sensitivity to show improvement to enable meaningful trainee feedback (low-stakes assessments) as well as the ability to determine safe surgical practice to enable promotion to greater autonomous practice.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy.

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review. Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT). Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = -0.82 to 0.02), operative time (r = -0.42 to 0.07), and the MIE-CAT scores (r = -0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = -0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42). GOALS assessments by crowd workers lacked convergent validity and showedpoor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures.

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric. The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2. The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively). The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Laparoscopic training on virtual-reality simulators or live pigs-a randomized controlled trial.

Thirty-six novice surgical residents without independent laparoscopic experience were randomly paired with a peer and randomized into three groups: VR simulator group (dyad training on LapSim VR simulators), pig surgery group (training on live, anesthetized pigs) and control group (training by a lecture on laparoscopic surgery, surgical videos and textbooks). After 6h of training, all participants performed a simulated cholecystectomy procedure using a pig liver with adherent gallbladder working in pairs. All procedures were video-recorded and the recordings were saved on USB-sticks in a blinded fashion identifiable only by the unique participant number. All video-recordings were scored blindly and independently by two expert raters using the Global Operative Assessment of Laparoscopic Skills (GOALS) assessment instrument. The performances in the three groups were significantly different, P less than 0.001. Both the VR simulation training group and the live pigs training group performed significantly better than the control group, both P values less than 0.001. However, there was no significant difference in the performance of the two simulation-based training groups, P=0.66. Novice surgical trainees can benefit from both VR simulator training and pig surgery simulation compared with traditional studying and there was no significant difference between the two modalities. The authors recommend that VR simulators should be used for basic training of laparoscopic skills and surgery on live animals should be reserved for higher-level surgical training.

Standardizing Resident Laparoscopic Hysterectomy Performance Evaluations Using Global Operative Assessment of Laparoscopic Skills

Objective: Surgical feedback is critical for resident education, but lacks standardization. Global Operative Assessment of Laparoscopic Skills (GOALS) is used to evaluate resident training in laparoscopic cholecystectomy and appendectomy. This study examined the effectiveness of adapting GOALS for laparoscopic hysterectomy in a single residency-training program. Methods: A single-page evaluation form was used for senior resident–attending dyads who were performing laparoscopic hysterectomies together. Senior residents completed self-evaluations and attendings completed trainee evaluations after each case. A 10-item checklist detailed key steps in the operation. Evaluators used the checklist to indicate if trainees completed each step successfully. Five laparoscopic domains (GOALS) and assessment of case difficulty were scored on a 5-point scale. Overall satisfaction with resident performance was rated on a 10-point scale. Results: Over 21 months, 10 residents and 3 attendings completed 97 evaluations. The mean step completion resident score was 62% (attending score: 63%; p = 0.10), indicating agreement between operators and evaluators in case assessments. Compared to attending ratings of resident performances, residents rated themselves lower overall in the 5 laparoscopic domains (3.26 versus 3.53; p < 0.05) despite similar case difficulty estimates (2.66 versus 2.76; p = 0.43). Positive correlations between attending and resident scores indicated consistency between users. Conclusions: GOALS can standardize resident performance evaluation during laparoscopic hysterectomy. There was interevaluator agreement; trainees were more self-critical. An electronic format could make the form easier to use. (J GYNECOL SURG 20XX:000)

Real-Time Augmented Reality Annotation for Surgical Education During Laparoscopic Surgery: Results from a Single-Center Randomized Controlled Trial and Future Aspects.

We developed an interactive augmented reality tool (HoloPointer) that enables real-time annotation on a laparoscopy monitor for intraoperative guidance. This application operates exclusively via verbal commands and head movements to ensure a sterile workflow. Purpose of this randomized controlled clinical trial was to evaluate the integration of this new technology into the operating room. This prospective single-center study included 32 elective laparoscopic cholecystectomies (29 surgical teams, 15 trainees, 13 trainers). Primary objectives and assessment measures was the HoloPointer's influence on surgical performance (subjective assessment, global operative assessment of laparoscopic skills - GOALS, and Critical View of Safety -CVS). Secondary objectives and outcome variables were its influence on operation time, quality of assistance (5 point likert-scale), and user-friendliness (System Usability Scale - SUS, 0-100 points). Gestural corrections were reduced by 59.4% (4.6 SD 8.1 vs. 1.9 SD 4.7; p > 0.05) and verbal corrections by 36.1% (17.8 SD 12.9 vs. 11.4 SD 8.1; p > 0.05). Subjective surgical performance could be improved by 84,6% of participants. No statistically significant differences were observed for objective parameters GOALS, CVS and operation time. In the SUS, the application achieved an average score of 72.5 SD 16.3 (good user-friendliness). Of the participants, 69.2% wanted to use the HoloPointer more frequently. The majority of trainees had improved their surgical performance using the HoloPointer in elective laparoscopic cholecystectomies, and the rate of classic but potentially misleading corrections was noticeably reduced. The HoloPointer has the potential to improve education in minimally invasive surgery.

Value of the Crowd-Sourced Assessment of Technical Skills (C-SATS) Platform in Surgical Procedures: A Systematic Review of Evidence

Background: Crowd-Sourced Assessment of Technical Skills (C-SATS) is a surgical data management and learning platform that leverages the knowledge of large expert surgeon and lay groups to assess the technique and technical skills of surgeons in a highly efficient manner. The aim of this systematic review was to summarize published literature on the performance of C-SATS as compared to expert evaluations and assess its use as a training and validation tool in minimally invasive surgery (MIS). Methods: A comprehensive literature search was performed per PRISMA guidelines using the Medline, Embase, and Google Scholar databases on published studies that evaluated the use of C-SATS following MIS, such as laparoscopic or robotic-assisted surgery. Results: A total of 21 reports were included in the review. Twelve studies comparing crowd-sourcing evaluations against expert opinion indicated overall excellent or good correlation with Global Operative Assessment of Laparoscopic Skills (GOALS), Global Evaluative Assessment of Robotic Skills (GEARS), and Robotic Objective Structured Assessment of Technical Skills (R-OSATS) scores, with correlation coefficients (Pearson or Spearman) ranging from 0.69 to 0.95 and reliability index (Cronbach’s alpha) from 0.63 to 0.93 across different specialties and surgical approaches. When using C-SATS to assess performance and validation, assessments positively correlated with traditional methods of time and error-based scoring and global rating scale. Conclusions: Based on the current published literature, the C-SATS platform has been shown to efficiently provide crowd-sourced evaluations that correlate favorably with expert evaluation across a range of surgical specialties and approaches. Use of crowdsourcing has uniformly yielded accurate evaluations of surgeons’ technical skills in a markedly shorter time than expert reviews. C-SATS may be a cost-effective complement or alternative to traditional models of evaluating surgical proficiency. Future studies are needed to determine whether the use of C-SATS will lead to improved surgical performance and patient outcomes.

Impact of the Recorded Audio and Video Feedback on Simulated Surgical Performance: A Randomized Controlled Trial.

Simulation and coaching have become increasingly important in laparoscopic skills acquisition.This study was designed to evaluate if access to the recorded audio and video feedback after a single proctored session improves the acquisition of laparoscopic suturing skills in obstetrics and gynecology (OB/GYN) residents. Twenty OB/GYN residents received a single face-to-face coaching session on a laparoscopic vaginal cuff closure model. The session was recorded and residents were randomized to access either the video-only or the audio and video recording of the proctored session. The primary outcome measure was comparison of Global Operative Assessment of Laparoscopic Skills plus Vaginal Cuff Metrics (GOALS+) scores of the vaginal cuff closure prior to and following the proctored session. Only 30% of residents accessed the recorded sessions with junior residents most likely to access the recording.Baseline GOALS+ scores were significantly higher in senior residents (mean 21.7, SD 3.9) as compared to junior residents (mean 14.7, SD 3.2) (p<.001). While all learners' GOALS+ scores significantly improved after proctoring the intervention (p<.001), the senior residents continued to have significantly higher GOALS+ scores at the final assessment (mean 28.3, SD 4.2, p=.01) when compared to their junior residents (mean 24.0, SD 3.1). Due to the low uptake of the review of recorded proctored sessions among OB/GYN residents across skill and year levels, we were unable to assess the effect of recorded audio and video feedback on resident performance. However, the intervention of a single proctored session of simulated laparoscopic vaginal cuff closure significantly improved resident performance as assessed with GOALS+ scores.

Video Playback as an Educational Tool for Improving Resident Laparoscopic Performance: A Randomized Controlled Trial

Study ObjectiveIn this single-masked randomized controlled study, we evaluate whether watching video recordings of oneself performing Fundamentals of Laparoscopic Skills (FLS) exercises results in an improvement on the 30-point Global Operative Assessment of Laparoscopic Skills (GOALS) assessment among Obstetrics and Gynecology (Ob/Gyn) residents. DesignTwenty-three Ob/Gyn residents in the 2020–2021 academic year completed the FLS exercises while being timed, video recorded, and receiving real-time feedback from an Ob/Gyn faculty member. Baseline GOALS assessment was completed by participants and faculty. After the intervention, all participants then repeated the FLS exercises while being timed and were again scored using the GOALS assessment. Each participant completed the study in a single session. In addition, all participants completed a pre- and post-test survey. SettingUniversity of Louisville Laparoscopic Skills Labortaory. ParticipantsUniversity of Louisville Ob/Gyn residents in the 2020–2021 academic year. InterventionTwelve participants were randomized to the intervention and were allowed to watch their video recording in addition to receiving verbal feedback whereas the remaining 11 received verbal feedback only. Measurements and Main ResultsThere were significant improvements in faculty (p <.01) and self-reported GOALS scores (p <.01) when comparing both the intervention and control group with baseline scores. The intervention group improved by 3.2 points more than that of the control group based on masked faculty evaluation (95% confidence interval, 1.4–5.0 points; p <.01). The difference was not significant in resident self-scores. Overall time improved for all participants (15:54 ± 0.21 minutes before and 13:13 ± 0.14 minutes after), but this difference was not significant between the 2 groups. Higher postgraduate year (PGY) residents reported significantly more comfort performing laparoscopic tasks, earned higher GOALS score (faculty and self-scores), and completed the first set of exercises in less time. A significant interaction between PGY and intervention was detected with improvement in GOALS score most strongly associated with PGY4 residents (p <.01). ConclusionAlthough all learners objectively benefit from watching recordings of themselves performing surgical tasks, advanced learners may benefit the most when video recordings are used as an educational tool.

Assessing the Learning Process of Transforaminal Endoscopic Discectomy for Sciatica.

ObjectivePercutaneous transforaminal endoscopic discectomy (PTED) is gaining popularity by both surgeons and patients as a less invasive treatment option for sciatica. Concerns, however, exist for its learning curve. No previous study has assessed the learning process of PTED. Hereby we present the learning process of 3 surgeons learning PTED. MethodsThis analysis was conducted alongside a multicenter randomized controlled trial. After attending a cadaveric workshop, 3 spine-dedicated surgeons started performing PTED, initially under the supervision of a senior surgeon. After each 5 cases, and up to case 20, the learning process was evaluated using the validated questionnaires (objective structured assessment of technical skills [OSATS], global operative assessment of laparoscopic skills [GOALS]) and a 10-step checklist specifically developed for PTED. ResultsIn total, 3 learning curve surgeons performed a total of 161 cases. Based on self-assessment, surgeons improved mostly in the domains “time and motion,” “respect for tissue,” and “knowledge and handling of instruments.” Learning curve surgeons were more able to detect differences in performances on the OSATS than the senior surgeon. Based on the GOALS, the biggest improvements could be seen in “depth-perception” and “autonomy.” Based on the 10-item specific checklist, all surgeons performed all 10 steps by case 10, while only 1 surgeon performed all steps adequately by case 15. ConclusionBased on these study results, PTED appears to be successfully adopted stepwise by 3 spine-dedicated surgeons. From 15 cases on, most steps are performed adequately. However, more cases might be necessary to achieve good clinical results. Validated tools are needed to determine the cutoff when a surgeon should be able to perform PTED independently.

282. VALIDITY AND RELIABILITY OF CROWD-SOURCED AND EXPERT VIDEO ASSESSMENT IN MINIMALLY INVASIVE ESOPHAGECTOMY USING GLOBAL RATING SCALES

Abstract Minimally Invasive Esophagectomy (MIE) is a complex and technically demanding procedure and implementation is associated with a substantial learning curve. Suboptimal surgical performance in complex minimally invasive procedures has convincingly been related to less favorable patient outcomes. Structured video-based assessment by other surgeons can objectively evaluate surgical performance, but requires a surgeon’s scarce time. This study investigated whether crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, could be a valid alternative for MIE. Two video assessments types were conducted via the C-SATS platform with eight MIE operative videos. First, videos were cut into eight procedural phase-based video clips of maximum ten minutes. These 64 video clips were assessed with the validated Global Operative Assessment of Laparoscopic Skills (GOALS)‘s four domains (depth perception, bimanual dexterity, efficiency and tissue handling) using a 1-5 Likert scale. Each video clip was assessed by a pair of two MIE experts and &amp;gt; 30 crowd workers, recruited via Amazon Mechanical Turk. Second, the same expert pairs assessed the corresponding complete video with the procedure-specific MIE competency assessment tool (MIE-CAT). Experts assessed 128 clips and sixteen complete videos in 3.5 months. Crowd workers assessed 1984 clips in &amp;lt; two days. Expert scores showed a wider range (expert GOALS score range 7.50-19.50 versus 12.84-15.63 for crowds). When comparing experts and crowd workers scores, a poor correlation was found on all domains and total GOALS scores (ICC ≤ 0.42). When correlating GOALS scores with MIE-CAT scores a strong, positive correlation between experts’ scores was found (r¬s = 0.76, p = 0.03) while between crowd’s GOALS and expert’s MIE-CAT scores a small correlation was found (r¬s = 0.23, p = 0.58) (Figure 1). Despite rapid results from crowd workers, a poor agreement between crowd and experts’ GOALS scores was observed. MIE might be technically too difficult to assess for laypeople. Furthermore, GOALS might not be extensive enough to assess MIE performance in detail, while the procedure-specific MIE-CAT might be more comprehensive. However, the correlation between the experts’ MIE-CAT and GOALS scores indicates video clip assessments (instead of full length video assessments) could be helpful to shorten assessment time.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy.

IntroductionSelf-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.Methods and proceduresUsing a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1–3] and senior trainees (PGY 4–5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.ResultsA total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.ConclusionsOPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00464-022-09466-6.

Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools

Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training. The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale. Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson's correlations were calculated for each scale. Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores. The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.

The impact of virtual reality simulation training on operative performance in laparoscopic cholecystectomy: meta-analysis of randomized clinical trials.

BackgroundSimulation training can improve the learning curve of surgical trainees. This research aimed to systematically review randomized clinical trials (RCT) evaluating the performance of junior surgical trainees following virtual reality training (VRT) and other training methods in laparoscopic cholecystectomy.MethodsMEDLINE (PubMed), Embase (Ovid SP), Web of Science, Scopus and LILACS were searched for trials randomizing participants to VRT or no additional training (NAT) or simulation training (ST). Outcomes of interest were the reported performance using global rating scores (GRS), the Objective Structured Assessment of Technical Skill (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS), error counts and time to completion of task during laparoscopic cholecystectomy on either porcine models or humans. Study quality was assessed using the Cochrane Risk of Bias Tool. PROSPERO ID: CRD42020208499.ResultsA total of 351 titles/abstracts were screened and 96 full texts were reviewed. Eighteen RCT were included and 15 manuscripts had data available for meta-analysis. Thirteen studies compared VRT and NAT, and 4 studies compared VRT and ST. One study compared VRT with NAT and ST and reported GRS only. Meta-analysis showed OSATS score (mean difference (MD) 6.22, 95%CI 3.81 to 8.36, P < 0.001) and time to completion of task (MD −8.35 min, 95%CI 13.10 to 3.60, P = <0.001) significantly improved after VRT compared with NAT. No significant difference was found in GOALS score. No significant differences were found between VRT and ST groups. Intraoperative errors were reported as reduced in VRT groups compared with NAT but were not suitable for meta-analysis.ConclusionMeta-analysis suggests that performance measured by OSATS and time to completion of task is improved with VRT compared with NAT for junior trainee in laparoscopic cholecystectomy. However, conclusions are limited by methodological heterogeneity and more research is needed to quantify the potential benefit to surgical training.

Surgesture: a novel instrument based on surgical actions for objective skill assessment

Due to varied surgical skills and the lack of an efficient rating system, we developed Surgesture based on elementary functional surgical gestures performed by surgeons, which could serve as objective metrics to evaluate surgical performance in laparoscopic cholecystectomy (LC). We defined 14 LC basic Surgestures. Four surgeons annotated Surgestures among LC videos performed by experts and novices. The counts, durations, average action time, and dissection/exposure ratio (D/E ratio) of LC Surgestures were compared. The phase of mobilizing hepatocystic triangle (MHT) was extracted for skill assessment by three professors using a modified Global Operative Assessment of Laparoscopic Skills (mGOALS). The novice operation time was significantly longer than the expert operation time (58.12 ± 19.23min vs. 26.66 ± 8.00min, P < 0.001), particularly during MHT phase. Novices had significantly more Surgestures than experts in both hands (P < 0.05). The left hand and inefficient Surgesture of novices were dramatically more than those of experts (P < 0.05). The experts demonstrated a significantly higher D/E ratio of duration than novices (0.79 ± 0.37 vs. 2.84 ± 1.98, P < 0.001). The counts and time pattern map of LC Surgestures during MHT demonstrated that novices tended to complete LC with more types of Surgestures and spent more time exposing the surgical scene. The performance metrics of LC Surgesture had significant but weak associations with each aspect of mGOALS. The newly constructed Surgestures could serve as accessible and quantifiable metrics for demonstrating the operative pattern and distinguishing surgeons with various skills. The association between Surgestures and Global Rating Scale laid the foundation for establishing a bridge to automated objective surgical skill evaluation.