Global Operative Assessment Of Laparoscopic Skills Research Articles

Optimizing modern surgical simulation through instructor feedback - insights from a retrospective observational study in a tertiary hospital.

Laparoscopic surgery is associated with a prolonged learning curve for emerging surgeons, and simulation-based training (SBT) has become increasingly prominent in this context due to stringent working time regulations and heightened concerns regarding patient safety. While SBT offers a safe and ethical learning environment, the accuracy of simulators in the context of evaluating surgical skills remains uncertain. This study aims to assess the precision of a laparoscopic simulator with regard to evaluating surgical performance and to identify the instructor's role in SBT. This retrospective study focused on surgical residents in their 1st through 5th years at the Department of Surgery of Linkou Chang Gung Memorial Hospital. The residents participated in a specially designed SBT program using the LapSim laparoscopic simulator. Following the training session, each resident was required to perform a laparoscopic procedure and received individualized feedback from an instructor. Both simulator and instructor evaluated trainees' performance on the LapSim, focusing on identifying correlations between the simulator's metrics and traditional assessments. Senior residents (n = 15), who employed more complex laparoscopic procedures, exhibited more significant improvements after receiving instructor feedback than did junior residents (n = 17). Notably, a stronger correlation between the simulator and instructor assessments was observed in the junior group (junior Global Operative Assessment of Laparoscopic Skills (GOALS) adjusted R2 = 0.285, p = 0.016), while no such correlations were observed among the senior group. A well-designed, step-by-step SBT can be a valuable tool in laparoscopic surgical training. LapSim simulator has demonstrated its potential in assessing surgical performances during the early stages of surgical training. However, instructors must provide intuitive feedback to ensure appropriate learning in later stages.

Video-Based Coaching for Laparoscopic Salpingectomy: A Randomized Controlled Trial.

To evaluate the effect of video-based coaching on the gynecology resident performance of laparoscopic salpingectomy. PGY-1 and PGY-2 residents were randomized before their gynecology rotations to standard gynecology curriculum (control group) or standard curriculum plus two video-coaching sessions by a fellowship-trained minimally invasive gynecologic surgeon (VBC group). Residents were video recorded intraoperatively performing three unilateral laparoscopic salpingectomies. Participants in the VBC group were coached between the procedures. The primary outcome was the improvement in modified GOALS (Global Operative Assessment of Laparoscopic Skills) and OSA-LS (Objective Structured Assessment of Laparoscopic Salpingectomy) scores, compared with baseline, in the VBC and control groups, with videos independently graded by three blinded minimally invasive gynecologic surgeons. A minimum sample size of 18 participants (nine per group) was needed to achieve 90% power to detect a difference of 5.0 points. From October 2021 to December 2022, 28 PGY-1 and PGY-2 residents completed the study with 14 participants per group. Baseline characteristics were similar between groups. In the VBC group, modified GOALS scores significantly improved by 3.0 points from video 1 to video 2 ( P =.04) and by 3.2 points from video 1 to video 3 ( P =.02). Modified OSA-LS scores also increased significantly in the VBC group, by 6.1 points from video 1 to video 3 ( P =.02). In the control group, modified GOALS and OSA-LS scores improved from baseline but were not significant ( P =.2, P =.5). Video-based coaching also enhanced resident comfort and confidence in performing laparoscopic surgery. Video-based coaching improves resident performance of laparoscopic salpingectomy and can be used as an adjunct to resident surgical training. ClinicalTrials.gov , NCT05103449.

Be prepared! Impact of structured video-assisted coaching on performance in a simulated bleeding exercise during laparoscopic surgery

BackgroundBleeding during laparoscopic surgery is stressful and requires immediate efficient management. Skills for complication management are rarely trained. This study aims to investigate the impact of video-assisted coaching on laparoscopic skills acquisition and performance in emergency bleeding situations.MethodsParticipants faced simulated emergency scenarios during laparoscopy involving bleeding management in porcine aorta/kidney specimens. Four sequences were conducted over two days, with a structured video-assisted coaching provided between sequences. Performance was assessed using the Global Operative Assessment of Laparoscopic Skills (GOALS) score. The study involved 27 participants attending the advanced colorectal surgery module at the 40th Annual Davos Course in 2023.Results54 video sequences were analyzed. Structured video-assisted coaching improved the GOALS sum score by 0.36 (95%CI: 0.21–0.50, P < 0.001) in contrast to simple repetition (0.05 with 95%CI: −0.43 to 0.53, P = 0.826). This association was observed for depth of perception (P < 0.001), bimanual dexterity (P < 0.001), tissue handling (P < 0.001), overall performance (P < 0.001), and efficiency (P < 0.001). Autonomy did not significantly improve (P = 0.55). Findings were consistent regardless of age, gender, and overall laparoscopic experience of the participants. However, a weaker effect of structured video-assisted coaching was observed in participants with experience in laparoscopic surgery.ConclusionStructured video-assisted coaching improved performance in laparoscopic skills in complex and stress-inducing bleeding scenarios. The findings of this study support the incorporation of video-assisted coaching and complication management exercises into surgical training curricula.

Surgical skill analysis focused on tissue traction in laparoscopic wet lab training

BackgroundTissue handling is one of the pivotal parts of surgical procedures. We aimed to elucidate the characteristics of experts' left-hand during laparoscopic tissue dissection. MethodsParticipants performed tissue dissection around the porcine aorta. The grasping force/point of the grasping forceps were measured using custom-made sensor forceps, and the forceps location was also recorded by motion capture system (Mocap). According to the global operative assessment of laparoscopic skills (GOALS), two experts scored the recorded movies, and based on the mean scores, participants were divided into three groups: novice (<10), intermediate (10≤ to <20), and expert (≤20). Force-based metrics were compared among the three groups using the Kruskal-Wallis test. Principal component analysis (PCA) using significant metrics was also performed. ResultsA total of 42 trainings were successfully recorded. The statistical test revealed that novices frequently regrasped a tissue (median total number of grasps, novices: 268.0 times, intermediates: 89.5, experts: 52.0, p < 0.0001), the traction angle became stable against the aorta (median weighted standard deviation of traction angle, novices: 30.74°, intermediates: 26.80, experts: 23.75, p = 0.0285), and the grasping point moved away from the aorta according to skill competency [median percentage of grasping force applied in close zone (0 to 2.0 cm from aorta), novices: 34.96 %, intermediates: 21.61 %, experts: 10.91 %, p = 0.0032]. PCA showed that the efficiency-related (total number of grasps) and effective tissue traction-related (weighted average grasping position in Y-axis and distribution of grasping area) metrics mainly contributed to the skill difference (proportion of variance of first principal component: 60.83 %). ConclusionThe present results revealed experts' left-hand characteristics, including correct tissue grasping, sufficient tissue traction from the aorta, and stable traction angle. Our next challenge is the provision of immediate and visual feedback onsite after the present wet-lab training, and shortening the learning curve of trainees.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study

BackgroundThe learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees.MethodsThis randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores.ResultsThe participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001).ConclusionsStructured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

Keep it simple: a crosshair on the screen during proctored laparoscopic surgeries.

Inside the operating room, experts use verbal instructions to guide surgical novices through laparoscopic procedures. In this study, we evaluated the use of a crosshair attached to the video monitor, as a hands-free pointing tool to simplify instructions during operation. Ten surgical novices performed two elective laparoscopic cholecystectomies within a week of each other, randomized such that one was performed with and the other without using the crosshair. Directly after operation, questionnaires were completed by the novices and the consultant surgeons. Measures including the comprehensibility of instructions, subjective feeling of safety during preparation, time delays due to different instruction options, and disruptive influence while instructors used the crosshair. Differences in operative performance were evaluated based on the global operative assessment of laparoscopic skills (GOALS) scores. When the crosshair was used, surgical novices had a better understanding of which anatomical structure should be shown (p = 0.028). Operating time (p = 0.222) and feeling of confidence during preparation did not differ with versus without crosshair use (p = 0.081). All participants stated that the crosshair did not negatively affect the field of vision. In terms of the median GOALS score, the operative performance was improved when the crosshair was used compared with verbal instructions only (median 15, IQR (11; 21) vs. median 12, IQR (5; 19), p < 0.001). The crosshair is a simple, inexpensive, and widely available method to improve communication between instructors and novices in everyday training.

Development and validation of a simulation training platform for the ligation of deep dorsal vein complex in radical prostatectomy.

This study aimed to design a low-cost, simulation training platform for the ligation of deep dorsal vein (DVC) complex in radical prostatectomy and validate its training effectiveness. A simplified prostate urethra model was produced by 0-degree silica gel and pulse pressure banding. This model was placed on a slope of about 30 degrees using cardboard to thus creating a narrow environment of the pelvis. The DVC ligation was performed by a 2D laparoscopy simulator. A total of 27 participants completed the study include 13 novices, 10 surgical residents and 4 urology experts. The novices were trained five trails with 24 hours interval, the residents and experts completed the DVC ligation once. The construct validity of this simulation training platform was performed by completing time, the GOALS (Global Operative Assessment of Laparoscopic Skills) and TSA (i.e. Task Specific Assessments) score. The face validity and content validity were performed by a specific closed-ended questionnaire. There was no significant difference among three groups in demographic or psychometric variables (p > 0.05). Compared to the novices, the residents spend a shorter time to complete the DVC ligation (p < 0.05) and had higher GOALS scores (p < 0.05), but had no significant difference in TSA scores (p > 0.05). Additionally, the experts groups had a better performance compared to residents group in the completing time (p < 0.05), GOALS score (p < 0.05) and TSA score (p < 0.05). The learning curve of novices significantly promoted along with the increased times of training. Almost 90 percent of subjects considered that this simulator had a good performance in the realism and practicability. We developed a novel low-cost a simulation training platform for the ligation of deep dorsal vein complex in radical prostatectomy, and this simulator had a good performance in the construct validity, face validity and content validity.

Using video-based assessment (VBA) to document fellow improvement in safely completing the jejunojejunostomy portion of laparoscopic Roux-en-Y gastric bypass (RYGB) surgery.

Surgical assessment instruments are used for formative and summative trainee evaluations. To characterize the features of existing instruments and a novel 12-item objective, procedure-specific assessment tool for Roux-en-Y Gastric Bypass (RYGB-OPSA), we evaluated the progress of a single surgical fellow over 17 consecutive surgeries. Seventeen consecutive RYGB videos completed between 8/2021 and 1/2022 by an academic hospital surgical fellow were de-identified and assessed by four board-certified bariatric surgeons using Global Operative Assessment of Laparoscopic Skills (GOALS), General Assessment of Surgical Skill (GASS), and RYGB-OPSA which includes the reflection of transverse colon, identification of ligament of Treitz, biliopancreatic and Roux limbs orientation, jejunal division point selection, stapler use, mesentery division, bleeding control, jejunojejunostomy (JJ) anastomotic site selection, apposition of JJ anastomotic site, JJ creation, common enterotomy closure of JJ, and integrity of anastomosis. The GASS measured economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, and achievement of hemostasis. RYGB-OPSA and GASS items were scored "poor-unsafe," "acceptable-safe," or "good-safe." Change in performance was measured by linear trendline slope. Over the course of 17 procedures, significant improvement was demonstrated by three GOALS items, GOALS overall score, GASS bimanual dexterity, and three RYGB-OPSA tasks: JJ creation, jejunal division point selection, and stapler use. Achievement of hemostasis declined but never rated "poor-unsafe." Overall RYGB-OPSA and GOALS trendlines documented significant increase across the 17 procedures. This examination of a bariatric surgery fellow's operative training experience as measured by three surgical assessment instruments demonstrated anticipated improvements in general skills and safe completion of procedure-specific tasks. Effective surgical assessment instruments have enough sensitivity to show improvement to enable meaningful trainee feedback (low-stakes assessments) as well as the ability to determine safe surgical practice to enable promotion to greater autonomous practice.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy.

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review. Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT). Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = -0.82 to 0.02), operative time (r = -0.42 to 0.07), and the MIE-CAT scores (r = -0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = -0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42). GOALS assessments by crowd workers lacked convergent validity and showedpoor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures.

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric. The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2. The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively). The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Laparoscopic training on virtual-reality simulators or live pigs-a randomized controlled trial.

Thirty-six novice surgical residents without independent laparoscopic experience were randomly paired with a peer and randomized into three groups: VR simulator group (dyad training on LapSim VR simulators), pig surgery group (training on live, anesthetized pigs) and control group (training by a lecture on laparoscopic surgery, surgical videos and textbooks). After 6h of training, all participants performed a simulated cholecystectomy procedure using a pig liver with adherent gallbladder working in pairs. All procedures were video-recorded and the recordings were saved on USB-sticks in a blinded fashion identifiable only by the unique participant number. All video-recordings were scored blindly and independently by two expert raters using the Global Operative Assessment of Laparoscopic Skills (GOALS) assessment instrument. The performances in the three groups were significantly different, P less than 0.001. Both the VR simulation training group and the live pigs training group performed significantly better than the control group, both P values less than 0.001. However, there was no significant difference in the performance of the two simulation-based training groups, P=0.66. Novice surgical trainees can benefit from both VR simulator training and pig surgery simulation compared with traditional studying and there was no significant difference between the two modalities. The authors recommend that VR simulators should be used for basic training of laparoscopic skills and surgery on live animals should be reserved for higher-level surgical training.

Standardizing Resident Laparoscopic Hysterectomy Performance Evaluations Using Global Operative Assessment of Laparoscopic Skills

Objective: Surgical feedback is critical for resident education, but lacks standardization. Global Operative Assessment of Laparoscopic Skills (GOALS) is used to evaluate resident training in laparoscopic cholecystectomy and appendectomy. This study examined the effectiveness of adapting GOALS for laparoscopic hysterectomy in a single residency-training program. Methods: A single-page evaluation form was used for senior resident–attending dyads who were performing laparoscopic hysterectomies together. Senior residents completed self-evaluations and attendings completed trainee evaluations after each case. A 10-item checklist detailed key steps in the operation. Evaluators used the checklist to indicate if trainees completed each step successfully. Five laparoscopic domains (GOALS) and assessment of case difficulty were scored on a 5-point scale. Overall satisfaction with resident performance was rated on a 10-point scale. Results: Over 21 months, 10 residents and 3 attendings completed 97 evaluations. The mean step completion resident score was 62% (attending score: 63%; p = 0.10), indicating agreement between operators and evaluators in case assessments. Compared to attending ratings of resident performances, residents rated themselves lower overall in the 5 laparoscopic domains (3.26 versus 3.53; p < 0.05) despite similar case difficulty estimates (2.66 versus 2.76; p = 0.43). Positive correlations between attending and resident scores indicated consistency between users. Conclusions: GOALS can standardize resident performance evaluation during laparoscopic hysterectomy. There was interevaluator agreement; trainees were more self-critical. An electronic format could make the form easier to use. (J GYNECOL SURG 20XX:000)

Value of the Crowd-Sourced Assessment of Technical Skills (C-SATS) Platform in Surgical Procedures: A Systematic Review of Evidence

Background: Crowd-Sourced Assessment of Technical Skills (C-SATS) is a surgical data management and learning platform that leverages the knowledge of large expert surgeon and lay groups to assess the technique and technical skills of surgeons in a highly efficient manner. The aim of this systematic review was to summarize published literature on the performance of C-SATS as compared to expert evaluations and assess its use as a training and validation tool in minimally invasive surgery (MIS). Methods: A comprehensive literature search was performed per PRISMA guidelines using the Medline, Embase, and Google Scholar databases on published studies that evaluated the use of C-SATS following MIS, such as laparoscopic or robotic-assisted surgery. Results: A total of 21 reports were included in the review. Twelve studies comparing crowd-sourcing evaluations against expert opinion indicated overall excellent or good correlation with Global Operative Assessment of Laparoscopic Skills (GOALS), Global Evaluative Assessment of Robotic Skills (GEARS), and Robotic Objective Structured Assessment of Technical Skills (R-OSATS) scores, with correlation coefficients (Pearson or Spearman) ranging from 0.69 to 0.95 and reliability index (Cronbach’s alpha) from 0.63 to 0.93 across different specialties and surgical approaches. When using C-SATS to assess performance and validation, assessments positively correlated with traditional methods of time and error-based scoring and global rating scale. Conclusions: Based on the current published literature, the C-SATS platform has been shown to efficiently provide crowd-sourced evaluations that correlate favorably with expert evaluation across a range of surgical specialties and approaches. Use of crowdsourcing has uniformly yielded accurate evaluations of surgeons’ technical skills in a markedly shorter time than expert reviews. C-SATS may be a cost-effective complement or alternative to traditional models of evaluating surgical proficiency. Future studies are needed to determine whether the use of C-SATS will lead to improved surgical performance and patient outcomes.

Video Playback as an Educational Tool for Improving Resident Laparoscopic Performance: A Randomized Controlled Trial

Study ObjectiveIn this single-masked randomized controlled study, we evaluate whether watching video recordings of oneself performing Fundamentals of Laparoscopic Skills (FLS) exercises results in an improvement on the 30-point Global Operative Assessment of Laparoscopic Skills (GOALS) assessment among Obstetrics and Gynecology (Ob/Gyn) residents. DesignTwenty-three Ob/Gyn residents in the 2020–2021 academic year completed the FLS exercises while being timed, video recorded, and receiving real-time feedback from an Ob/Gyn faculty member. Baseline GOALS assessment was completed by participants and faculty. After the intervention, all participants then repeated the FLS exercises while being timed and were again scored using the GOALS assessment. Each participant completed the study in a single session. In addition, all participants completed a pre- and post-test survey. SettingUniversity of Louisville Laparoscopic Skills Labortaory. ParticipantsUniversity of Louisville Ob/Gyn residents in the 2020–2021 academic year. InterventionTwelve participants were randomized to the intervention and were allowed to watch their video recording in addition to receiving verbal feedback whereas the remaining 11 received verbal feedback only. Measurements and Main ResultsThere were significant improvements in faculty (p <.01) and self-reported GOALS scores (p <.01) when comparing both the intervention and control group with baseline scores. The intervention group improved by 3.2 points more than that of the control group based on masked faculty evaluation (95% confidence interval, 1.4–5.0 points; p <.01). The difference was not significant in resident self-scores. Overall time improved for all participants (15:54 ± 0.21 minutes before and 13:13 ± 0.14 minutes after), but this difference was not significant between the 2 groups. Higher postgraduate year (PGY) residents reported significantly more comfort performing laparoscopic tasks, earned higher GOALS score (faculty and self-scores), and completed the first set of exercises in less time. A significant interaction between PGY and intervention was detected with improvement in GOALS score most strongly associated with PGY4 residents (p <.01). ConclusionAlthough all learners objectively benefit from watching recordings of themselves performing surgical tasks, advanced learners may benefit the most when video recordings are used as an educational tool.

282. VALIDITY AND RELIABILITY OF CROWD-SOURCED AND EXPERT VIDEO ASSESSMENT IN MINIMALLY INVASIVE ESOPHAGECTOMY USING GLOBAL RATING SCALES

Abstract Minimally Invasive Esophagectomy (MIE) is a complex and technically demanding procedure and implementation is associated with a substantial learning curve. Suboptimal surgical performance in complex minimally invasive procedures has convincingly been related to less favorable patient outcomes. Structured video-based assessment by other surgeons can objectively evaluate surgical performance, but requires a surgeon’s scarce time. This study investigated whether crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, could be a valid alternative for MIE. Two video assessments types were conducted via the C-SATS platform with eight MIE operative videos. First, videos were cut into eight procedural phase-based video clips of maximum ten minutes. These 64 video clips were assessed with the validated Global Operative Assessment of Laparoscopic Skills (GOALS)‘s four domains (depth perception, bimanual dexterity, efficiency and tissue handling) using a 1-5 Likert scale. Each video clip was assessed by a pair of two MIE experts and &amp;gt; 30 crowd workers, recruited via Amazon Mechanical Turk. Second, the same expert pairs assessed the corresponding complete video with the procedure-specific MIE competency assessment tool (MIE-CAT). Experts assessed 128 clips and sixteen complete videos in 3.5 months. Crowd workers assessed 1984 clips in &amp;lt; two days. Expert scores showed a wider range (expert GOALS score range 7.50-19.50 versus 12.84-15.63 for crowds). When comparing experts and crowd workers scores, a poor correlation was found on all domains and total GOALS scores (ICC ≤ 0.42). When correlating GOALS scores with MIE-CAT scores a strong, positive correlation between experts’ scores was found (r¬s = 0.76, p = 0.03) while between crowd’s GOALS and expert’s MIE-CAT scores a small correlation was found (r¬s = 0.23, p = 0.58) (Figure 1). Despite rapid results from crowd workers, a poor agreement between crowd and experts’ GOALS scores was observed. MIE might be technically too difficult to assess for laypeople. Furthermore, GOALS might not be extensive enough to assess MIE performance in detail, while the procedure-specific MIE-CAT might be more comprehensive. However, the correlation between the experts’ MIE-CAT and GOALS scores indicates video clip assessments (instead of full length video assessments) could be helpful to shorten assessment time.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy.

IntroductionSelf-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.Methods and proceduresUsing a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1–3] and senior trainees (PGY 4–5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.ResultsA total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.ConclusionsOPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00464-022-09466-6.

Assessment of laparoscopic skills: comparing the reliability of global rating and entrustability tools

Competence by design (CBD) residency programs increasingly depend on tools that provide reliable assessments, require minimal rater training, and measure progression through the CBD milestones. To assess intraoperative skills, global rating scales and entrustability ratings are commonly used but may require extensive training. The Competency Continuum (CC) is a CBD framework that may be used as an assessment tool to assess laparoscopic skills. The study aimed to compare the CC to two other assessment tools: the Global Operative Assessment of Laparoscopic Skills (GOALS) and the Zwisch scale. Four expert surgeons rated thirty laparoscopic cholecystectomy videos. Two raters used the GOALS scale while the remaining two raters used both the Zwisch scale and CC. Each rater received scale-specific training. Descriptive statistics, inter-rater reliabilities (IRR), and Pearson's correlations were calculated for each scale. Significant positive correlations between GOALS and Zwisch (r = 0.75, p < 0.001), CC and GOALS (r = 0.79, p < 0.001), and CC and Zwisch (r = 0.90, p < 0.001) were found. The CC had an inter-rater reliability of 0.74 whereas the GOALS and Zwisch scales had inter-rater reliabilities of 0.44 and 0.43, respectively. Compared to GOALS and Zwisch scales, the CC had the highest inter-rater reliability and required minimal rater training to achieve reliable scores. The CC may be a reliable tool to assess intraoperative laparoscopic skills and provide trainees with formative feedback relevant to the CBD milestones. Further research should collect further validity evidence for the use of the CC as an independent assessment tool.

The impact of virtual reality simulation training on operative performance in laparoscopic cholecystectomy: meta-analysis of randomized clinical trials.

BackgroundSimulation training can improve the learning curve of surgical trainees. This research aimed to systematically review randomized clinical trials (RCT) evaluating the performance of junior surgical trainees following virtual reality training (VRT) and other training methods in laparoscopic cholecystectomy.MethodsMEDLINE (PubMed), Embase (Ovid SP), Web of Science, Scopus and LILACS were searched for trials randomizing participants to VRT or no additional training (NAT) or simulation training (ST). Outcomes of interest were the reported performance using global rating scores (GRS), the Objective Structured Assessment of Technical Skill (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS), error counts and time to completion of task during laparoscopic cholecystectomy on either porcine models or humans. Study quality was assessed using the Cochrane Risk of Bias Tool. PROSPERO ID: CRD42020208499.ResultsA total of 351 titles/abstracts were screened and 96 full texts were reviewed. Eighteen RCT were included and 15 manuscripts had data available for meta-analysis. Thirteen studies compared VRT and NAT, and 4 studies compared VRT and ST. One study compared VRT with NAT and ST and reported GRS only. Meta-analysis showed OSATS score (mean difference (MD) 6.22, 95%CI 3.81 to 8.36, P < 0.001) and time to completion of task (MD −8.35 min, 95%CI 13.10 to 3.60, P = <0.001) significantly improved after VRT compared with NAT. No significant difference was found in GOALS score. No significant differences were found between VRT and ST groups. Intraoperative errors were reported as reduced in VRT groups compared with NAT but were not suitable for meta-analysis.ConclusionMeta-analysis suggests that performance measured by OSATS and time to completion of task is improved with VRT compared with NAT for junior trainee in laparoscopic cholecystectomy. However, conclusions are limited by methodological heterogeneity and more research is needed to quantify the potential benefit to surgical training.

Establishing validity evidence for device-assisted advanced laparoscopic suturing tasks using simulation.

Laparoscopic suturing (LS) is an essential technique required for a wide range of procedures, and it is one of the most challenging for surgical trainees to master. We designed and collected validity evidence for advanced LS tasks using an automated suturing device and evaluated the perceived educational value of the tasks. This project was a multicentre prospective study involving McGill University, University of Toronto (UofT), and Louisiana State University (LSU) Health New Orleans. Novice (NS) and experienced (ES) surgeons performed suturing under tension (UT) and continuous suturing (CS) tasks. ES performed the tasks twice to establish proficiency benchmarks, and they were interviewed to develop formative feedback tools (FFT). Participants were assessed on completion time, error, Global Operative Assessment of Laparoscopic Skills (GOALS), and FFT. Data were analyzed using descriptive and inferential statistical methods. Twenty-seven participants (13 ES, 14 NS, median age 34years; 85% male) completed the study. Eight were attending surgeons, 7 fellows, 4 PGY5, 5 PGY4, and 3 PGY3 (18 from McGill, 5 UofT, and 4 LSU). Comparing ES and NS, for UT task, ES had significantly greater task scores (570 [563-648] vs 323 [130-464], p value 0.00036) and GOALS scores (14 [13-16] vs 10 [8-12], p value 0.0038). Similarly, for CS, ES had significantly greater task scores (976 [959-1010] vs 785 [626-856], p value 0.00009) and GOALS scores (16 [12-17] vs 12.5 [8.25-15], p value 0.028). After FFTs were developed, comparing ES and NS, for both UT and CS tasks, ES had significantly greater FFT scores (UT 25 [24-26] vs 17 [14-20], p value 0.0016 and CS 30 [27-32] vs 22[17.2-25.8], p value 0.00061). In conclusion, preliminary validity evidence was provided for the tasks. Once further validity evidence is established, incorporating the tasks into the training curricula could improve trainee skills and help to meet the need for better advanced suturing models.

Pilot Evaluation of a Novel, Low-Cost, Simulation Model for Training and Assessment of Laparoscopic Intracorporeal Continuous Suturing.

Background: Laparoscopic intracorporeal continuous suturing is being employed in a growing number of minimally invasive procedures. However, there is a lack of adequate bench models for gaining proficiency in this complex task. The purpose of this study was to assess a novel simulation model for running suture. Methods: Participants were grouped as novice (LSN) or expert (LSE) at laparoscopic suturing based on prior experience and training level. A novel low-cost bench model was developed to simulate laparoscopic intracorporeal continuous closure of a defect. The primary outcome measured was time taken to complete the task. Videos were scored by independent raters for Global Operative Assessment of Laparoscopic Skills (GOALS). Results: Sixteen subjects (7 LSE and 9 LSN) participated in this study. LSE completed the task significantly faster than LSN (430 ± 107 vs 637 ± 164seconds, P ≤ .05). LSN scored higher on accuracy penalties than LSE (Median 30 vs 0, P ≤ .05). Mean GOALS score was significantly different between the 2 groups (LSE 20.64 ± 2.64 vs LSN 14.28 ± 1.94, P < .001) with good inter-rater reliability (ICC ≥ .823). An aggregate score using the formula: Performance Score = 1200-time(sec)-(accuracy penalties x 10) was significantly different between groups with a mean score of 741 ± 141 for LSE vs 285 ± 167 for LSN (P < .001). Conclusion A novel bench model for laparoscopic continuous suturing was able to significantly discriminate between laparoscopic experts and novices. This low-cost model may be useful for both training and assessment of laparoscopic continuous suturing proficiency.