Assessment Of Technical Skills Research Articles

Minimally trained analysts can perform fast, objective assessment of orthopedic technical skill from fluoroscopic images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive and easily repeatable assessment strategy without bias. Despite the advantages, surgeon mentors generally refrain from using the scores of images obtained throughout an operation to evaluate a performer’s technical skill because of other practical constraints. A new system was designed to rapidly facilitate objective analysis of fluoroscopy using minimally trained analysts. Four expert and four novice analysts independently measured one objective metric of orthopedic technical skill using both a custom analysis software and a commercial alternative. Novice analysts using the custom software produced measurements three times faster than experts using the commercial software without a practical difference in annotation error. Expert analysts leveraged the custom software automation features to further improve their measurements. These results suggest that a well-designed fluoroscopy analysis system can facilitate regular, practical, and objective assessment of orthopedic technical skill.

Overcoming Gender Bias in Assessment of Surgical Skills

Female and male surgical residents are treated differently based on their gender. The use of assessment tools can help obtain objectivity in surgical skills assessment, avoid gender bias, and promote equal learning opportunities. The objective of the study was to explore whether knowledge of gender causes bias in the rating of surgical skills, and whether the raters' gender, surgical specialty, or experience affect ratings. The study is designed as a cross-sectional study, where an anonymous video showing surgical performance was rated by surgeons from different surgical specialties and different levels of surgical experience. The same video was presented to two randomized groups as either a male or female surgical trainee performing the procedure. The participants used Objective Structured Assessment of Technical Skills (OSATS) to rate a video-recorded exam from a surgical skills course at Copenhagen Academy for Medical Education and Simulation (CAMES). The video rating was done through an internet-based platform and could thus be performed at home or at another location of choice. Novice and experienced surgeons from departments of gastrointestinal surgery, gynaecology, and urology in Denmark. There was no difference in OSATS rating score of the perceived female and male trainee (female trainee (n = 50):12.26 (SD = 3.08), male trainee (n = 57): 12.00 (SD = 3.28), p-value = 0.67). Rater characteristics: gender, age, surgical experience, and surgical specialty did not affect ratings. When OSATS a systematic assessment tool was used in rating surgical trainees we did not find any significant implicit gender bias. The rating was unaffected by the raters' gender, age, surgical specialty, or experience.

Pilot study for low-cost model validation in laparoscopic pediatric pyeloplasty simulation.

To describe the creation of an original 3D-printed liquid latex model designed for laparoscopic pyeloplasty (LP) simulation in infants, and to assess its usefulness. A 3D model of a dilated pelvis and a ureter with ureteropelvic junction obstruction (UPJO) was designed. It was covered in liquid latex, which allowed flexible models to be achieved in order to conduct pyeloplasty in a pelvitrainer. The total price of each model was 6 euros. A nearly-experimental, non-randomized, blind study was carried out, while measuring operating times and OSATS (Objective Structured Assessment of Technical Skills) scores. Following simulation completion, a survey based on Likert scale was conducted to assess overall appearance, texture, usefulness, and probability of recommending the model for regular training. 8 pediatric surgeons spent a median of 71.5 minutes (range: 50-86), and rated the model with a median 20.1/30 (range: 17-24) OSATS score. The model received a 4.25 (range: 3-5) score in terms of overall appearance, a 4.37 (range: 3-5) score in terms of texture, a 4.5 (range: 4-5) score in terms of usefulness, and a 4.6 (range: 4-5) score in terms of probability of recommending the model for regular training. Our liquid latex model for laparoscopic pyeloplasty simulation is feasible, with favorable preliminary results. Its usefulness in laparoscopic pyeloplasty training is promising.

Video Grading of the Pancreatico-Jejunostomy in Robotic Pancreatoduodenectomy to Determine the Learning Curve and the Risk of Postoperative Pancreatic Fistula

Introduction: Video grading may be used to assess the surgical learning curve and could predict complications. Predicting the risk of postoperative pancreatic fistula (POPF) could guide treatment, quality control, or be used for benchmarking however a cut-off variable regarding surgical expertise is lacking. We graded the pancreatico-jejunostomy of robotic pancreatoduodenectomy (RPD) to assess learning curve and the risk of POPF. Methods: We included patients who underwent RPD for all indications from a recently completed multicenter training program in RPD (LAELAPS-3). Technical performance was graded by two blinded graders using the Objective Structured Assessment of Technical Skills (OSATS) score, ranging from 12-60. Cumulative sum (CUSUM) analysis was used to assess a learning curve inflection point. Logistic regression was used to determine the cut-off OSATS score to predict the increased chance of POPF grade B/C. Results: Videos of 153 patients from four centers were assessed. POPF B/C occurred in 35 patients (22.9%). Based on uaFRS criteria, 72 (47.1%) patients had intermediate or high-risk anastomoses. Median OSATS was 48 [interquartile range 41–52]. CUSUM analysis of OSATS found an inflection point after 33 procedures. An OSATS score below 49 was associated with a higher risk of POPF, OR 2.27 (1.04–4.99, p=0.041). Conclusion: This multicenter study demonstrates that video grading of the pancreatico-jejunostomy can determine both the learning curve and the risk of POPF after RPD.

A cadaveric model for transesophageal echocardiography transducer placement training: A pilot study.

Transesophageal echocardiography (TEE) is used in the emergency department to guide resuscitation during cardiac arrest. Insertion of a TEE transducer requires manual skill and experience, yet in some residency programs cardiac arrest is uncommon, so some physicians may lack the means to acquire the manual skills to perform TEE in clinical practice. For other infrequently performed procedural skills, simulation models are used. However, there is currently no model that adequately simulates TEE transducer insertion. The aim of this study is to evaluate the feasibility and efficacy of using a cadaveric model to teach TEE transducer placement among novice users. A convenience sample of emergency medicine residents was enrolled during a procedure education session using cadavers as tissue models. A pre-session assessment was used to determine prior knowledge and confidence regarding TEE manipulation. Participants subsequently attended a didactic and hands-on education session on TEE placement. All participants practised placing the TEE transducer until they were able to pass a standardized assessment of technical skill (SATS). After the educational session, participants completed a post-session assessment. Twenty-five residents participated in the training session. Mean assessment of knowledge improved from 6.2/10 to 8.7/10 (95% confidence interval [CI] of knowledge difference 1.6-3.2, P<0.001) and confidence improved from 1.6/5 to 3.1/5 (95% CI of confidence difference 1.1-2.0, P<0.001). There was no relationship between training level and the delta in knowledge or confidence. In this pilot study, the use of a cadaveric model to teach TEE transducer placement methods among novice users is feasible and improves both TEE manipulation knowledge and confidence levels.

Objective Evaluation of Clock Face Suture Using the Objective Structured Assessment of Technical Skill (OSATS) Checklist

Assessment of the quality of the final product (QFP) is critical in simulation training, such as the clock face suture (CFS) exercise that is used to assess trainees' needle handling and suturing accuracy. Objective Structured Assessment of Technical Skill (OSATS) scores are the gold standard for the evaluation of trainees. The aim was to investigate variability in the use of OSATS checklists and to evaluate a semi-automatic method of suture analysis vs. OSATS scores. Details of 287 CFSs performed by trainees during Fundamentals in Vascular Surgery examinations were collected. All were rated according to a seven item OSATS checklist, including QFP score and an overall score by one or two expert surgeons immediately after completion. Interassessor variability was assessed for the CFS that were assessed by two assessors.In order to assess intra- and interassessor variability, 50 CFS pictures were chosen randomly and submitted to three expert surgeons to rate the QFP twice and to carry out a semi-automatic image analysis of each CFS and the estimated cumulative error (CE; mm) recorded. It was hypothesised that the CE correlates to OSATS checklist items or overall score. Variables were compared for correlation with OSATS results using a linear regression. A Pearson's test was used to confirm the proposed hypothesis. Mean ± standard deviation overall score for the OSATS checklist was 20.61 ± 6.33. Inter- and intra-assessor correlation were statistically significant regarding OSATS checklist items. Both correlations presented a low coefficient of determination, indicating variability. The mean CE was 16.07 ± 4.84 mm, and the correlation between the QFP and CE was statistically significant, proving that CE is an objective metric by which to assess the QFP. OSATS score demonstrated intra- and interassessor variability, although there was a significant correlation between scores. CE is an objective metric that is not subject to assessor subjectivity or interassessor variability and is correlated with the gold standard of evaluation.

Building and training a low-cost and portable vascular anastomosis simulator: Initial experience of a surgical resident

Introduction and objectivesSeveral factors have caused the operating room to no longer be the ideal place for early surgical training, forcing the search for more effective options for simulation-based surgical learning. The objective of our study was to design and build a home-made proposal for a vascular anastomosis simulator for surgical training. MethodsDescriptive, preliminary, observational, analytical, and prospective study. The study was divided into two phases, the simulator construction with low-cost common materials (9.75$), and its operative evaluation with the Objective Structured Assessment of Technical Skills (OSATS-Modified). The study subject was a cardiothoracic surgery junior resident (postgraduate year two, PGY-2) who used the simulator and was evaluated through recorded sessions in 2 consecutive 3-month periods (first period, FP vs. second period, SP) by a senior surgeon. The acquisition of operative skills in the creation of end-to-side (E-T-S), side-to-side (S-T-S), end-to-end (E-T-E) vascular anastomoses were evaluated. ResultsThere were 273 sessions with a total of 4904min of simulation training between the three anastomosis modalities. A significant improvement was identified in the final OSATS-Modified scores (FP vs. SP) in the use of the Castroviejo needle holder, needle angles, and needle transfer in the E-T-S anastomosis (p<.05); graft orientation, appropriate spacing, use of Castroviejo needle holder, and the needle angles in the S-T-S (p<.05), and in the use of the Castroviejo needle holder, and needle transfer in the E-T-E (p<.05). Significant improvement was reported in E-T-S anastomosis time and S-T-S (p<.05). ConclusionsOur home-made portable prototype has proven to be very efficient in improving operational skills on a smaller scale; however, our findings need to be extrapolated to a larger population of surgical residents. Current evidence suggests that skills acquired through simulation-based training are transferred and positively impact the surgical environment.

Executive summary of the artificial intelligence in surgery series

As opportunities for artificial intelligence to augment surgical care expand, the accompanying surge in published literature has generated both substantial enthusiasm and grave concern regarding the safety and efficacy of artificial intelligence in surgery. For surgeons and surgical data scientists, it is increasingly important to understand the state-of-the-art, recognize knowledge and technology gaps, and critically evaluate the deluge of literature accordingly. This article summarizes the experiences and perspectives of a global, multi-disciplinary group of experts who have faced development and implementation challenges, overcome them, and produced incipient evidence thereof. Collectively, evidence suggests that artificial intelligence has the potential to augment surgeons via decision-support, technical skill assessment, and the semi-autonomous performance of tasks ranging from resource allocation to patching foregut defects. Most applications remain in preclinical phases. As technologies and their implementations improve and positive evidence accumulates, surgeons will face professional imperatives to lead the safe, effective clinical implementation of artificial intelligence in surgery. Substantial challenges remain; recent progress in using artificial intelligence to achieve performance advantages in surgery suggests that remaining challenges can and will be overcome.

Performance with robotic surgery versus 3D- and 2D­laparoscopy during pancreatic and biliary anastomoses in a biotissue model: pooled analysis of two randomized trials

BackgroundRobotic surgery may improve surgical performance during minimally invasive pancreatoduodenectomy as compared to 3D- and 2D-laparoscopy but comparative studies are lacking. This study assessed the impact of robotic surgery versus 3D- and 2D-laparoscopy on surgical performance and operative time using a standardized biotissue model for pancreatico- and hepatico-jejunostomy using pooled data from two randomized controlled crossover trials (RCTs).MethodsPooled analysis of data from two RCTs with 60 participants (36 surgeons, 24 residents) from 11 countries (December 2017–July 2019) was conducted. Each included participant completed two pancreatico- and two hepatico-jejunostomies in biotissue using 3D-robotic surgery, 3D-laparoscopy, or 2D-laparoscopy. Primary outcomes were the objective structured assessment of technical skills (OSATS: 12–60) rating, scored by observers blinded for 3D/2D and the operative time required to complete both anastomoses. Sensitivity analysis excluded participants with excess experience compared to others.ResultsA total of 220 anastomoses were completed (robotic 80, 3D-laparoscopy 70, 2D­laparoscopy 70). Participants in the robotic group had less surgical experience [median 1 (0–2) versus 6 years (4–12), p < 0.001], as compared to the laparoscopic group. Robotic surgery resulted in higher OSATS ratings (50, 43, 39 points, p = .021 and p < .001) and shorter operative time (56.5, 65.0, 81.5 min, p = .055 and p < .001), as compared to 3D- and 2D­laparoscopy, respectively, which remained in the sensitivity analysis.ConclusionIn a pooled analysis of two RCTs in a biotissue model, robotic surgery resulted in better surgical performance scores and shorter operative time for biotissue pancreatic and biliary anastomoses, as compared to 3D- and 2D-laparoscopy.

A high-fidelity, virtual-reality, transurethral resection of bladder tumor simulator: Validation as a tool for training.

Simulation-based training is used to help trainees learn surgical procedures in a safe environment. The objective of our study was to test the face, content, and construct validity of the transurethral resection of bladder tumor (TURBT) module built on the Simbionix TURP Mentor simulator. Participants performed five standardized cases on the simulator. Domains of the simulator were evaluated on a five-point Likert scale to establish face and content validity. Construct validity was assessed through the simulator's built-in scoring metrics, as well as video recordings of the simulator screen and an anonymized view of participants' hands and feet, which were evaluated using an objective structured assessment of technical skills (OSATS) tool. Ten experienced operators and 15 novices participated. Face validity was somewhat acceptable (mean realism 3.8/5±1.03 standard deviation [SD]; mean appearance 4.1/5±0.57), as was content validity, represented by simulation of key steps (mean 3.9±0.57). The simulator failed to achieve construct validity. There was no difference in mean simulator scores or OSATS scoring between experienced operators and novices. Novices significantly improved their mean simulator scores (305.9 vs. 332.4, p=0.006) and OSATS scoring (15.8 vs. 18.1, p=0.001), while 87% felt their confidence to perform TURBT improved. Overall, 92% of participants agreed that the simulator should be incorporated into residency training. Our study suggests a role for the TURBT module of the Simbionix TURP Mentor simulator as an introduction to TURBT for urology trainees. Strong support was found from both experienced operators and novices for its formal inclusion in resident education.

A cost-effective IoT learning environment for the training and assessment of surgical technical skills with visual learning analytics

BackgroundSurgeons need to train and certify their technical skills. This is usually done with the intervention of experts who monitor and assess trainees. Nevertheless, this is a time-consuming task that is subject to variations among evaluators. In recent decades, subjectivity has been significantly reduced through 1) the introduction of standard curricula, such as the Fundamentals of Laparoscopic Surgery (FLS) program, which measures students’ performance in specific exercises, and 2) rubrics, which are widely accepted in the literature and serve to provide feedback about the overall technical skills of the trainees. Although these two elements reduce subjectivity, they do not, however, eliminate the figure of the expert evaluator, and so the process remains time consuming. ObjectivesThe objective of this work is to automate those parts of the work of the expert evaluator that the technology can measure objectively, using sensors to collect evidence, and visualizations to provide feedback. We designed and developed 1) a cost-effective IoT (Internet of Things) learning environment for the training and assessment of surgical technical skills and 2) visualizations supported by the literature on visual learning analytics (VLA) to provide feedback about the exercises (in real time) and overall performance (at the end of the training) of the trainee. MethodsA hybrid approach was followed based on previous research for the design of the sensor based IoT learning environment. Previous studies were used as the basis for getting best practices on the tracking of surgical instruments and on the detection of the force applied to the tissue, with a focus on reducing the costs of data collection. The monitoring of the specific exercises required the design of sensors and collection mechanisms from scratch as there is little existing research on this subject. Moreover, it was necessary to design the overall architecture to collect, process, synchronize and communicate the data coming from the different sensors to provide high-level information relevant to the end user. The information to be presented was already validated by the literature and the focus was on how to visualize this information and the optimal time for its presentation to end users. The visualizations were validated with 18 VLA experts assessing the technical aspects of the visualizations and 4 medical experts assessing their functional aspects. ResultsThis IoT learning environment amplifies the evaluation mechanisms already validated by the literature, allowing automatic data collection. First, it uses IoT sensors to automatically correct two of the exercises defined in the FLS (peg transfer and precision cutting), providing real-time visualizations. Second it monitors the movement of the surgical instruments and the force applied to the tissues during the exercise, computing 6 of the high-level indicators used by expert evaluators in their rubrics (efficiency, economy of movement, hand tremor, depth perception, bimanual dexterity, and respect for tissue), providing feedback about the technical skills of the trainee using a radar chart with these six indicators at the end of the training (summative visualizations). ConclusionsThe proposed IoT learning environment is a promising and cost-effective alternative to help in the training and assessment of surgical technical skills. The system shows the trainees’ progress and presents new indicators about the correctness of each specific exercise through real-time visualizations, as well as their general technical skills through summative visualizations, aligned with the 6 more frequent indicators in standardized scales. Early results suggest that although both types of visualizations are useful, it is necessary to reduce the cognitive load of the graphs presented in real time during training. Nevertheless, an additional evaluation is needed to confirm these results.

Effect of Simulation-based Training on Surgical Proficiency and Patient Outcomes: A Randomised Controlled Clinical and Educational Trial

BackgroundIt is hypothesised that simulation enhances progression along the initial phase of the surgical learning curve. ObjectiveTo evaluate whether residents undergoing additional simulation, compared to conventional training, are able to achieve proficiency sooner with better patient outcomes. Design, setting, and participantsThis international, multicentre, randomised controlled trial recruited 94 urology residents with experience of zero to ten procedures and no prior exposure to simulation in ureterorenoscopy, selected as an index procedure. InterventionParticipants were randomised to simulation or conventional operating room training, as is the current standard globally, and followed for 25 procedures or over 18 mo. Outcome measurements and statistical analysisThe number of procedures required to achieve proficiency, defined as achieving a score of ≥28 on the Objective Structured Assessment of Technical Skill (OSATS) scale over three consecutive operations, was measured. Surgical complications were evaluated as a key secondary outcome. This trial is registered at www.isrctn.com as ISCRTN 12260261. Results and limitationsA total of 1140 cases were performed by 65 participants, with proficiency achieved by 21 simulation and 18 conventional participants over a median of eight and nine procedures, respectively (hazard ratio [HR] 1.41, 95% confidence interval [CI] 0.72–2.75). More participants reached proficiency in the simulation arm in flexible ureterorenoscopy, requiring a lower number of procedures (HR 0.89, 95% CI 0.39–2.02). Significant differences were observed in overall comparison of OSATS scores between the groups (mean difference 1.42, 95% CI 0.91–1.92; p < 0.001), with fewer total complications (15 vs 37; p = 0.003) and ureteric injuries (3 vs 9; p < 0.001) in the simulation group. ConclusionsAlthough the number of procedures required to reach proficiency was similar, simulation-based training led to higher overall proficiency scores than for conventional training. Fewer procedures were required to achieve proficiency in the complex form of the index procedure, with fewer serious complications overall. Patient summaryThis study investigated the effect of simulation training in junior surgeons and found that it may improve performance in real operating settings and reduce surgical complications for complex procedures.

Technical Skill Impacts the Success of Sequential Robotic Suturing Substeps.

Introduction: Robotic surgical performance, in particular suturing, has been associated with postoperative clinical outcomes. Suturing can be deconstructed into substep components (needle positioning, needle entry angle, needle driving, and needle withdrawal) allowing for the provision of more specific feedback while teaching suturing and more precision when evaluating suturing technical skill and prediction of clinical outcomes. This study evaluates if the technical skill required for particular substeps of the suturing process is associated with the execution of subsequent substeps in terms of technical skill, accuracy, and efficiency. Materials and Methods: Training and expert surgeons completed standardized sutures on the Mimic™ Flex virtual reality robotic simulator. Video recordings were deidentified, time annotated, and provided technical skill scores for each of the four suturing substeps. Hierarchical Poisson regression with generalized estimating equation was used to examine the association of technical skill rating categories between substeps. Results: Twenty-two surgeons completed 428 suturing attempts with 1669 individual technical skill assessments made. Technical skill scores between substeps of the suturing process were found to be significantly associated. When needle positioning was ideal, needle entry angle was associated with a significantly greater chance of being ideal (risk ratio [RR] = 1.12, p = 0.05). In addition, ideal needle entry angle and needle driving technical skill scores were each significantly associated with ideal needle withdrawal technical skill scores (RR = 1.27, p = 0.03; RR = 1.3, p = 0.03, respectively). Our study determined that ideal technical skill was associated with increased accuracy and efficiency of select substeps. Conclusions: Our study found significant associations in the technical skill required for completing substeps of suturing, demonstrating inter-relationships within the suturing process. Together with the known association between technical skill and clinical outcomes, training surgeons should focus on mastering not just the overall suturing process, but also each substep involved. Future machine learning efforts can better evaluate suturing, knowing that these inter-relationships exist.

Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score

IntroductionThe aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS).Materials and methodsThe Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8).ResultsThe Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807–0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923–0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale.ConclusionWith the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Learning and application of intracorporal slipping knot techniques in minimally invasive surgery

AbstractAimIntracorporal knot tying (ICKT) and suturing in minimally invasive surgery (MIS) are key skills for advanced procedures. The best choice for an intracorporal slipping knot tying technique had not been defined. The aim of this study was to compare two intracorporal slipping knot techniques: the classical C‐loop technique (Variant I) and the square‐to‐slip technique without changing leadership hand (Variant II).MethodsA laparoscopic box trainer including laparoscope was used for ICKT. A total of 120 slipping knots were evaluated in two groups with different levels of surgical education. The first group (n = 5) consisted of senior surgical physicians proficient in laparoscopic surgery. The second group (n = 10) comprised medical students without any prior experience in laparoscopic surgery. The medical student group received a 1‐hour hands‐on training session. Participants were assessed using the Global Rating Scale of the Objective Structured Assessment of Technical Skills, procedural implementation, knot quality, and task time.ResultsIn the medical student group, performance in all parameters was greater for Variant I compared with Variant II (P &lt; .005). Contrary to the students group, senior physicians demonstrated significantly faster task time in Variant II compared with Variant I (P = .001). No significant differences were observed in the remaining parameters.ConclusionFor novices, the intracorporal slipping knot is easier to learn with the C‐loop technique (Variant I) than with the square‐to‐slip technique (Variant II). For surgeons experienced in MIS, the square‐to‐slip technique without changing leadership hand (Variant II) was superior to the slipping knot in the C‐loop technique (Variant I) only by time savings. The knot quality, as the most important variable, showed equally good results between both ICKT techniques.

SP2.1.1Continuous Monitoring and Assessment of Surgical Technical Skills Using Deep Learning

Abstract Aims Excellent surgical technical skills are of paramount importance to perform surgical procedures, safely and efficiently. Virtual reality surgical simulators can both simulate real operations while providing standardized, risk-free surgical hands-on experience. The integration of AI (artificial intelligence) and virtual reality simulators provides opportunities to carry out comprehensive continuous assessments of surgical performance. We developed and tested a deep learning algorithm which can continuously monitor and assess surgical bimanual performance on virtual reality surgical simulators. Methods Fifty participants from four expertise levels (14 experts/neurosurgeons, 14 senior residents, 10 junior residents, 12 medical students) performed a simulated subpial tumor resection 5 times and a complex simulated brain tumor operation once on the NeuroVR platform. Participants were asked to remove the tumors completely while minimizing bleeding and damage to surrounding tissues employing a simulated ultrasonic aspirator and bipolar forceps. A deep neural network continually tracked the surgical performance utilizing 16 performance metrics generated every 0.2-seconds. Results The deep neural network was successfully trained using neurosurgeons and medical students’ data, learning the composites of expertise comparing high and lower skill levels. The trained algorithm was able to score the technical skills of individuals continuously at 0.2-second intervals. Statistically significant differences in average scores were identified between the 4 groups. Conclusions AI-powered surgical simulators provide continuous assessment of bimanual technical skills during surgery which may further define the composites necessary to train surgical expertise. To our knowledge, this is the first attempt in surgery to continuously assess surgical technical skills using deep learning.

EP.TU.566Virtual Interactive surgical skills classroom – An Ongoing Randomized Controlled Trial

IntroductionHigh costs and inaccessibility are significant barriers to face-to-face basic surgical skills (BSS) training. Virtual classrooms enable the combination of computer-based learning with interactive expert instruction. They may optimise resources and increase accessibility, facilitating larger-scale training with a similar educational benefit. We aim to evaluate the efficacy of virtual BSS classroom training compared to both non-interactive video and face-to-face teaching.Method72 medical students will be randomly assigned to three equal intervention groups based on year group and surgical skill confidence. Interventions will be implemented following an instructional video. Group A will practice independently, Group B will receive face-to-face training, and Group C will attend a virtual classroom. Participants will be recorded placing three interrupted sutures with hand tied knots pre- and post-intervention, and Objective Structured Assessment of Technical Skills (OSATS) will be blind marked by two experts. Change in confidence, time to completion and a granular performance score will also be measured. Each intervention’s feasibility and accessibility will be assessed.ResultsData collection will be completed in February 2021. Significant improvement in OSATS within groups will be indicative of intervention quality. Difference in improvement between groups will determine the relative performance of the interventions.ConclusionTo our knowledge, this will be the largest randomised control trial investigating virtual BSS classroom training. It will serve as a comprehensive appraisal of the virtual classroom’s suitability as an alternative to face-to-face training. The findings will assist the development of further resource-efficient training programs during the COVID-19 pandemic and afterwards.

EP.TH.589Comparison of Surgeon Gaze Behaviour Against Objective Skill Assessment in Open Inguinal Hernia Repair – A Pilot Study

Abstract Aims Technical skill is associated with improved postoperative outcomes. Adoption of a formalised high-stakes assessment of surgical skill is technically challenging and limited by the financial and human resources available. We aimed to assess the ability to adopt gaze behaviour analysis as an assessment of surgical skill within live open inguinal herniorrhaphy. Methods Surgeons’ gaze was measured with Tobii Pro eye-tracking Glasses 2 (Tobii AB). All grades of surgeons were included. Primary outcomes were dwell time (%) and fixation frequency (count/s), as markers of cognition, on areas of interest correlated to mean Objective Skill Assessment of Technical Skill score. Secondary outcomes assessed effort and concentration levels through maximum pupil diameter (mm) and rate of pupil change (mm/s) correlated to perceived workload (SURG-TLX). Three operative segments underwent analysis: mesh preparation, fixation and muscle closure. Spearman’s and Pearson’s correlation were performed with significance set at p &amp;lt; 0.05. Results 5 cases were analysed, totalling 270 minutes of video footage. All participants were senior surgical trainees and right-hand-dominant. The median number of hernia operations performed was 160 (range:100-500). The median ASA score of each patient participant was 2 (range:1-2). The median operation length was 45 mins (range:40-90 mins). There were no statistically significant primary outcomes from this pilot data (p &amp;gt; 0.05). Conclusions This pilot study demonstrated the feasibility of recording gaze behaviours for comparison against formal skills assessment to determine the role of eye tracking in live high stakes technical skills assessment. A full study will now commence based on formal power calculation.

Procedure-based assessment for laparoscopic cholecystectomy can replace global rating scales

Introduction Global rating scales (GRSs) such as the Objective Structured Assessment of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Surgery (GOALS) are assessment methods for surgical procedures. The aim of this study was to establish construct validity of Procedure-Based Assessment (PBA) and to compare PBA with GRSs for laparoscopic cholecystectomy. Material and methods OSATS and GOALS GRSs were compared with PBA in their ability to discriminate between levels of performance between trainees who can perform the procedure independently and those who cannot. Three groups were formed based on the number of procedures performed by the trainee: novice (1–10), intermediate (11–20) and experienced (>20). Differences between groups were assessed using the Kruskal–Wallis and Mann–Whitney U tests. Results Increasing experience correlated significantly with higher GRSs and PBA scores (all p < .001). Scores of novice and intermediate groups overlapped substantially on the OSATS (p = .1) and GOALS (p = .1), while the PBA discriminated between these groups (p = .03). The median score in the experienced group was higher with less dispersion for PBA (97.2[85.3–100]) compared to OSATS (82.1[60.7–100]) and GOALS (80[60–100]). Conclusion For assessing skill level or the capability of performing a laparoscopic cholecystectomy independently, PBA has a higher discriminative ability compared to the GRSs.

Is Technical Expertise Necessary to Assess Technical Expertise? Let's Ask the Kids.

Background: Simulation has an increasing role in surgical training; however, using validated tools such as the Objective Structured Assessment of Technical Skills (OSATS) is time-consuming, which may be a potential barrier to simulation-based training. This study tests the hypothesis that assessors with technical expertise are necessary to objectively score a technical task. Methods: Three tasks, ring transfer, needle pass, and atresia cut, were performed using a synthetic thoracoscopic simulator for esophageal atresia/tracheoesophageal fistula. Three pediatric surgeons, 6 novice adults, and 3 children aged 9-13 years scored each attempt using the overall global OSATS rating from 1 to 5 (1 "repeated or awkward movements" to 5 "fluid movement. No awkwardness"). Results: For the ring transfer, all assessors scored novice participants significantly less than expert and intermediate scores (surgeon P = .0004, nonsurgeon adults P = .0009 and children P = .0003). For the needle pass, all assessors gave significantly different scores between novices and experts (surgeon P = .0007, nonsurgeon adult P = .0008, and children P = .0040). For the atresia cut, surgeon assessors gave significantly higher scores for experts and intermediate and novice (P = .0004). Nonsurgeon assessors gave experts and intermediates significantly higher scores than novices (P = .0001). Surgeon assessors achieved good reliability for ring transfer (0.8252) and needle pass (0.7769) compared with nonsurgeon assessors who showed poor reliability for the ring transfer (0.3959) and moderate for the needle pass (0.6551). Conclusions: Expertise in performing these procedures is not a prerequisite for an assessor to evaluate the technical skill, hence assessors of skill acquisition can be nonexpert, a nonsurgeon, or even a child. The variability in all groups suggests that reliability overall is increased with multiple assessors. Although nonsurgeon assessors may be appropriate for formative assessments, they lack the reliability to provide assessment of competence for high stakes complex tasks. Summative assessment will likely require at least 1 surgeon/expert assessor to provide reliability.