Assessment Of Technical Skills Research Articles

PD01-10 USING SURGICAL GESTURES TO BUILD EXPLAINABLE ARTIFICIAL INTELLIGENCE FOR SURGICAL SKILLS ASSESSMENT

You have accessJournal of UrologyCME1 Apr 2023PD01-10 USING SURGICAL GESTURES TO BUILD EXPLAINABLE ARTIFICIAL INTELLIGENCE FOR SURGICAL SKILLS ASSESSMENT Mitchell G Goldenberg, Runzhuo Ma, Elyssa Y. Wong, Timothy N. Chu, Christian Wagner, and Andrew J. Hung Mitchell G GoldenbergMitchell G Goldenberg More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Timothy N. ChuTimothy N. Chu More articles by this author , Christian WagnerChristian Wagner More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003218.10AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Traditional assessment of surgical performance is based on the evaluation of human experts. Artificial intelligence (AI)-based assessments enhance the efficiency and scalability of these assessments but lack transparency in how these scores are calculated. Deconstructing surgical actions into their most basic movements may improve the objectivity and explainability of these assessments. We sought to understand whether a surgical gestures-trained AI algorithm can predict validated measures of technical skill, as a first step toward transparency in automated surgical skills assessment. METHODS: Data was prospectively collected from two international institutions. Videos of the nerve-spare (NS) step of Robotic-assisted radical prostatectomy (RARP) cases were blindly analyzed by trained human raters using the validated Dissection Assessment for Robotic Technique (DART) tool, and surgical gestures based on a previously published classification were separately annotated (Figure 1a). Surgeon and patient demographics for each case were abstracted. Spearman’s rank correlation was used to identify significant associations between DART skill domains and the proportion of surgical gestures used. The cohort was divided 80:20 for training and validation, and an interpretable recurrent neural network (IMV-LSTM) was used to extract information from gesture sequences to predict DART domains. RESULTS: Eighty cases were included in the analysis from 21 surgeons. Median prior experience was 450 cases (IQR 230-2000). A median of 438 discrete gestures (IQR 254-559) was identified per NS case. Grouping gestures by DART skill domains we found multiple positive and negative correlations (Figure 1a), with total DART score significantly associated with proportion of hook and clip gestures (p<.001) (Figure 1b). The neural network was able to predict DART domains, and AUCs for predicting tissue handling, tissue retraction, and efficiency were 0.64, 0.66, and 0.64 respectively (Figure 1c). CONCLUSIONS: Surgical gestures may provide a link between the objectivity and scalability of AI-based technical skill assessments, and the explainability and familiarity of human expert-based ones. As metrics of surgical performance continue to diversify, these data support a multifaceted approach in evaluation of surgical performance. Source of Funding: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA273031. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e67 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Mitchell G Goldenberg More articles by this author Runzhuo Ma More articles by this author Elyssa Y. Wong More articles by this author Timothy N. Chu More articles by this author Christian Wagner More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...

MP68-18 END TO END ASSESSMENT OF SUTURING EXPERTISE (EASE) IN VIRTUAL REALITY SURGICAL SIMULATION WITH ASSOCIATED CLINICAL OUTCOMES

You have accessJournal of UrologyCME1 Apr 2023MP68-18 END TO END ASSESSMENT OF SUTURING EXPERTISE (EASE) IN VIRTUAL REALITY SURGICAL SIMULATION WITH ASSOCIATED CLINICAL OUTCOMES Timothy N. Chu, Runzhuo Ma, Alvin Hui, Oscar Gomez, Cherine H. Yang, Istabraq S. Dalieh, Elyssa Y. Wong, Ahmed Ghazi, John W. Davis, Brian J. Miles, Clayton Lau, and Andrew J. Hung Timothy N. ChuTimothy N. Chu More articles by this author , Runzhuo MaRunzhuo Ma More articles by this author , Alvin HuiAlvin Hui More articles by this author , Oscar GomezOscar Gomez More articles by this author , Cherine H. YangCherine H. Yang More articles by this author , Istabraq S. DaliehIstabraq S. Dalieh More articles by this author , Elyssa Y. WongElyssa Y. Wong More articles by this author , Ahmed GhaziAhmed Ghazi More articles by this author , John W. DavisJohn W. Davis More articles by this author , Brian J. MilesBrian J. Miles More articles by this author , Clayton LauClayton Lau More articles by this author , and Andrew J. HungAndrew J. Hung More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003331.18AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Virtual reality (VR) provides a standardized setting for surgical skill assessment but currently lacks high-level multi-institutional evidence to validate the predictive value of VR. We have previously developed a granular suturing assessment tool (EASE), which can accurately distinguish surgeon experience in live surgery. This present study aims to further validate EASE in a VR environment and evaluate the utility of VR as a tool to predict clinical outcomes. METHODS: 21 surgeons in this prospective five-center study completed VR simulation suturing exercises on the Surgical Science™ Flex VR simulator (Fig 1A). 5 independent and blinded graders received standardized training and provided technical skill assessment for VR using EASE. EASE was adapted to be relevant for a simulated environment (EASE-VR). Inter-rater reliability was measured using prevalence-adjusted bias-adjusted kappa (PABAK) with agreement >0.6 in all graded subskills. Expert (>100 robotic cases) and trainee (<100 cases) scores were compared using generalized estimating equation. Correlations for expert surgeon EASE-VR scores and post-surgical clinical outcomes were calculated using Cox regression while controlling for patient factors (age, BMI, preoperative PSA, biopsy Gleason Score, pathological tumor stage). RESULTS: 7 trainees (median caseload 60, range 0-80) and 14 expert surgeons (1500, range 275-3000) participated in this study. In VR, experts had significantly better Needle Handling (hold anglep=0.048) and Needle Driving (wrist rotation p=0.014) scores compared to trainees. There were no statistically significant differences in all other domains (p>0.05). For expert surgeons, Needle Driving (wrist rotation) was not significantly associated with urinary continence recovery. Expert Needle Handling (hold angle) scores were found to have a significant positive association with urinary continence recovery (Hazard Ratio 2.11, 95% Confidence Interval 1.44-3.09, p<0.001) (Fig 1B). CONCLUSIONS: EASE-VR can distinguish surgical experience in VR simulation exercises. Our results suggest that expert surgeon EASE-VR scores have positive association with post-surgical clinical outcomes. Source of Funding: Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA251579. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e960 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Timothy N. Chu More articles by this author Runzhuo Ma More articles by this author Alvin Hui More articles by this author Oscar Gomez More articles by this author Cherine H. Yang More articles by this author Istabraq S. Dalieh More articles by this author Elyssa Y. Wong More articles by this author Ahmed Ghazi More articles by this author John W. Davis More articles by this author Brian J. Miles More articles by this author Clayton Lau More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...

Surgical skills assessment of pancreaticojejunostomy using a simulator may predict patient outcomes: A multicenter prospective observational study

BackgroundPancreatoduodenectomy, an advanced surgical procedure with a high complication rate, requires surgical skill in performing pancreaticojejunostomy, which correlates with operative outcomes. We aimed to analyze the correlation between pancreaticojejunostomy assessment conducted in a simulator environment and the operating room and patient clinical outcomes. MethodsWe recruited 30 surgeons (with different experience levels in pancreatoduodenectomy) from 11 institutes. Three trained blinded raters assessed the videos of the pancreaticojejunostomy procedure performed in the operating room using a simulator according to an objective structured assessment of technical skill and a newly developed pancreaticojejunostomy assessment scale. The correlations between the assessment score of the pancreaticojejunostomy performed in the operating room and using the simulator and between each assessment score and patient outcomes were calculated. The participants were also surveyed regarding various aspects of the simulator as a training tool. ResultsThere was no correlation between the average score of the pancreaticojejunostomy performed in the operating room and that in the simulator environment (r = 0.047). Pancreaticojejunostomy scores using the simulator were significantly lower in patients with postoperative pancreatic fistula than in those without postoperative pancreatic fistula (P = .05). Multivariate analysis showed that pancreaticojejunostomy assessment scores were independent factors in postoperative pancreatic fistula (P = .09). The participants highly rated the simulator and considered that it had the potential to be used for training. ConclusionThere was no correlation between pancreaticojejunostomy surgical performance in the operating room and the simulation environment. Surgical skills evaluated in the simulation setting could predict patient surgical outcomes.

Implementation of a Canine Ergonomic Abdominal Simulator for Training Basic Laparoscopic Skills in Veterinarians.

The validity of the CALMA Veterinary Lap-trainer simulator (CVLTS) for training basic veterinary laparoscopic skills was assessed and compared to a simple collapsible mobile box trainer. Ten veterinarian surgeons with no experience in laparoscopic surgery and four experts with at least two years of experience in minimally invasive surgery (MIS) were included. The training curriculum included object transfer, non-woven gauze cutting with curved scissors, and interrupted and continuous intracorporeal sutures, which were practiced on the CVLTS. The initial and final assessments were carried out in both the CVLTS and in a collapsible mobile simulator. These were video-recorded and evaluated by external experts using the Objective Structured Assessment of Technical Skills (OSATS) and a specific scale evaluation in a double-blinded schedule. The time, angular displacement, number, and movement smoothness were recorded using a hands movement assessment system (HMAS). Through a survey, the face validity and content were evaluated. The data were analyzed by a Pearson's proportions comparison or Mann Whitney U test and a bilateral Student's t-test. The experimental group OSATS, specific scores, and HMAS values, with the exception of the smoothness of movements, significantly improved after training, with no statistically significant differences compared to the expert group. No differences were found between the two simulators. The experts' and experimental participants' CVLTS mean score was 4.8. Our data support the CVLTS validations for laparoscopic surgery basic skills training.

Developing an Objective Structured Assessment of Technical Skills (OSATS) for Microlaryngoscopy.

Microlaryngoscopy is a basic technical skill in Oto-HNS. It is essential for residency programs to have a competency-based assessment tool to evaluate residents' performance of this procedure. An Objective Structured Assessment of Technical Skills (OSATS) is a procedure-specific assessment, which consists of the following: (a) Operation-Specific Checklist and (b) Global Rating Scale (GRS). The objective of this study was to create an OSATS for adult microlaryngoscopy. This was a prospective study, with an initial qualitative phase for OSATS development (Phase I), and a clinical pilot phase (Phase II). In Phase I, interviews were conducted with three laryngologists to establish a stepwise description of adult microlaryngoscopy and review a previously validated GRS for relevance to microlaryngoscopy. Responses were used to create a framework for the OSATS. The OSATS was then presented to Oto-HNS residents and laryngologists in an alternating fashion, for review of clarity and relevance. A pilot study was then performed to evaluate the resident performance of adult microlaryngoscopy. Multiple regression analysis was carried out to investigate whether training level, case complexity, and previous OSATS exposure could predict participant scores. Phase I of this study led to the creation of a 34-item OSATS. The pilot study (N=28 procedures) revealed that training level was significantly correlated with increased OSATS scores. There was no statistically significant correlation between case complexity and resident scores. Assessors reported the perceived utility of the OSATS and intent for use in residency training. Application of the proposed OSATS will allow for competency-based assessment of the resident performance of microlaryngoscopy. NA Laryngoscope, 133:2719-2724, 2023.

Force-based assessment of tissue handling skills in simulation training for robot-assisted surgery

IntroductionAlthough robotic-assisted surgery is increasingly performed, objective assessment of technical skills is lacking. The aim of this study is to provide validity evidence for objective assessment of technical skills for robotic-assisted surgery.MethodsAn international multicenter study was conducted with participants from the academic hospitals Heidelberg University Hospital (Germany, Heidelberg) and the Amsterdam University Medical Centers (The Netherlands, Amsterdam). Trainees with distinctly different levels of robotic surgery experience were divided into three groups (novice, intermediate, expert) and enrolled in a training curriculum. Each trainee performed six trials of a standardized suturing task using the da Vinci Surgical System. Using the ForceSense system, five force-based parameters were analyzed, for objective assessment of tissue handling skills. Mann–Whitney U test and linear regression were used to analyze performance differences and the Wilcoxon signed-rank test to analyze skills progression.ResultsA total of 360 trials, performed by 60 participants, were analyzed. Significant differences between the novices, intermediates and experts were observed regarding the total completion time (41 s vs 29 s vs 22 s p = 0.003), mean non zero force (29 N vs 33 N vs 19 N p = 0.032), maximum impulse (40 Ns vs 31 Ns vs 20 Ns p = 0.001) and force volume (38 N3 vs 32 N3 vs 22 N3p = 0.018). Furthermore, the experts showed better results in mean non-zero force (22 N vs 13 N p = 0.015), maximum impulse (24 Ns vs 17 Ns p = 0.043) and force volume (25 N3 vs 16 N3p = 0.025) compared to the intermediates (p ≤ 0.05). Lastly, learning curve improvement was observed for the total task completion time, mean non-zero force, maximum impulse and force volume (p ≤ 0.05).ConclusionConstruct validity for force-based assessment of tissue handling skills in robot-assisted surgery is established. It is advised to incorporate objective assessment and feedback in robot-assisted surgery training programs to determine technical proficiency and, potentially, to prevent tissue trauma.

Development and evaluation of a portable and soft 3D-printed cast for laparoscopic choledochojejunostomy model in surgical training

BackgroundLaparoscopic choledochojejunostomy (LCJ) is an essential basic skill for biliary surgeons. Therefore, we established a convenient and effective LCJ 3D printing model to evaluate whether the model could simulate the actual operation situation and determine its effectiveness and validity in surgical training.MethodsA 3D printing dry laboratory model was established to simulate LCJ. The face and content validity of the model were evaluated by six experienced biliary surgeons based on 5-point Likert scale questionnaires. A total of 15 surgeons with different levels of experience performed LCJ on the model and evaluated the structural validity of the model using the objective structured assessment of technical skills (OSATS). Simultaneously, the operation time of each surgery was also recorded. A study was also performed to further evaluate the learning curve of residents.ResultsThe operating space score of the model was 4.83 ± 0.41 points. The impression score of bile duct and intestinal canal was 4.33 ± 0.52 and 4.17 ± 0.41 points, respectively. The tactile sensation score of bile duct suture and intestinal canal suture was 4.00 ± 0.63 and 3.83 ± 0.41points, respectively. The OSATS score for model operation in the attending group was 29.20 ± 0.45 points, which was significantly higher than that in the fellow group (26.80 ± 1.10, P = 0.007) and the resident group (19.80 ± 1.30, P < 0.001). In addition, there was a statistical difference in operation time among surgeons of different experience levels (P < 0.05). Residents could significantly improve the surgical score and shorten the time of LCJ through repeated training.ConclusionsThe 3D printing LCJ model can simulate the real operation scenes and distinguish surgeons with different levels of experience. The model is expected to be one of the training methods for biliary tract surgery in the future.

Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria.

Classic methods of surgery skills evaluation tend to classify the surgeon performance in multi-categorical discrete classes. If this classification scheme has proven to be effective, it does not provide in-between evaluation levels. If these intermediate scoring levels were available, they would provide more accurate evaluation of the surgeon trainee. We propose a novel approach to assess surgery skills on a continuous scale ranging from 1 to 5. We show that the proposed approach is flexible enough to be used either for scores of global performance or several sub-scores based on a surgical criteria set called Objective Structured Assessment of Technical Skills (OSATS). We established a combined CNN+BiLSTM architecture to take advantage of both temporal and spatial features of kinematic data. Our experimental validation relies on real-world data obtained from JIGSAWS database. The surgeons are evaluated on three tasks: Knot-Tying, Needle-Passing and Suturing. The proposed framework of neural networks takes as inputs a sequence of 76 kinematic variables and produces an output float score ranging from 1 to 5, reflecting the quality of the performed surgical task. Our proposed model achieves high-quality OSATS scores predictions with means of Spearman correlation coefficients between the predicted outputs and the ground-truth outputs of 0.82, 0.60 and 0.65 for Knot-Tying, Needle-Passing and Suturing, respectively. To our knowledge, we are the first to achieve this regression performance using the OSATS criteria and the JIGSAWS kinematic data. An effective deep learning tool was created for the purpose of surgical skills assessment. It was shown that our method could be a promising surgical skills evaluation tool for surgical training programs.

Video Grading of Pancreatic Anastomoses During Robotic Pancreatoduodenectomy to Assess Both Learning Curve and the Risk of Pancreatic Fistula: A Post Hoc Analysis of the LAELAPS-3 Training Program.

To assess the learning curve of pancreaticojejunostomy during robotic pancreatoduodenectomy (RPD) and to predict the risk of postoperative pancreatic fistula (POPF) by using the objective structured assessment of technical skills (OSATS), taking the fistula risk into account. RPD is a challenging procedure that requires extensive training and confirmation of adequate surgical performance. Video grading, modified for RPD, of the pancreatic anastomosis could assess the learning curve of RPD and predict the risk of POPF. Post hoc assessment of patients prospectively included in 4 Dutch centers in a nationwide LAELAPS-3 training program for RPD. Video grading of the pancreaticojejunostomy was performed by 2 graders using OSATS (attainable score: 12-60). The main outcomes were the combined OSATS of the 2 graders and POPF (grade B/C). Cumulative sum analyzed a turning point in the learning curve for surgical skill. Logistic regression determined the cutoff for OSATS. Patients were categorized for POPF risk (ie, low, intermediate, and high) based on the updated alternative fistula risk scores. Videos from 153 pancreatic anastomoses were included. Median OSATS score was 48 (interquartile range: 41-52) points and with a turning point at 33 procedures. POPF occurred in 39 patients (25.5%). An OSATS score below 49, present in 77 patients (50.3%), was associated with an increased risk of POPF (odds ratio: 4.01, P =0.004). The POPF rate was 43.6% with OSATS < 49 versus 15.8% with OSATS ≥49. The updated alternative fistula risk scores category "soft pancreatic texture" was the second strongest prognostic factor of POPF (odds ratio: 3.37, P =0.040). Median cumulative surgical experience was 17 years (interquartile range: 8-21). Video grading of the pancreatic anastomosis in RPD using OSATS identified a learning curve and a reduced risk of POPF in case of better surgical performance. Video grading may provide a valid method to surgical training, quality control, and improvement.

Design and Validation of a 3D Printed Cranio-Facial Simulator: A Novel Tool for Surgical Education.

To assess the ability of current 3D printing technology to generate a craniofacial bony and soft tissue anatomical model for use in simulating the performance of a fronto-orbital advancement (FOA) osteotomy and then to further assess the value of the model as an educational tool. Anatomic models were designed with a process of serial anatomic segmentation/design, 3D printing, dissection, and device refinement. A validation study was conducted with 5 junior and 5 senior plastic surgery residents. The validation study incorporated a multiple-choice Knowledge Assessment test (KA), an Objective Structured Assessment of Technical skills (OSATs), a Global Rating Scale (GRS) and a Michigan Standard Simulation Experience Scale (MiSSES). We compared the scores of both the junior and senior residents and compared junior resident scores, before and after viewing a lecture/demonstration. MiSSES showed high face validity with a score of 85.1/90, signifying high satisfaction with the simulator learning experience. Simulation and the lecture/demonstration improved the junior resident average KA score from 5.6/10 to 9.6/10 (P = .02), OSATs score from 32.4/66 to 64.4/66 (P < .001) and GRS score from 13.9/35 to 27.5/35 (P < .001). The senior residents OSATs score of 56.3/66 was higher than the pre-lecture juniors (32.4/66) (P < .001), but lower than the post-lecture juniors (64.4/66) (P < .001). We have successfully fabricated a 3D printed craniofacial simulator capable of being used as an educational tool alongside traditional surgical training. Next steps would be improving soft tissue realism, inclusion of patient and disease specific anatomy and creation of models for other surgical specialties.

Performance in the Fundamentals of Laparoscopic Surgery: Does it reflect global rating scales in the Objective Structured Assessment of Technical Skills in porcine laparoscopic surgery?

ObjectiveTo correlate the utility of the Fundamentals of Laparoscopic Surgery (FLS) manual skills program with the Objective Structured Assessment of Technical Skills (OSATS) global rating scale in evaluating operative performance. MethodsThe Asian Urological Surgery Training and Educational Group (AUSTEG) Laparoscopic Upper Tract Surgery Course implemented and validated the FLS program for its usage in laparoscopic surgical training. Delegates’ basic laparoscopic skills were assessed using three different training models (peg transfer, precision cutting, and intra-corporeal suturing). They also performed live porcine laparoscopic surgery at the same workshop. Live surgery skills were assessed by blinded faculty using the OSATS rating scale. ResultsFrom March 2016 to March 2019, a total of 81 certified urologists participated in the course, with a median of 5 years of post-residency experience. Although differences in task time did not reach statistical significance, those with more surgical experience were visibly faster at completing the peg transfer and intra-corporeal suturing FLS tasks. However, they took longer to complete the precision cutting task than participants with less experience. Overall OSATS scores correlated weakly with all three FLS tasks (peg transfer time: r=−0.331, r2=0.110; precision cutting time: r=−0.240, r2=0.058; suturing with intra-corporeal knot time: r=−0.451, r2=0.203). ConclusionFLS task parameters did not correlate strongly with OSATS globing rating scale performance. Although FLS task models demonstrated strong validity, it is important to assimilate the inconsistencies when benchmarking technical proficiency against real-life operative competence, as evaluated by FLS and OSATS, respectively.

A randomized study on the value of self-directed versus traditional mentor-led microsurgical training.

Analyze efficacy of self-directed resident microvascular training versus a mentor-led course. Randomized, single-blinded cohort study. Academic tertiary care center. Sixteen resident and fellow participants were randomized into two groups stratified by training year. Group A completed a self-directed microvascular course with instructional videos and self-directed lab sessions. Group B completed a traditional mentor-led microvascular course. Both groups spent equal time in the lab. Video recorded pre and post-course microsurgical skill assessments were performed to assess the efficacy of the training. Two microsurgeons, blinded to participant identity, evaluated the recordings and inspected each microvascular anastomosis (MVA). Videos were scored using an objective-structured assessment of technical skills (OSATS), a global rating scale (GRS), and quality of anastomosis scoring (QoA). The pre-course assessment identified that the groups were well matched with only "Economy of Motion" on the GRS favoring the mentor led group (p=.02). This difference remained significant on the post assessment (p=.02) Both groups significantly improved in OSATS and GRS scoring (p < .05). There was no significant difference in OSATS improvement between the two groups (p=.36) or improvement in MVA quality between groups (p > .99). Time to completion of MVA significantly improved overall by a mean of 8min and 9s (p=.005) with no significant difference between post training times to complete (p=.63). Different microsurgical training models have previously been validated as effective methods for improved MVA performance. Our findings indicate that a self-directed microsurgical training model is an effective alternative to a traditional mentor driven models. Level 2.

Origami Box Folding Exercise: A laparoscopic box simulation training and assessment method in facilitating laparoscopic psychomotor skills with minimal preparation

ObjectiveLaparoscopic box simulation training is widely recognized as an assessment tool to facilitate psychomotor skills especially for novice surgeons. However, current commercialized training modules including pegs, gauze, clips, pins etc. are generally costly and relatively inaccessible. We introduce a simple and pioneer surgical training drill, the Origami Box Folding Exercise (OBFE), based on the validated evaluating system of objective structured assessment of technical skills (OSATS) constructed with the scoring system of procedure-specific checklist (PSC) and global rating scale (GRS). Materials and methodsFace and content validation of the OBFE and OSATS are evaluated by five endoscopic experts from two medical centers in Taiwan. This is a prospective observational study analyzing the pre-test/post-test result of OBFE from 37 participants in two individual workshops as training and evaluating method for laparoscopic psychomotor skills. Both the pre and post tests are video recorded with a time limit of 5 min graded by two independent evaluators based on the OSATS scoring system. ResultsThe reliability of PSC, GRS, and intergroup value between PSC and GRS were 0.923, 0.926 and 0.933, respectively. Inter-rater reliability of PSC, GRS, and both were 0.985, 0.932 and 0.977, respectively. Construct validity of PSC and GRS were statistically significant, with p-value 0.006 and 0.001, respectively. ConclusionOBFE enhances laparoscopic psychomotor skills with requirement of a single piece of paper. The associated OSATS tool for a 5-min OBFE test was validated. OBFE training is an efficient training and assessment system to promote psychomotor skills in laparoscopic box simulation drill which requires simple and economical preparation.

A Novel Method of Real-Time Assessment for Coronary Artery Anastomosis Skill.

Coronary anastomosis is the most key factor to accomplish coronary artery bypass grafting, which is one of the largest areas in cardiovascular surgery. Although we have organized on-site simulator training courses of coronary anastomosis using BEAT YOUCAN, it became difficult to continue it because of COVID-19. Therefore, we established a real-time evaluation sheet instead of an Objective Structured Assessment of Technical Skills (OSATS) evaluation sheet. The purposes of this study was to develop the real-time assessment system and to prove the correlation between the score obtained by the OSATS and the score obtained by the real-time evaluation system. A total of 22 videos from the qualifying round of real-time coronary anastomosis competition evaluated by both the modified OSATS and the real-time evaluation system were utilized in this study. The global rating score of OSATS was compared with the global rating score of real-time evaluation system. When examined the relationship between the OSATS total score and the real-time total score, there was a significant correlation (R = 0.752, p <0.001). The OSATS general definition score and the real-time total score also showed a strong correlation (R = 0.733, p <0.001). We developed a real-time assessment sheet to evaluate coronary anastomosis. This assessment sheet had a good correlation with the OSATS evaluation sheet.

Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation

ObjectivesWorkplace-based assessment (WBA) is a key requirement of competency-based medical education in postgraduate surgical education. Although simulated workplace-based assessment (SWBA) has been proposed to complement WBA, it is insufficiently adopted...

Low-fidelity simulation models in urology resident's microsurgery training.

To evaluate the gain of microsurgical skills and competencies by urology residents, using low-fidelity experimental models. The study involved the use of training boards, together with a low-fidelity microsurgery simulator, developed using a 3D printer. The model consists in two silicone tubes, coated with a resin, measuring 10 cm in length and with internal and external diameters of 0.5 and 1.5 mm. The support for the ducts is composed by a small box, developed with polylactic acid. The evaluation of the gain of skills and competencies in microsurgery occurred throughout a training course consisting of five training sessions. The first sessions (S1-S4) took place at weekly intervals and the last session (S5) was performed three months after S4. During sessions, were analyzed: the speed of performing microsurgical sutures in the pre and post-training and the performance of each resident through the Objective Structure Assessment of Technical Skill (OSATS) and Student Satisfaction Self-Confidence tools in Learning (SSSCL). There was a decrease in the time needed to perform the anastomosis (p=0.0019), as well as a progressive increase in the score in the OSATS over during sessions S1 to S4. At S5, there was a slightly decrease in performance (p<0.0001), however, remaining within the expected plateau for the gain of skills and competences. The SSSCL satisfaction scale showed an overall approval rating of 96.9%, with a Cronback alpha coefficient of 83%. The low-fidelity simulation was able to guarantee urology residents a solid gain in skills and competencies in microsurgery.

Value of the Crowd-Sourced Assessment of Technical Skills (C-SATS) Platform in Surgical Procedures: A Systematic Review of Evidence

Background: Crowd-Sourced Assessment of Technical Skills (C-SATS) is a surgical data management and learning platform that leverages the knowledge of large expert surgeon and lay groups to assess the technique and technical skills of surgeons in a highly efficient manner. The aim of this systematic review was to summarize published literature on the performance of C-SATS as compared to expert evaluations and assess its use as a training and validation tool in minimally invasive surgery (MIS). Methods: A comprehensive literature search was performed per PRISMA guidelines using the Medline, Embase, and Google Scholar databases on published studies that evaluated the use of C-SATS following MIS, such as laparoscopic or robotic-assisted surgery. Results: A total of 21 reports were included in the review. Twelve studies comparing crowd-sourcing evaluations against expert opinion indicated overall excellent or good correlation with Global Operative Assessment of Laparoscopic Skills (GOALS), Global Evaluative Assessment of Robotic Skills (GEARS), and Robotic Objective Structured Assessment of Technical Skills (R-OSATS) scores, with correlation coefficients (Pearson or Spearman) ranging from 0.69 to 0.95 and reliability index (Cronbach’s alpha) from 0.63 to 0.93 across different specialties and surgical approaches. When using C-SATS to assess performance and validation, assessments positively correlated with traditional methods of time and error-based scoring and global rating scale. Conclusions: Based on the current published literature, the C-SATS platform has been shown to efficiently provide crowd-sourced evaluations that correlate favorably with expert evaluation across a range of surgical specialties and approaches. Use of crowdsourcing has uniformly yielded accurate evaluations of surgeons’ technical skills in a markedly shorter time than expert reviews. C-SATS may be a cost-effective complement or alternative to traditional models of evaluating surgical proficiency. Future studies are needed to determine whether the use of C-SATS will lead to improved surgical performance and patient outcomes.

High fidelity simulation of the endoscopic transsphenoidal approach: Validation of the UpSurgeOn TNS Box.

Endoscopic endonasal transsphenoidal surgery is an established technique for the resection of sellar and suprasellar lesions. The approach is technically challenging and has a steep learning curve. Simulation is a growing training tool, allowing the acquisition of technical skills pre-clinically and potentially resulting in a shorter clinical learning curve. We sought validation of the UpSurgeOn Transsphenoidal (TNS) Box for the endoscopic endonasal transsphenoidal approach to the pituitary fossa. Novice, intermediate and expert neurosurgeons were recruited from multiple centres. Participants were asked to perform a sphenoidotomy using the TNS model. Face and content validity were evaluated using a post-task questionnaire. Construct validity was assessed through post-hoc blinded scoring of operative videos using a Modified Objective Structured Assessment of Technical Skills (mOSAT) and a Task-Specific Technical Skill scoring system. Fifteen participants were recruited of which n = 10 (66.6%) were novices and n = 5 (33.3%) were intermediate and expert neurosurgeons. Three intermediate and experts (60%) agreed that the model was realistic. All intermediate and experts (n = 5) strongly agreed or agreed that the TNS model was useful for teaching the endonasal transsphenoidal approach to the pituitary fossa. The consensus-derived mOSAT score was 16/30 (IQR 14-16.75) for novices and 29/30 (IQR 27-29) for intermediate and experts (p < 0.001, Mann-Whitney U). The median Task-Specific Technical Skill score was 10/20 (IQR 8.25-13) for novices and 18/20 (IQR 17.75-19) for intermediate and experts (p < 0.001, Mann-Whitney U). Interrater reliability was 0.949 (CI 0.983-0.853) for OSATS and 0.945 (CI 0.981-0.842) for Task-Specific Technical Skills. Suggested improvements for the model included the addition of neuro-vascular anatomy and arachnoid mater to simulate bleeding vessels and CSF leak, respectively, as well as improvement in materials to reproduce the consistency closer to that of human tissue and bone. The TNS Box simulation model has demonstrated face, content, and construct validity as a simulator for the endoscopic endonasal transsphenoidal approach. With the steep learning curve associated with endoscopic approaches, this simulation model has the potential as a valuable training tool in neurosurgery with further improvements including advancing simulation materials, dynamic models (e.g., with blood flow) and synergy with complementary technologies (e.g., artificial intelligence and augmented reality).

Current Landscape of Surgical Assessment Models in Urology Residency Training.

Assessing trainees' surgical proficiency is an important aspect of urological surgical training. The current standard is the Urology Milestone Project, initially implemented in 2013. This evaluation is limited in that it contains only 3 questions on surgical competency per surgical modality with assessments occurring semi-annually without real-time operative feedback. However, since the Urology Milestones Project's inception a plethora of competency-based surgical assessment tools have been described. We aim to perform a comprehensive review of the literature of these available tools and analyze their strengths and weaknesses as a way of providing a repository of available assessment strategies for further development of a more comprehensive and standardized assessment tool. A review of the primary literature was performed using key words such as "surgical assessment tools urology," "surgical assessment tools prostate," "bladder surgical assessment tools," "renal surgical assessment tools urology," and "surgical assessment tools urology task specific." Technical and nontechnical skill assessments were included. One reviewer identified and analyzed studies that published assessment tools for use in surgical and urological training. A total of 1,497 articles published between 1997-2022 were identified. Of these, 34 met the inclusion criteria. Eighteen (52.9%) were specialty nonspecific and 16 (47.1%) were specific for urological training. Of the 18 tools developed for general surgical principles, 12 (66.7%) had some form of validity, 9 (50.0%) were significantly reliable, and 2 (11.1%) were externally validated. Of the 16 tools developed specifically for use in urology training, 13 (81.3%) had some form of validity, 7 (43.8%) were significantly reliable, and none were externally validated. Of these 16 tools, 12 (75.0%) were procedure-specific and 4 (25.0%) were developed for general use in endourological procedures. Surgical training is evolving toward a competency-based model, as evidenced by the increase in assessment tools created within the past 10 years. These instruments not only provide objective feedback to trainees, but also monitor progression. However, they are heterogeneous in construct and utilization. There remains a need for the adoption of a standardized, valid, and reliable tool, ie, both procedure-specific and generalizable across multiple procedures for use in urology training.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis.

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments. We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills. The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study. We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains. Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.