Objective Structured Assessment Of Technical Skills Research Articles

Hip Arthroscopy Simulator Training With Immersive Virtual Reality Has Similar Effectiveness to Nonimmersive Virtual Reality

PurposeThe purpose of this study was to (1) compare the efficacy of immersive virtual reality (iVR) to non-immersive virtual reality (non-iVR) training in hip arthroscopy on procedural and knowledge-based skills acquisition and (2) evaluate the relative cost of each platform. MethodsFourteen orthopaedic surgery residents were randomized to simulation training utilizing an iVR Hip Arthroscopy Simulator (n=7; PrecisionOS, Vancouver, Canada) or non-iVR simulator (n=7; ArthroS Hip VR; VirtaMed, Zurich, Switzerland). After training, performance was assessed on a cadaver by four expert hip arthroscopists through arthroscopic video review of a diagnostic hip arthroscopy. Performance was assessed using the Objective Structured Assessment of Technical Skills (OSATS) and Arthroscopic Surgery Skill Evaluation Tool (ASSET) scores. A cost analysis was performed using the transfer effectiveness ratio (TER) and a direct cost comparison of iVR to non-iVR. ResultsDemographic characteristics did not differ between treatment arms, by training level, hip arthroscopy experience, or prior simulator use. No significant differences were observed in OSATS and ASSET scores between iVR and non-iVR cohorts (OSATS: iVR 19.6±4.4, non-iVR 21.0±4.1, p=0.55; ASSET: iVR 23.7±4.5, non-iVR 25.8±4.8, p=0.43). The absolute TER was 0.06 and there was a 132-fold cost difference of iVR to non-iVR. ConclusionsHip arthroscopy simulator training with iVR had similar performance results to non-iVR for technical skill and procedural knowledge acquisition after expert arthroscopic video assessment. The iVR platform had similar effectiveness in transfer-of-skill compared to non-iVR with a 132-times cost differential.

Comparing the impact of three-dimensional digital visualization technology versus traditional microscopy on microsurgeons in microsurgery: a prospective self-controlled study.

Emerging three-dimensional digital visualization technology (DVT) provides more advantages than traditional microscopy in microsurgery; however, its impact on microsurgeons' visual and nervous systems and delicate microsurgery is still unclear, which hinders the wider implementation of DVT in digital visualization for microsurgery. Forty-two microsurgeons from the Zhongshan Ophthalmic Center were enrolled in this prospective self-controlled study. Each microsurgeon consecutively performed 30min conjunctival sutures using a three-dimensional digital display and a microscope, respectively. Visual function, autonomic nerve activity, and subjective symptoms were evaluated before and immediately after the operation. Visual functions, including accommodative lag, accommodative amplitude, near point of convergence and contrast sensitivity function (CSF), were measured by an expert optometrist. Heart rate variability was recorded by a wearable device for monitoring autonomic nervous activity. Subjective symptoms were evaluated by questionnaires. Microsurgical performance was assessed by the video-based Objective Structured Assessment of Technical Skill (OSATS) tool. Accommodative lag decreased from 0.63 (0.18) diopters (D) to 0.55 (0.16) D ( P =0.014), area under the log contrast sensitivity function increased from 1.49 (0.15) to 1.52 (0.14) ( P =0.037), and heart rate variability decreased from 36.00 (13.54) milliseconds (ms) to 32.26 (12.35) ms ( P =0.004) after using the DVT, but the changes showed no differences compared to traditional microscopy ( P >0.05). No statistical significance was observed for global OSATS scores between the two rounds of operations [mean difference, 0.05 (95% CI: -1.17 to 1.08) points; P =0.95]. Subjective symptoms were quite mild after using both techniques. The impact of DVT-based procedures on microsurgeons includes enhanced accommodation and sympathetic activity, but the changes and surgical performance are not significantly different from those of microscopy-based microsurgery. Our findings indicate that short-term use of DVT is reliable for microsurgery and the long-term effect of using DVT deserve more consideration.

Appraisal of surgical outcomes and oncological efficiency of intraoperative adverse events in robotic radical gastrectomy for gastric cancer.

Surgical quality control is a crucial determinant of evaluating the tumor efficacy. To assess the ClassIntra grade for quality control and oncological outcomes of robotic radical surgery for gastric cancer (GC). Data of patients undergoing robotic radical surgery for GC at a high-volume center were retrospectively analyzed. Patients were categorized into two groups, the intraoperative adverse event (iAE) group and the non-iAE group, based on the occurrence of intraoperative adverse events. The iAEs were further classified into five sublevels (ranging from I to V according to severity) based on the ClassIntra grade. Surgical performance was assessed using the Objective Structured Assessment of Technical Skill (OSATS) and the General Error Reporting Tool. This study included 366 patients (iAE group: n = 72 [19.7%] and non-iAE group: n = 294 [80.3%]). The proportion of ClassIntra grade II patients was the highest in the iAE group (54.2%). In total and distal gastrectomies, iAEs occurred most frequently in the suprapancreatic area (50.0% and 54.8%, respectively). In total gastrectomy, grade IV iAEs were most common during lymph node dissection in the splenic hilum area (once for bleeding [grade IV] and once for injury [grade IV]). The overall survival (OS) and disease-free survival of the non-iAE group were significantly better than those of the iAE group (Log rank P < 0.001). Uni- and multi-variate analyses showed that iAEs were key prognostic indicators, independent of tumor stage and adjuvant chemotherapy (P < 0.001). iAEs in patients who underwent robotic radical gastrectomy significantly correlated with the occurrence of postoperative complications and a poor long-term prognosis. Therefore, utilization and inclusion of ClassIntra grading as a crucial surgical quality control and prognostic indicator in the routine surgical quality evaluation system are recommended.

A New Procedure-Based Assessment of Operative Skills in Gastric Bypass Surgery, Evaluated by Video Fragment Rating

PurposeFeedback on technical and procedural skills is essential during the training of residents and fellows. The aim of this study was to assess the performance of a newly created instrument for the assessment of operative skills using laparoscopic Roux-en-Y gastric bypass (LRYGB) video fragments.Materials and MethodsA new procedure-based assessment (PBA) was created by combining LRYGB key steps with a 5-point independence scale. LRYGB performed by residents and surgeons with different levels of expertise were video recorded. Fragments of the pouch creation, gastro-jejunostomy and jejunojejunostomy, were review by 12 expert bariatric surgeons and the operative skills assessed with the PBA, Objective Structured Assessment of Technical Skill (OSATS), and the Bariatric OSATS (BOSATS). The PBA was compared to the OSATS and BOSATS. Mean scores for all items of the different assessments were summarized and compared using a T-test.ResultsThe scores of the procedural steps were combined and compared for all levels. The mean scores for beginner, intermediate, and expert level were 2.71, 3.70, and 3.90 for the PBA; for the OSATS 1.84, 2.86, and 3.44; and for the BOSATS 2.78, 3.56, and 4.19. Each of these assessments differentiated between the three skill levels (all p < 0.05).ConclusionThe PBA discriminates well between different levels of operative skills. Similar patterns were found for the OSATS and BOSATS, showing that the randomly selected video fragments are representative samples for assessing skill level. Future research will demonstrate whether these results can be extrapolated to clinical training, and which scores allow for procedure certification.Graphical

“Doing well”: Intraoperative entrustable professional activity assessments provided limited technical feedback

BackgroundEntrustable Professional Activities (EPAs) allow for the assessment of specific, observable, essential tasks in medical education. Since being developed in non-surgical fields, EPA assessments have been implemented in surgery to explore intraoperative entrustment. However, assessment burden is a significant problem for faculty, and it is unknown whether EPA assessments enable formative technical feedback. EPAs' formative utility could inform how surgical programs facilitate technical feedback for trainees. We aimed to assess the extent to which narrative comments provided through the Fellowship Council (FC) EPA assessments contained technical feedback. MethodsThe FC previously collected EPA assessments for subspecialty surgical fellows from September 2020 to October 2022. Two raters reviewed assessments' narrative comments for inclusion of each skill area that makes up part of the Objective Structured Assessment of Technical Skills (OSATS). A third rater reconciled discrepant ratings. ResultsDuring the study period, there were 3302 completed EPA assessments, including 1191 fellow self-assessments, 1124 faculty assessments, and 987 assessments without an identified assessor role. We found that assessments' narrative comments related to a median of two of the seven OSATS areas (IQR:1–2). There were no comments relevant to any of the seven OSATS areas in 16.0 % of all assessments. ConclusionsIn this review of narrative comments for EPA assessments from the FC, we found that limited technical feedback of the kind included in the OSATS was provided in many assessments. These results suggest benefit to adjusting the EPA form, enhancing faculty development, or continuing additional types of targeted technical assessment intraoperatively. Key messageThis analysis of narrative comments from fellowship EPA assessments showed that many assessments included limited technical feedback. To allow for continued technical feedback for fellows, these results highlight the need for further refinements of the EPA assessment form, additional faculty development, or ongoing use of other types of technical assessment.

High-fidelity, low-cost synthetic training model for fetoscopic spina bifida repair

Fetoscopic spina bifida repair is increasingly being practiced, but limited skill acquisition poses a barrier to widespread adoption. Extensive training in relevant models, including both ex vivo and in vivo models may help. To address this, a synthetic training model that is affordable, realistic, and that allows skill analysis would be useful. This study aimed to create a high-fidelity model for training in the essential neurosurgical steps of fetoscopic spina bifida repair using synthetic materials. In addition, we aimed to obtain a cheap and easily reproducible model. We developed a 3-layered, silicon-based model that resemble the anatomic layers of a typical myelomeningocele lesion. It allows for filling of the cyst with fluid and conducting a water tightness test after repair. A compliant silicon ball mimics the uterine cavity and is fixed to a solid 3-dimensional printed base. The fetal back with the lesion (single-use) is placed inside the uterine ball, which is reusable and repairable to allow for practicing port insertion and fixation multiple times. Following cannula insertion, the uterus is insufflated and a clinical fetoscopic or robotic or prototype instruments can be used. Three skilled endoscopic surgeons each did 6 simulated fetoscopic repairs using the surgical steps of an open repair. The primary outcome was surgical success, which was determined by water tightness of the repair, operation time <180 minutes and an Objective Structured Assessment of Technical Skills score of ≥18 of 25. Skill retention was measured using a competence cumulative sum analysis of a composite binary outcome of surgical success. Secondary outcomes were cost and fabrication time of the model. We made a model that can be used to simulate the neurosurgical steps of spina bifida repair, including anatomic details, port insertion, placode release and descent, undermining of skin and muscular layer, and endoscopic suturing. The model was made using reusable 3-dimensional printed molds and easily accessible materials. The 1-time startup cost was €211, and each single-use, simulated myelomeningocele lesion cost €9.5 in materials and 50 minutes of working time. Two skilled endoscopic surgeons performed 6 simulated, 3-port fetoscopic repairs, whereas a third used a Da Vinci surgical robot. Operation times decreased by more than 30% from the first to the last trial. Six experiments per surgeon did not show an obvious Objective Structured Assessment of Technical Skills score improvement. Competence cumulative sum analysis confirmed competency for each surgeon. This high-fidelity, low-cost spina bifida model allows simulated dissection and closure of a myelomeningocele lesion. VIDEO ABSTRACT.

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

Laparoscopic surgical skill assessment and machine learning are often inaccessible to low-and-middle-income countries (LMIC). Our team developed a low-cost laparoscopic training system to teach and assess psychomotor skills required in laparoscopic salpingostomy in LMICs. We performed video review using AI to assess global surgical techniques. The objective of this study was to assess the validity of artificial intelligence (AI) generated scoring measures of laparoscopic simulation videos by comparing the accuracy of AI results to human-generated scores. Seventy-four surgical simulation videos were collected and graded by human participants using a modified OSATS (Objective Structured Assessment of Technical Skills). The videos were then analyzed via AI using 3 different time and distance-based calculations of the laparoscopic instruments including path length, dimensionless jerk, and standard deviation of tool position. Predicted scores were generated using 5-fold cross validation and K-Nearest-Neighbors to train classifiers. Surgical novices and experts from a variety of hospitals in Ethiopia, Cameroon, Kenya, and the United States contributed 74 laparoscopic salpingostomy simulation videos. Complete accuracy of AI compared to human assessment ranged from 65-77%. There were no statistical differences in rank mean scores for 3 domains, Flow of Operation, Respect for Tissue, and Economy of Motion, while there were significant differences in ratings for Instrument Handling, Overall Performance, and the total summed score of all 5 domains (Summed). Estimated effect sizes were all less than 0.11, indicating very small practical effect. Estimated intraclass correlation coefficient (ICC) of Summed was 0.72 indicating moderate correlation between AI and Human scores. Video review using AI technology of global characteristics was similar to that of human review in our laparoscopic training system. Machine learning may help fill an educational gap in LMICs where direct apprenticeship may not be feasible.

SimLife®: A New Dynamic Model for Head and Neck Surgical Oncology Simulation.

The SimLife® model consists in a human cadaver dynamized by pulsatile vascularization. The objective was to evaluate the face, contents, and constructs validity of the SimLife® model in head and neck surgical oncology simulation. Head and neck surgical oncology simulation sessions on SimLife® models were organized with lateral neck dissection and total laryngectomy. Face and contents validity were addressed by questionnaires. Constructs validity was assessed by objective structured assessment of technical skills (OSATS) score. High realism was demonstrated for consistency of tissues (7.1 ± 1.4), color of arteriesand veins (7.3 ± 1.9, 8.5 ± 1.1, respectively), and vein consistency (8.5 ± 1.2). The mean OSATS score was 19.7 ± 5.4 for residents and 32.7 ± 1.9 for senior surgeon (P = .0022). SimLife® is a hyperrealistic model for head and neck surgical oncology simulation and it might become a core component of the surgical resident curriculum.

Objective structured assessment of medical students’ technical skills in second-degree perineal laceration repair with sponge model-based training

Abstract Purpose In this cohort study, we used a sponge simulator to train students in second-degree perineal laceration repair. We examined whether the training course improved the students’ skills, as measured with an objective structured assessment of technical skills (OSATS) and by a senior physician. We also examined the correlation between these ratings to assess the validity of OSATS application in this context. Methods Between April and July 2022, 40 medical students took part in gynecological/obstetrics training that included a lecture about perineal trauma and the viewing of a video that demonstrated second-degree perineal laceration repair using a sponge model. They then underwent initial evaluation by a senior physician and OSATS application, yielding two independent scores. After training with the sponge model, a second evaluation was performed. The OSATS assessed practical skills (8 items) and suture results (2 items). The senior physician assigned ratings on a five-point ordinal scale ranging from 1 (excellent) to 5 (poor). Results Training with the sponge simulator significantly increased students’ OSATS (practical skills, p &lt; 0.001; suture results, p &lt; 0.05) and senior physician (p &lt; 0.001) ratings. The OSATS and senior physician ratings correlated strongly (Spearman’s r: first assessment, – 0.72; second assessment, – 0.74; p &lt; 0.01). Conclusion The sponge-based training improves students’ skills for the repair of a second-degree perineal laceration. The OSATS for the sponge model might be a valid option to examine medical students in an obstetrical course.

Assessment of suturing and scaling skills of periodontology and oral medicine residents by OSATS method: a pilot study

IntroductionUpdating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method.MethodsAll six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents’ satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used.ResultsThere was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96–0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86–0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores.ConclusionConsidering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

Inanimate 3D printed model for thoracoscopic repair of esophageal atresia with tracheoesophageal fistula.

Thoracoscopic repair of esophageal atresia (EA) and tracheoesophageal fistula (TEF) poses significant technical challenges. This study aimed to develop an inexpensive, reusable, high-fidelity synthetic tissue model for simulating EA/TEF repairs and to assess the validity of the simulator. By using 3D printing and silicone casting, we designed an inexpensive and reusable inanimate model for training in thoracoscopic EA/TEF repair. The objective was to validate the model using a 5-point Likert scale and the Objective Structured Assessment of Technical Skills (OSATS) to evaluate participants' surgical proficiency. A total of 18 participants (7 medical students, 4 pediatric surgery trainees, and 7 experienced surgeons), after being instructed and trained, were asked to perform TEF ligation, dissection, as well as esophageal anastomosis using six sliding knots on the EA/TEF simulator. All participants in the expert group completed the task within the 120-minute time limit, however only 4 (57%) participants from the novice/intermediate completed the task within the time limit. There was a statistically significant difference in OSATS scores for the "flow of task" (p = 0.018) and scores for the "overall MIS skills" (p = 0.010) task distinguishing between novice and intermediates and experts. The simulator demonstrated strong suitability as a training tool, indicated by a mean score of 4.66. The mean scores for the model's realism and the working environment were 4.25 and 4.5, respectively. Overall, the face validity was scored significantly lower in the expert group compared to the novice/intermediate groups (p = 0.0002). Our study established good face and content validity of the simulator. Due to its reusability, and suitability for individual participants, our model holds promise as a training tool for thoracoscopic procedures among surgeons. However, novices and trainees struggled with advanced minimally invasive surgical procedures. Therefore, a structured and focused training curriculum in pediatric MIS is needed for optimal utilization of the available training hours.

A comparison between augmented reality and traditional in-person teaching for vascular anastomotic surgical skills training

BackgroundAugmented reality (AR) superimposes computer-generated content to a real-world environment through multitudinous devices, and is used across multiple training fora. Its use in vascular surgery education is yet to be formally investigated. The aim is to assess feasibility and effectiveness of remote teaching of vascular anastomosis skills enhanced by AR in the form of the HoloLens2 Head-Mounted Display technology with traditional in-person skills teaching. A remote trainer used video, gestures and images superimposed over participants' field of vision via the HoloLens2 to teach the skills. MethodsTwenty-eight participants underwent a preassessment performing an end-to-end vascular anastomosis on an artificial vessel. They were allocated randomly to an AR or in-person group, and underwent two teaching sessions. Individuals were asked to complete a postsession feedback form and assessment (video recorded and anonymized). The videos were marked by two blinded, independent assessors using the Objective Structured Assessment of Technical Skills (OSATS) scoring. ResultsThere was an overall improvement in both cohorts in OSATS score after the intervention by +7.083 in the in-person group and +8.275 in the AR. Independent t test was performed and a P value of .422 was obtained, indicating no statistically significant difference in the change in OSATS scores when comparing the skills teaching received in-person with that through AR. ConclusionsRemote teaching enhanced by AR is feasible and effective for the teaching of vascular surgical anastomosis skills and noninferior to in-person teaching. There is scope for development of the use of AR in vascular surgical skills training.

Feedback Versus Compliments Versus Both in Suturing and Knot Tying Simulation: A Randomized Controlled Trial

IntroductionSuturing is an expected skill for students graduating from health professions programs. Previous studies investigated student experience with teaching sessions utilizing constructive feedback versus compliments but did not investigate the combination of both. MethodsIn this parallel, randomized controlled trial, participants were divided into three groups: feedback (F), compliments (C), or feedback and compliments (FC). Participants received standardized instruction on simple interrupted suturing and two-handed knot-tying, and were videotaped performing this skill before and after the intervention. Performance was evaluated using a validated Objective Structured Assessment of Technical Skills (OSATS) instrument. Participants completed a preintervention and postintervention survey rating their task enjoyment and self-assessment of performance. Analysis was performed to determine differences between and within the groups using Kruskal–Wallis, Wilcoxon rank-sum, and Mann–Whitney U tests. ResultsA total of 31 students participated: 11 in C, 10 in F, and 10 in FC. The groups had similar preintervention OSATS scores. The F and FC groups demonstrated significant improvement in OSATS score after intervention, group C was not significantly different: F median of 11.25-19.75 points (P = 0.002); FC median of 11.75-21 points (P = 0.002); C median of 13-14 points (P = 0.2266). Between the groups FC and F both had significant performance improvement compared with C (P < 0.001 and P = 0.001 respectively). The FC group had a significantly higher rating of their enjoyment of the task on the postintervention survey compared with both the C and F groups with a median rating of 10 compared with 8 and 8 (P = 0.0052 and P = 0.0126, respectively). ConclusionsThe combination of feedback and compliments was associated with improvement in performance on suturing and knot-tying similar to the feedback-only group. The FC group rated a higher level of enjoyment of the activity compared to feedback or compliments alone.

Comparison of practice vs theory model training for vacuum-assisted delivery: A randomized controlled trial

ObjectiveTo assess the effect of two modalities of simulation training on acquisition/retention of skills for performing operative vaginal delivery. DesignRandomized, controlled, single-centre study. SettingA tertiary referral hospital in Italy. ParticipantsTwenty residents from a single university programme and two young specialists. InterventionGroup 1 had an individual training session with a single senior specialist using a fixed simulator model. After the session, trainees watched a pre-recorded 1-h lecture on vacuum-assisted operative vaginal delivery. Group 1 repeated the simulator session using the same test after 8–12 weeks and 12 months. Group 2 watched the pre-recorded lecture then undertook the same sessions as Group 1. Video recordings of all test performances were evaluated by five specialists in a blinded manner. Main outcome measureEach procedure was evaluated using a Global Rating Scale (GRS), scored with 0–5 points for each item on an Objective Structured Assessment of Technical Skills (OSATS) dataset with seven items (total 35 points). OSATS were evaluated over time, compared for the whole population, and weighted for route and year of residency. The primary outcome was comparison of the effectiveness of training between the two groups based on year of residency by assessing videos of the baseline test and GRS for OSATS scores. The secondary outcome was overall retention of skills at 8–12 weeks and 12 months. ResultsTwenty-two participants were recruited and randomized to either Group 1 (n = 11) or Group 2 (n = 11). Five participants did not complete follow-up. The primary outcome of GRS for OSATS scores at time 0 differed significantly between groups for total GRS score, and ‘Call for help’ and ‘Explanation to woman and relatives’ item scores (p = 0.002 and p = 0.007, respectively). In a multiple linear regression analysis, OSATS scores were not independently influenced by year of residency. At 8–12-week follow-up, the ‘Call for help’ item score showed a significant improvement in Group 1 (p = 0.018), although this was not confirmed when year of residency was included as an independent variable. At 12-month follow-up, none of the item scores demonstrated a significant change (p = 0.033). Year of residency did not influence the difference between groups. One-way analysis of variance found significant differences between the groups for ‘Localization of the flexion point’ (p = 0.005), ‘Traction of vacuum cup’ (p = 0.039) and ‘Use of second hand of the operator’ (p = 0.009) item scores and total GRS score (p = 0.007). The values weighted by year of residency did not demonstrate any significant difference. The secondary outcomes evaluated all the candidates of both groups for retention of technical skills over time. A significant effect of time was found for total GRS score (p < 0.001) and OSATS item scores. ConclusionsIndependent of the sequence of theoretical teaching and simulation training, trainees demonstrated high retention – and, actually, improvement – of technical skills for operative vaginal delivery at 12-month follow-up.

High-Fidelity Cleft Simulation Maintains Improvements in Performance and Confidence: A Prospective Study

High-fidelity simulation has a growing role in plastic surgical education. This study tests the hypothesis that cleft lip repair simulation followed by structured debriefing improves performance and self-confidence and that gains are maintained. Prospective, single-blinded interventional study with repeated measures. Trainees performed cleft lip repair on a high-fidelity simulator followed by debriefing, immediately completed a second repair, and returned 3 months later for a third session. Anonymized simulation videos were rated using the modified Objective Structured Assessment of Technical Skills (OSATS) and the Unilateral Cleft Lip Repair competency assessment tool (UCLR). Self-assessed cleft lip knowledge/confidence and procedural self-confidence were surveyed after each simulation. Boston Children's Hospital, a tertiary care academic hospital in Boston, MA, USA. All trainees rotating through the study setting were eligible. Twenty-six participated; 21 returned for follow-up. Significant improvements (p < 0.05) occurred between the first and second simulations for OSATS, UCLR, and procedural self-confidence. Significant improvement occurred between the second and third simulations cleft lip knowledge/confidence. Compared to the first simulation, improvements were maintained at the third simulation for all variables. Training level moderately correlated with score for UCLR for the first simulation (r = 0.55, p < 0.01), deteriorated somewhat with the second (r = 0.35, p = 0.08), and no longer corelated by the third (r = 0.02, p = 0.92). Objective performance and subjective self-assessed knowledge and confidence improve with high-fidelity simulation plus structured debriefing and improvement is maintained. Differences in procedure-specific performance seen with increasing training level are reduced with simulation, suggesting it may accelerate knowledge and skill acquisition.

Preoperative Preparatory Talk (PPT): Developing Operative Expertise of Plastic Surgery Trainees by Three Rounds of Preceptor-Trainee Discussions.

Background Plastic surgery training requires the trainee to assist in surgeries to improve their on-table decision making and hone their surgical skills, but this results in an increased risk of intraoperative complications and increased operative time. It is important to have a training method that orients the trainee toward the surgery to ensure patient safety. Materials and Methods A training method called preoperative preparatory talk (PPT) was devised in which the preceptor orients the trainee toward the planned surgery in three phases. Comparison and statistical analysis of mean operative times of four stages of free flap surgeries after PPT and without PPT were done. Objective Structured Assessment of Technical Skill (OSATS) scores of surgical trainees were also documented for surgeries done with and without PPT and statistical analysis was done for comparing these scores. Results Statistical analysis via unpaired t -test confirmed that after applying PPT, there was a significant decrease in time taken in three out of four stages of free flap surgeries: flap planning and harvesting, recipient site preparation and vessel dissection, and flap division and partial inset. Trainees were found to be better oriented toward the surgery which resulted in a better performance on table that was confirmed by statistical analysis of OSATS score via unpaired t -test. Conclusion PPT ensures better learning for the resident and improves patient safety because of better orientation of the operating team toward the procedure and operating steps. This reduces the operative time of free flap surgeries. We recommend this training method to be incorporated in plastic surgery training programs.

SP11.8 Deep learning for performance assessment of laparoscopic sleeve gastrectomy (LSG): validation against objective structured assessment of technical skills (OSATS) and clinical outcomes

Abstract Background Current methods of surgeon performance assessment are unstructured, labour-intensive and subjective. Machine learning could provide rapid, automated and subjective assessment. We design a custom deep learning model to output performance metrics from videos of LSG and validate these against OSATS and clinical outcomes. Methods A novel dataset of 3210 images from videos of LSG was annotated to train an instrument segmentation framework based on Mask R-CNN, a state-of-the-art deep learning algorithm. Spatial outputs of detected instruments were used to determine workflow through a Markov chain and secondly to determine instrument metrics for assessment of surgical performance. Metrics including time spent in stage and instrument time, distance, smoothness, concentration, and efficiency were then validated against OSATS ratings from 2 independent scorers and clinical outcomes from 35 LSGs. Results Grasper and ligasure efficiency and smoothness were higher in videos rated within the top quartile by OSATS score when compared to those in the bottom quartile and when comparing consultants to trainees. Videos in the top quartile for grasper smoothness had higher total OSATS scores (17.31±2.27) compared to those in the lowest quartile (15.69±2.93) and across all 5 individual OSATS domains in addition to higher percentage of total weight loss at 3, 6, and 12 months. Conclusions This study demonstrates how vision based deep learning can be used for surgeon performance assessment. Future work will aim to validate this against a large number of videos, expand upon the range of metrics produced, and look towards the steps needed for clinical translation.

SP11.16 Development and Validation of a Novel Simulation-Based Assessment of Operative Skill for Higher Specialist Trainees in General Surgery

Abstract Introduction Simulation is an established training modality, though its use in high-stakes assessment is under-explored. This study sought to develop and validate a novel assessment of operative competence for higher specialist trainees in general surgery. Methods Content validity was derived from a previously conducted Delphi consensus study of trainers. Eight stations were created using a combination of virtual reality and bio-hybrid models. The assessment was designed to assess performance at the end of ‘phase 2’ of training. Performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure Based Assessment (PBA) tool. Validity evidence was analysed in accordance with Messick's validity framework. Both ‘junior’ (ST2 - ST4) and ‘senior’ trainees (ST 5 - ST8) were included to allow for comparative analysis. Results Thirteen trainees (six ‘junior’ trainees and seven ‘senior’ trainees) were observed and scored by a total of ten assessors across eight stations. Inter-station reliability was high (Cronbach's alpha 0.81). Inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004). Mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall’s tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all ‘senior’ trainees passing and 4/6 (66.66%) of junior trainees failing the assessment. Conclusion This study demonstrates the validity, reliability and feasibility of a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.

Effect of head-mounted displays on students' acquisition of surgical suturing techniques compared to an e-learning and tutor-led course: A randomized controlled trial.

Although surgical suturing is one of the most important basic skills, many medical school graduates do not acquire sufficient knowledge of it due to its lack of integration into the curriculum or a shortage of tutors. E-learning approaches attempt to address this issue but still rely on the involvement of tutors. Furthermore, the learning experience and visual-spatial ability appear to play a critical role in surgical skill acquisition. Virtual reality head-mounted displays (HMDs) could address this, but the benefits of immersive and stereoscopic learning of surgical suturing techniques are still unclear. In this multi-arm randomized controlled trial, 150 novices participated. Three teaching modalities were compared: an e-learning course (monoscopic), an HMD-based course (stereoscopic, immersive), both self-directed, and a tutor-led course with feedback. Suturing performance was recorded by video camera both before and after course participation (>26 hours of video material) and assessed in a blinded fashion using the OSATS Global Rating Score (GRS). Furthermore, the optical flow of the videos was determined using an algorithm. The number of sutures performed was counted, visual-spatial ability was measured with the mental rotation test (MRT), and courses were assessed with questionnaires. Students' self-assessment in the HMD-based course was comparable to that of the tutor-led course and significantly better than in the e-learning course (P=0.003). Course suitability was rated best for the tutor-led course (x̄=4.8), followed by the HMD-based (x̄=3.6) and e-learning (x̄=2.5) courses. The median ΔGRS between courses was comparable (P=0.15) at 12.4 (95% CI 10.0-12.7) for the e-learning course, 14.1 (95% CI 13.0-15.0) for the HMD-based course, and 12.7 (95% CI 10.3-14.2) for the tutor-led course. However, the ΔGRS was significantly correlated with the number of sutures performed during the training session (P=0.002), but not with visual-spatial ability (P=0.626). Optical flow (R2=0.15, P<0.001) and the number of sutures performed (R2=0.73, P<0.001) can be used as additional measures to GRS. The use of HMDs with stereoscopic and immersive video provides advantages in the learning experience and should be preferred over a traditional web application for e-learning. Contrary to expectations, feedback is not necessary for novices to achieve a sufficient level in suturing; only the number of surgical sutures performed during training is a good determinant of competence improvement. Nevertheless, feedback still enhances the learning experience. Therefore, automated assessment as an alternative feedback approach could further improve self-directed learning modalities. As a next step, the data from this study could be used to develop such automated AI-based assessments.

Residents perform better technically, have less stress and workload, and prefer robotic to laparoscopic technique during inanimate simulation.

With the widespread adoption of minimally invasive surgery, there is a growing need for surgical residents to be trained by a procedure-specific curriculum. This study aimed to evaluate the technical performance and feedback of surgical residents undergoing the robotic and laparoscopic hepaticojejunostomy (HJ) and gastrojejunostomy (GJ) biotissue modules. A total of 23 PGY-3 surgical residents participated in this study and performed the laparoscopic and robotic HJ and GJ drills, which were recorded and scored by two independent graders using the modified objective structured assessment of technical skills (OSATS). After completing each drill, all participants filled out the NASA Task Load Index (NASA-TLX), Borg Exertion Scale, and Edwards Arousal Rating Questionnaire. Twenty-two (95.7%) residents had already received fundamentals of laparoscopic surgery certification. Eighteen (78.3%) residents had robotic virtual simulation training and the median (range) number of robotic surgery console experience was 4 (0-30). In the HJ comparison of the six OSATS domains, the robotic system was superior in Gentleness (p = 0.031). In the GJ comparison, the robotic system was superior in Time and Motion (p < 0.001), Instrument Handling (p = 0.001), Flow of Operation (p = 0.002), Tissue Exposure (p = 0.013), and Summary (p < 0.001). Participants answered significantly higher demand scores for laparoscopy on all six facets of NASA-TLX for both HJ and GJ (p < 0.05). The Borg Level of Exertion was > 2 points higher for laparoscopic HJ and GJ (p < 0.001). Residents rated more Nervousness and Anxiety for laparoscopic compared to robotic (p < 0.05) HJ and GJ. Additionally, when asked to score preference for robotic and laparoscopic approach in terms of technique and ergonomics, residents scored robot as better (laparoscopy worse) for both HJ and GJ in both domains. The robotic surgical system provided a more favorable environment for trainees with less mental and physical burden for minimally invasive HJ and GJ curriculum.