Objective Structured Assessment Research Articles

Evaluating knowledge, confidence, and self-efficacy of NPs and PAs in the emergency department: extended focused assessment using sonography in trauma.

Developing and translating knowledge gained in the classroom into skills that are practiced in the clinical setting is an ongoing, iterative, and dynamic process. Health care clinicians require continually evolving knowledge, ongoing education, and hands-on skills practice particular to the specialty. This prospective observational study was conceived based on the existing evidence demonstrating that Point of Care Ultrasound (POCUS) is not routinely taught in nurse practitioner (NP) or physician assistant (PA) graduate programs. This study examines outcomes of an educational intervention aimed at improving knowledge, self-confidence, and self-efficacy of the Extended Focused Assessment using Sonography in Trauma (E-FAST examination) for both NPs and PAs working in emergency settings. Twenty participants attended a short course including both didactic learning and hands-on learning of the E-FAST examination. Participants rated themselves using a self-assessment tool for pre- and postintervention self-evaluation. The Wilcoxon signed-rank test was used to evaluate data. A second tool used to evaluate participants' hands-on use of the E-FAST examination is the Objective Structured Assessment of Ultrasound, in which participants were evaluated by clinical experts. These data were assessed using regression analysis. Participants showed improvement on the pre/post course self-assessment tool. Participants also showed proficiency using ultrasound and analyzing images as evaluated by experts. A short multimodal course can improve clinicians' knowledge, confidence, and self-efficacy in the use of POCUS and the E-FAST examination. Hands-on education is a valuable tool for ongoing learning.

Validation and reliability testing of a rating scale for objective assessment of performance in laparoscopic appendicectomy surgery

BackgroundsTo achieve a competency‐based training paradigm, the ability to obtain reliable and valid quantitative assessments of intraoperative performance is required. Through this, weaknesses can be identified and practiced, and competency assessed. This study aimed to determine the validity and reliability an objective evaluation tool for assessment of performance in laparoscopic appendicectomy (LA).MethodsA prospective single‐blinded observational study design was used. Videos of inexperienced (performed <10 LAs) and experienced (performed >100 LAs) surgeons performing LA surgery were collected. Surgical performance during each recording was rated by two independent, blinded expert surgeons using the LA Rating Scale (LARS) and the modified Objective Structured Assessment of Technical Skill (OSATS) scale.ResultsThe intraclass correlation coefficient (ICC) for LARS was 0.95 (95%CI 0.83–0.98). The ICC for each step ranged from 0.48 to 0.90, and the test–retest ICC for LARS was 0.91 (95%CI 0.69–0.98). Significant differences (P < 0.001) between median performance scores as rated by LARS were observed between the inexperienced and experienced surgeons. A Spearman's correlation coefficient of 0.87 (P < 0.001) was observed between LARS performance scores and modified OSATS scores.ConclusionLARS demonstrated excellent inter‐rater and test–retest reliability, and construct and concurrent validity and can be used to quantitatively evaluate performance during LA. This can potentially allow specific weaknesses to be identified and improved upon through deliberate practice. Progress can be tracked through re‐evaluation and scores of expert surgeons can be used as performance goals for credentialing in LA.

15. Utilization of Augmented Reality Smart Glasses for the Provision of Remote Surgical Training in the Resource-Constrained Setting

Background: Surgical mission trips constitute an important strategy by which the global surgery campaign addresses inequities in surgical disease worldwide. Travel restrictions related to the COVID-19 pandemic, however, impose a challenging limit on the reach and impact of surgical mission work. In this study, we demonstrate the utility of a remote surgical mentorship program equipped with cutting-edge augmented reality (AR) technology for sustaining global surgery capacity building efforts within the constraints of the COVID-19 pandemic. Methods: Our remote surgical training program connects mentor and mentee surgeons via an AR-enabled digital headset and integrated smartphone application. The Vuzix® Smart Glasses system allows mentee surgeons to transmit a point-of-view livestream of activity in the operating room while receiving direct audio and visual feedback from their remote mentor in realtime. To facilitate their engagement, mentor and mentee surgeons interact using an integrated smartphone application that enables case scheduling, clinical documentation, and performance evaluation. Performance is assessed on the basis of three validated assessment tools that are well-established in the surgical education literature: the Objective Structured Assessment of Technical Skills, the Self-Confidence Scale, and the Student Evaluation of Educational Quality Survey. This survey is administered via the mobile application to both mentor and mentee surgeons at the time of their onboarding as well as after every subsequent three-month training quarter for the duration of the year-long program. Results: To date, 83 mentor and mentee surgeons across 33 countries and 6 continents are enrolled and actively participating in our remote training program. Baseline survey data suggest a mentee surgeon cohort that stands to benefit greatly from inclusion in our training program. A majority of mentee surgeons report incomplete knowledge and anxiety in association with their specialty prior to initiating training. Conclusion: The COVID-19 pandemic has created an urgent need for sustainable remote surgical capacity building strategies. Our remote surgical training program powered by AR-enabled telecommunication and smartphone technology constitutes a promising solution towards this end.

P.149 High-fidelity simulation-based microsurgical training for neurosurgical residents

Background: Mastery of microsurgical skills requires regular hands-on practice. Simulation is increasingly recognized as an important supplement to operative training experience. The live rat femoral artery model is the gold standard model for microneurosurgical skills simulation. We present an 11-year experience incorporating simulation-based microsurgical training into an established Canadian neurosurgery program. Methods: Postgraduate year 2 (PGY-2) neurosurgery residents completed a one-year curriculum spanning 17 training sessions divided into 5 modules of increasing fidelity. Both perfused duck wing and live rat vessel training modules were used. Three modules comprised live microvascular anastomosis. Trainee performance was video recorded and blindly graded using the Objective Structured Assessment of Technical Skills Global Rating Scale. Results: Eighteen participants completed 107 microvascular anastomoses. There was significant improvement in six measurable skills during the curriculum. Mean overall score was significantly higher on the fifth attempt compared to the first attempt for all 3 live anastomotic modules (p&lt;0.001). Each module had a different improvement profile across the skills assessed. The greatest improvement was observed during artery-to-artery anastomosis. Conclusions: This high-fidelity microsurgical simulation curriculum demonstrated a significant improvement in the six microneurosurgical skills assessed, supporting its use as an effective teaching model. Transferability to the operative environment is actively being investigated.

Active Retrieval Improves Procedural Learning in Orthopedic Surgery

Tests are shown to enhance learning: this is known as the "testing effect". The benefit of testing is theorized to be through "active retrieval", which is the effortful process of recalling stored knowledge. This differs from "passive studying", such as reading, which is a low effort process relying on recognition. The testing effect is commonly studied in random word list scenarios and is thought to disappear as complexity of material increases. Little is known about the testing effect in complex situations such as procedural learning. Therefore, we investigated if testing improves procedural learning of fracture fixation as compared to "passive studying". Fifty participants watched an instructional video of an open reduction internal fixation of a Sawbones™ femur. Participants then performed the procedure under guided supervision (pretest). After randomization, they either read the steps (passive studying group), or wrote down the steps from memory (active retrieval group) for a period of 15 minutes. After a washout period, all participants performed the procedure without guidance (posttest) and then once more, 1 week after the initial testing (retention test). The participants were assessed using the Objective Structured Assessment of Technical Skill. Each performance was video recorded for data analysis purposes. Participants in the passive studying group had significantly higher Objective Structured Assessment of Technical Skill scores during immediate assessment compared to the active retrieval group (p = 0.001), especially with respect to remembering the correct order of the steps (p = 0.002). The percentage of information forgotten was significantly less in the active retrieval group (p = 0.02) at the retention test. We demonstrated that, compared to passive studying, testing with active retrieval through writing resulted in better retention of fracture fixation knowledge (i.e., less forgetting). These findings can easily be applied and incorporated in existing curricula. Future studies are needed to determine the effects of different kinds of active retrieval methods such as verbal retrieval (e.g., dictating) in surgical practice.

Simulation-based training using a novel Surabaya hysterectomy mannequin following video demonstration to improve abdominal hysterectomy skills of obstetrics and gynecology residents during the COVID-19 pandemic in Indonesia: a pre- and post-intervention study.

PurposeDuring the coronavirus disease 2019 (COVID-19) pandemic, the number of abdominal hysterectomy procedures decreased in Indonesia. The existing commercial abdominal hysterectomy simulation model is expensive and difficult to reuse. This study compared residents’ abdominal hysterectomy skills after simulation-based training using the Surabaya hysterectomy mannequin following a video demonstration.MethodsWe randomized 3rd- and 4th-year obstetrics and gynecology residents to a video-based group (group 1), a simulation-based group (group 2), and a combination group (group 3). Abdominal hysterectomy skills were compared between before and after the educational intervention. The pre- and post-tests were scored by blinded experts using the validated Objective Structured Assessment of Technical Skills (OSATS) and Global Rating Scale (GRS).ResultsA total of 33 residents were included in the pre- and post-tests. The OSATS and GRS mean differences after the intervention were higher in group 3 than in groups 1 and 2 (OSATS: 4.64 [95% confidence interval [CI], 2.90–6.37] vs. 2.55 [95% CI, 2.19–2.90] vs. 3.82 [95% CI, 2.41–5.22], P=0.047; GRS: 10.00 [95% CI, 7.01–12.99] vs. 5.18 [95% CI, 3.99–6.38] vs. 7.18 [95% CI, 6.11–8.26], P=0.006). The 3rd-year residents in group 3 had greater mean differences in OSATS and GRS scores than the 4th-year residents (OSATS: 5.67 [95% CI, 2.88–8.46]; GRS: 12.83 [95% CI, 8.61–17.05] vs. OSATS: 3.40 [95% CI, 0.83–5.97]; GRS: 5.67 [95% CI, 2.80–8.54]).ConclusionSimulation-based training using the Surabaya hysterectomy mannequin following video demonstration can be a bridge to learning about abdominal hysterectomy for residents who had less surgical experience during the COVID-19 pandemic.

The Use of a Low-Fidelity Simulator to Improve Vascular Anastomosis Skills of Residents during the COVID-19 Pandemic

BackgroundTo evaluate a workshop using a low-fidelity simulator for training vascular surgery residents in vascular anastomosis during the COVID-19 pandemic.DesignProspective, controlled, single-center.MethodsVascular surgery residents at the São Paulo University Medical School were enrolled in the COVID Group (five post graduation year 3 residents) or Control Group (five PGY-4 residents). The COVID Group was trained via a vascular anastomosis workshop. The residents were evaluated using Objective Structured Assessment of Technical Skills (OSATS), Final Product Analysis and time to perform the procedure. The number of anastomoses performed by the residents were calculated. Data were subjected to statistical analysis, and P < 0.05 was considered significant.ResultsThere was a significant reduction in the number of vascular anastomoses performed by the residents between the COVID group and the control group (mean 22.6 ± 7.76 vs. 35.2 ± 3.9, P = 0.01, Student's t-test). Before the workshop, 80% of the residents from the COVID group failed to perform a vascular anastomosis on the simulator. During the workshop, there was improvement in the Objective Structured Assessment of Technical Skills (OSATS) score (initial: 16.5, interquartile range (IQR) 0, under supervision: 25, IQR 5, and at the end of the workshop: 26.5, IQR 2.5; P = 0.049, Friedman's test) and in the Final Product Analysis (initial: 14.5, IQR 6, under supervision: 26.5, IQR 4.625, end of the workshop: 27, IQR 4, P = 0.049, Friedman's test). Time was not significantly different (initial: 35.6, IQR 2.77; under supervision: 25.8 min, IQR 4.53; P = 0.07, Friedman's test). The residents' technical scores were stable 6 months after the training, and there was no difference between their final scores and those of the control group. The residents from the COVID Group reported an improvement in their knowledge, technical skills and confidence after the workshop.ConclusionsA workshop using a low-fidelity simulator improved vascular surgery residents’ skills and confidence in vascular anastomosis during the pandemic year, when they performed fewer surgical procedures.

Minimum Case Numbers: The New Pandemic in a Changing Clinical Environment.

Minimum Case Numbers: The New Pandemic in a Changing Clinical Environment.

National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients

BackgroundTransurethral resection of bladder tumours (TURBT) is the initial diagnostic treatment for patients with bladder cancer. TURBT is not an easy procedure to master and simulator training may play a role in improving the learning curve. ObjectiveTo implement a national training programme for simulation-based mastery learning in TURBT and explore operating theatre performance after training. Design, setting, and participantsFrom June 2019 to March 2021, 31 doctors at urology departments in Denmark performed two pretraining TURBT procedures on patients, followed by proficiency-based mastery learning on a virtual reality simulator and then two post-training TURBTs on patients. Outcome measurements and statistical analysesOperating theatre performances were video-recorded and assessed by two independent, blinded raters using the Objective Structured Assessment for Transurethral Resection of Bladder Tumours Skills (OSATURBS) assessment tool. Paired-sample t tests were used to compare pretraining and post-training analyses and independent t tests for between-group comparisons. This trial is registered at ClinicalTrials.gov as NCT03864302. Results and limitationsBefore training, novices had significantly lower performance scores in comparison to those with intermediate experience (p = 0.017) and experienced doctors (p < 0.001). After training, novices significantly improved their clinical performance score (from 11.4 to 17.1; p = 0.049, n = 10). Those with intermediate experience and experienced doctors did not benefit significantly from simulator training (p = 0.9 and p = 0.8, respectively). ConclusionsNovices improved their TURBT performance in the operating theatre after completing a proficiency-based training programme on a virtual reality simulator. Patient summaryWe trained surgeons in an operation to remove bladder tumours using a virtual reality simulator. Novice doctors improved their performance significantly after the training, but the training effects for more experienced doctors were minimal. Therefore, we suggest the introduction of mandatory simulator training in the residency programme for urologists.

Telestration in the Teaching of Basic Surgical Skills: A Randomized Trial

To evaluate how an affordable course using telestration with augmented reality can be compared to the traditional teaching of basic surgical skills. Prospective, randomized and blinded study. Faculty of Medicine of PortoUniversity. Twenty medical students without any experience in basic surgical skills were randomized into two different learning groups: telestration and traditional teaching (on-site mentoring) groups. Five different types of sutures were taught: the single interrupted, the cruciate mattress, the horizontal mattress, the vertical mattress and the simple continuous sutures. Data was obtained on the time taken to learn each of the techniques and to perform each exercise without any support from the faculty, tension of the suture, quality of the procedure using a modified Objective Structured Assessment of Technical Skills and participants' answers to a Likert questionnaire in terms of their learning experience, confidence, and self-evaluation. Trainees in the telestration group were globally faster when performing independently (1393.40 [SD 288.89] vs 1679.00 [SD 328.22] seconds, p = 0.04) particularly during the cruciate mattress suture (235.50 [SD 61.81] vs 290.00 [SD 68.77] seconds, p = 0.05) and the simple continuous suture (492.40 [SD 87.49] vs 630.30 [SD 132.34] seconds, p = 0.01).Time needed for students to learn the procedures was similar between the groups. There were also no statistically significant differences in terms of the quality of the surgical gesture, tension of the suture, self-evaluation or confidence. A basic surgical skills course using telestration through a head-mounted device with augmented reality capabilities can be a viable alternative to traditional teaching, considering time and quality of the gesture. Though costs can discourage from using this technology in basic procedures, the use of free software may turn it into an affordable option in the context of distant learning.

Resident performance in simulation module is associated with operating room performance for laparoscopic cholecystectomy.

Simulation is an important tool in surgical training. However, the transferability of skills obtained in the simulation setting to the operating room (OR) is uncertain. This study explores the association between resident simulation performance and OR performance in a laparoscopic cholecystectomy (LC) simulation module. A simulation module focused on LC utilizing a virtual reality simulator was completed by general surgery residents. Simulation performance was evaluated using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) and Objective Structured Assessment of Technical Skills (OSATS), as well as a LC-specific simulation assessment form (LC-SIM). Resident subsequent OR performances of LC were measured by the Surgical Training and Assessment Tool (STAT), an online mobile-based evaluation completed by attending surgeons. Twenty-one residents who completed the simulation module and also with STAT data on LC from 2016 to 2020 were included. Higher scores on incision/port placement on LC-SIM is associated with better tissue handling (coefficient 0.20, p = 0.048) and better time & economy of motion on STAT (coefficient 0.22, p = 0.037). However, higher scores on time and motion on OSATS are associated with worse tissue handling (-0.28, p = 0.046), worse time & economy of motion (-0.37, p = 0.009), and worse overall grade (-0.21, p = 0.044). Higher scores on overall performance on OSATS is associated with worse time & economy of motion (-0.80, p = 0.008). Higher scores on depth perception on GOALS are associated with worse tissue handling (-0.28, p = 0.044). We found significant positive and negative associations between resident simulation performance and OR performance, particularly in tissue handling and economy of motion. This could suggest that simulation performance does not reliably predict OR performance. However, this could highlight the concept of excessive caution in the real OR environment and longer operative time which could be interpreted as worse time and economy of motion by the attending surgeons.

Does group size matter during collaborative skills learning? A randomised study.

BackgroundCollaborative skills learning in the form of dyad learning compared with individual learning has been shown to lead to non‐inferior skills retention and transfer. However, we have limited knowledge on which learning activities improve collaborative skills training and how the number of collaborators may impact skills transfer. We explored the effects of skills training individually, in dyads, triads or tetrads on learning activities during training and on subsequent skills transfer.MethodsIn a randomised, controlled study, participants completed a pre‐post‐transfer‐test set‐up in groups of one to four. Participants completed 2 hours of obstetric ultrasound training. In the dyad, triad and tetrad group participants took turns actively handling the ultrasound probe. All performances were rated by two blinded experts using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale and a Global Rating Scale (GRS). All training was video recorded, and learning activities were analysed using the Interactive‐Constructive‐Active‐Passive (ICAP) framework.ResultsOne hundred one participants completed the simulation‐based training, and ninety‐seven completed the transfer test. Performance scores improved significantly from pre‐ to post‐test for all groups (p < 0.001, ηp2 = 0.55). However, group size did not affect transfer test performance on OSAUS scores (p = 0.13, ηp2 = 0.06) or GRS scores (p = 0.23, ηp2 = 0.05). ICAP analyses of training activities showed that time spent on non‐learning and passive learning activities increased with group size (p < 0.001, ηp2 = 0.31), whereas time spent on constructive and interactive learning activities was constant between groups compared with singles (p < 0.001, ηp2 = 0.72).ConclusionCollaborative skills learning in groups of up to four did not impair skills transfer despite less hands‐on time. This may be explained by a compensatory shift towards constructive and interactive learning activities that outweigh the effect of shorter hands‐on time.

A scoping review of assessment methods of competence of general surgical trainees

BackgroundOnly rigorous evaluation of competence will result in the production of safe surgeons that are able to provide the best care for patients. The development of competency-based assessment should ultimately be evidence driven. ObjectivesExplore the volume of existing evidence pertaining to the different objective assessment methods reported in the literature. Eligibility criteriaStudies describing objective assessment of postgraduate general surgical trainees within the last 20 years. Sources of evidencePubMed, Ovid Medline and Web of Sciences. Charting methodsA data chart proforma was designed and data were extracted into tables. Basic numerical analysis of extracted data and narrative synthesis of charted data. ResultsA total of 343 papers were reviewed. 26 were eligible for inclusion. 92% of articles were published from 2008 onwards. 50% have been published in the last five years. The articles originated from 6 different countries, predominantly the United Kingdom (42%), followed by the United States of America (38%). In addition, a small number were published from Canada (8%), Japan (4%), Germany (4%) and Australia (4%). UK publications were predominantly between 2008 and 2014 while the USA had a later predominance between 2015 and 2018. 42% were based on quantitative methodology, 27% had a qualitative approach while 31% had mixed analysis. There were sixteen assessment methods presented. The most common type of assessment was Objective Structured Assessments (27%), which included Objective Structured Assessment of Technical Skill (OSATS) (23%) and Objective Structured Assessment of Non-Technical Skill (4%). Procedure Based Assessment (PBA) (23%) and Entrustability Scales (23%) were also prevalent. ConclusionsThis scoping review has identified a range of different assessment methods. The assessment methods with a higher volume and level of supporting evidence were OSATS, PBAs and Entrustability Scales. There was a lower volume and level of supporting evidence found within this review for the remaining assessment methods.

Telestration with augmented reality for visual presentation of intraoperative target structures in minimally invasive surgery: a randomized controlled study

AimsIn minimally invasive surgery (MIS), intraoperative guidance has been limited to verbal communication without direct visual guidance. Communication issues and mistaken instructions in training procedures can hinder correct identification of anatomical structures on the MIS screen. The iSurgeon system was developed to provide visual guidance in the operating room by telestration with augmented reality (AR).MethodsLaparoscopic novices (n = 60) were randomized in two groups in a cross-over design: group 1 trained only with verbal guidance first and then with additional telestration with AR on the operative screen and vice versa for group 2. Training consisted of laparoscopic basic training and subsequently a specifically designed training course, including a porcine laparoscopic cholecystectomy (LC). Outcome included time needed for training, performance with Global Operative Assessment of Laparoscopic Skills (GOALS), and Objective Structured Assessment of Technical Skills (OSATS) score for LC, complications, and subjective workload (NASA-TLX questionnaire).ResultsTelestration with AR led to significantly faster total training time (1163 ± 275 vs. 1658 ± 375 s, p < 0.001) and reduced error rates. LC on a porcine liver was performed significantly better (GOALS 21 ± 5 vs. 18 ± 4, p < 0.007 and OSATS 67 ± 11 vs. 61 ± 8, p < 0.015) and with less complications (13.3% vs. 40%, p < 0.020) with AR. Subjective workload and stress were significantly reduced during training with AR (33.6 ± 12.0 vs. 30.6 ± 12.9, p < 0.022).ConclusionTelestration with AR improves training success and safety in MIS. The next step will be the clinical application of telestration with AR and the development of a mobile version for remote guidance.

Assessment of surgical competence for neck dissection: a pilot study

Background:Progressive implementation of the milestone competence-based curriculum has created a need for new objective and validated means to assess resident surgical proficiency. A previous systematic review of the literature by our group has highlighted a shortage of tools assessing surgical competence in oncologic procedures in otolaryngology — head and neck surgery.Methods:We developed a procedure-specific assessment tool for neck dissection using a modified Delphi method. The 2-part design was modelled on the previously validated Objective Structured Assessment of Technical Skills checklist. The tool was then validated through a 1-year multicentric prospective study in collaboration with the residents and faculty from our academic centre. Additionally, we developed an online survey to assess the acceptability by residents and staff before and after the validation studies.Results:A total of 29 evaluations were completed throughout the 2016–2017 academic year. Acceptability ranked high for both residents and staff, with a single discrepancy in responses regarding a potential formative as opposed to summative use of the tool. Validation study results showed significantly higher checklist scores among senior residents than junior residents, as well as a significant score progression over time (p < 0.05). Trends in scores on the task-specific tool correlated highly to results obtained on a validated global rating scale (p < 0.05).Conclusion:The first tool assessing surgical competence in oncologic otolaryngology — head and neck surgery has been developed and shows promising validity.

Evaluating validity evidence for 2 instruments developed to assess students' surgical skills in a simulated environment.

ObjectiveTo gather and evaluate validity evidence in the form of content and reliability of scores produced by 2 surgical skills assessment instruments, 1) a checklist, and 2) a modified form of the Objective Structured Assessment of Technical Skills (OSATS) global rating scale (GRS).Study designProspective randomized blinded study.Sample populationVeterinary surgical skills educators (n =10) evaluated content validity. Scores from students in their third preclinical year of veterinary school (n = 16) were used to assess reliability.MethodsContent validity was assessed using Lawshe's method to calculate the Content Validity Index (CVI) for the checklist and modified OSATS GRS. The importance and relevance of each item was determined in relation to skills needed to successfully perform supervised surgical procedures. The reliability of scores produced by both instruments was determined using generalizability (G) theory.ResultsBased on the results of the content validation, 39 of 40 checklist items were included. The 39‐item checklist CVI was 0.81. One of the 6 OSATS GRS items was included. The 1‐item GRS CVI was 0.80. The G‐coefficients for the 40‐item checklist and 6‐item GRS were 0.85 and 0.79, respectively.ConclusionContent validity was very good for the 39‐item checklist and good for the 1‐item OSATS GRS. The reliability of scores from both instruments was acceptable for a moderate stakes examination.ImpactThese results provide evidence to support the use of the checklist described and a modified 1‐item OSAT GRS in moderate stakes examinations when evaluating preclinical third‐year veterinary students' technical surgical skills on low‐fidelity models.

Surgical skill and oncological outcome of laparoscopic radical hysterectomy: JGOG1081s-A1, an ancillary analysis of the Japanese Gynecologic Oncology Group Study JGOG1081

ObjectivesWe investigated whether surgical skill and procedure were related to oncological outcomes in cervical cancer patients who underwent Laparoscopic Radical Hysterectomy (LRH). MethodsWe previously assessed data of LRH from 251 patients with FIGO stage (2009) IA2, IB1and IIA1 cervical cancer collected for JGOG 1081s study. 1) The JGOG 1081s cohort study was re-examined to refine the surgical details and extend the follow-up period as chart review. 2) Unedited videos for recurrent cases and matched non-recurrent control cases were newly compared by experts for various surgical skills and surgical procedures using the modified Objective Structured Assessment of Technical Skills (OSATS) tool, without awareness of the recurrence status as video review. ResultsAfter a median follow-up of 46 months, tumors had recurred in 31 of the 251 patients. The five-year Recurrence-Free Survival rate was 86.9% (81.8–90.6) and five-year Overall Survival rate was 93.7% (87.5–96.8). Multivariate analysis from chart reviews found that an experience with LRH of less than 20 cases per institution was an independent prognostic factor for recurrence (Hazard Ratio (HR) 2.49, 95%CI 1.12–5.53, p = 0.025). For the surgical video review, we compared 23 videos of recurrent cases with 23 background-matched non-recurrent controls. Lower modified OSATS scores from the video review were consistently trended to have a higher risk of recurrence. ConclusionsOur new study has found that LRH surgical experience and skill trended to have better oncological outcomes.

Non-technical skills simulation-based training model for managing intraoperative posterior capsule rupture during cataract surgery.

To develop and implement a simulation-based training model for the management of posterior capsule rupture (PCR) from a non-technical skills (NTS) perspective, and analyse changes in participant's NTS and technical skills (TS). The simulation-based training model consisted of two identical PCR simulations with NTS stressors applied, separated by a predominantly NTS focussed training intervention. Participants' TS and NTS were evaluated by two blinded assessors using the Objective Structured Assessment of Technical Skill (OSATS) global rating scale and the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES) respectively. Paired t-tests were used to establish the difference in mean HUFOES and OSATS scores between initial and repeat simulations; p < 0.05 indicated statistical significance. McGaghie's model of translational outcomes for simulation-based learning was used to establish the simulation model's educational status. Seventeen cataract surgeons of varying training grades participated in the simulation-based training model. NTS improved with statistical significance; mean HUFOES scores increased from 48.7 ± 16.6 to 59.2 ± 14.8 (p < 0.001). Mean OSATS scores increased without statistical significance from 16.0 ± 7.3 to 17.9 ± 8.3 (p = 0.07). This simulation model achieved Level 1 (internal acceptability) and Level 2 (contained effects) according to McGaghie's model. This novel simulation-based training model was designed to improve the NTS required for managing intraoperative PCR, through the provision of an interactive training session. Statistically significant improvements in participants' NTS in combination with statistically insignificant improvements in TS demonstrate that the simulation-based training model has specificity within the NTS domain.

The role of motion tracking in assessing technical skill acquisition using a neonatal 3D-printed thoracoscopic esophageal atresia/tracheo-esophageal fistula simulator

IntroductionAcquiring the technical skills required for thoracoscopic repair of esophageal atresia with tracheo-esophageal fistula (EA/TEF) is challenging. A high-fidelity 3D-printed pediatric thoracoscopic EA/TEF simulator has been developed to address this issue. This study explored motion-tracking as an assessment tool to distinguish between surgeons of different expertise using the simulator. MethodsParticipants performed a single intracorporeal suture between the esophageal ends in EA with TEF. Total relative path lengths of the right and left surgical instruments were recorded during the task. Each video-recorded attempt was assessed by a blinded pediatric surgeon using a modified Objective Structured Assessment of Technical Skills (OSATS) score. Data recorded as median (range) and statistical significance as p<0.05. ResultsThe task was performed by 17 participants. The median OSATS scores identified a significant difference between experts and novices. A difference between left- and right-hands was only found in the mid-skill level group. Right-hand path length was greatest in novices and lowest in experts. Left-hand path length was greatest in novices and the mid-skill level group compared to experts. ConclusionExperts had the lowest total path length for either hand, suggesting they had the greatest efficiency of movement. The similar high path lengths in both hands for novices indicate their relatively low level of skill with either hand. The difference between right- and left-hand path lengths in the mid-skill level group likely reflects the improved right-handed technical skills in contrast to the still developing left hand. Further focus on the left hand during simulation training may improve left-handed economy of movement.

Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?

Simulation using virtual reality (VR) simulators is an important tool in surgical training. VR laparoscopic simulators can provide immediate objective performance assessment without observer evaluation. This study aims to explore the correlation between subjective observer evaluation and VR laparoscopic simulator performance metrics in a laparoscopic cholecystectomy (LC) simulation module. A LC simulation module using a VR laparoscopic simulator was completed by PGY2-3 general surgery residents at a single institution. Simulation performance was recorded and evaluated by a trained evaluator using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) form, the Objective Structured Assessment of Technical Skills (OSATS) form, and a LC-specific simulation assessment form (LC-SIM). Objective performance metrics were also obtained from the simulator system. Performance before the curriculum (pre-test) and after the curriculum (post-test) were compared. Fourteen residents were included in the study. There were significant improvements from pre-test to post-test on each component of GOALS, OSATS, and LC-SIM scores (all p values < 0.05). In terms of objective simulator metrics, significant improvements were noted in time to extract gallbladder (481 ± 221 vs 909 ± 366min, p = 0.019), total number of movements (475 ± 264 vs 839 ± 324min, p = 0.012), and total path length (955 ± 475 vs 1775 ± 632cm, p = 0.012) from pre-test to post-test. While number of movements and total path lengths of both hands decreased, speed of right instrument also decreased from 4.1 + 2.7 to 3.0 ± 0.7cm/sec (p = 0.007). Average speed of left instrument was associated with respect for tissue (r = 0.60, p < 0.05) and depth perception (r = 0.68, p < 0.05) on post-test evaluations. Our study demonstrated significant improvement in technical skills based on subjective evaluator assessment as well as objective simulator metrics after simulation. The few correlations identified between the subjective evaluator and the objective simulator assessments suggest the two evaluation modalities were measuring different aspects of the technical skills and should both be considered in the evaluation process.