Assessment Of Technical Skills Research Articles

807 Reliability of Simulation-Based Assessment of Surgical Technical Skills

Abstract Aim To examine the reliability of surgical trainees’ assessment data from a structured simulation training and assessment programme. Method The General Surgical Skills Programme is a simulation-based surgical training programme integrated within the clinical training pathway for surgical trainees across London. It is tailored to each trainee’s level of training and subspecialty. The ‘Junior’ pathway involves nine training sessions and one assessment session performing three procedures (Inguinal hernia [IH], laparoscopic cholecystectomy [LC] and small bowel anastomoses [SBA]). Within the scope of the Programme, skills assessment data collection was initiated in 2012. Data collected includes demographics and training data (grade/subspecialty/case numbers). Each procedure is assessed by two consultant surgeons using two procedure specific assessment tools, a visual analogue scale (VAS) and competency assessment tool (CAT). Reliability analyses were performed using SPSS for ‘Junior’ data over a four-year period. Results In total, 355 surgical trainees (Male = 199, Female = 154, Missing = 2) were assessed from 2014 to 2017 (2014 n=105; 2015 n=92; 2016 n=89; 2017 n=69). Both assessment tools showed satisfactory inter-item reliability (IH-CAT α=.794-.903, LC-CAT α=.837–910, IH-VAS α=.851-.991, LC-VAS α=.831-.986, SBA-VAS α=.778-.984). Inter-rater reliability, ICC (1,2), was good to poor across all tools. (IH-CAT .345-.510, IH-VAS .243-.600, LC-CAT .271-.766, LC-VAS .157-.678, SBA-VAS .225-.278). Conclusions The assessment tools used within the Programme both demonstrate satisfactory inter-item reliability. Inter-rater reliability is variable, highlighting a potential utility for assessor training within the Programme.

809 A Systematic Review of Quality Assurance Processes in Simulated Surgical Training Programmes

Abstract Aim To provide a comprehensive overview of quality assurance processes implemented within published simulated surgical training programmes, identifying areas of deficit. Method A systematic review was conducted according to PRISMA guidelines. Databases were searched from 1947 until April 2023 to identify studies reporting structured simulation programmes undertaking qualitative assessment of laparoscopic technical skills. The assessment had to be procedural (e.g., appendicectomy) versus tasks (e.g., bean drop). Simulation had to be physical modalities (synthetic/cadaveric). Virtual and augmented reality modalities were excluded. Data was extracted against an existing quality assurance standards framework for simulation-based education, developed by the Association for Simulated Practice in Healthcare (ASPiH) in partnership with Health Education England. Study quality was appraised using the Medical Education Research Study Quality Instrument (MERSQI). Results 39 studies were included, with 17 from USA and 3 from UK simulation programmes. Range of participants was 6-256, with 7 studies reporting >50 participants. Faculty: 40% (n=15) of studies reported no description of faculty expertise or selection. 60% (n=24) of studies implemented no assessor training. 90% (n=35) described no faculty evaluation or feedback. Activity: Variability was seen in assessment methods including number of stations, assessors, and assessment tools. 60% (n=24) of studies sought no feedback on their programme. 90% (n=35) of the programmes were not aligned to curricula or learning needs analyses. Conclusions This review highlights deficits in the implementation of quality assurance processes including faculty selection, training, and programme feedback. It also highlights the variability of assessment methods, reiterating the need for individualised robust validation processes.

Robotic hepaticojejunostomy training in novices using robotic simulation and dry-lab suturing (ROSIM): randomized controlled crossover trial

BackgroundRobotic suturing training is in increasing demand and can be done using suture-pads or robotic simulation training. Robotic simulation is less cumbersome, whereas a robotic suture-pad approach could be more effective but is more costly. A training curriculum with crossover between both approaches may be a practical solution. However, studies assessing the impact of starting with robotic simulation or suture-pads in robotic suturing training are lacking.MethodsThis was a randomized controlled crossover trial conducted with 20 robotic novices from 3 countries who underwent robotic suturing training using an Intuitive Surgical® X and Xi system with the SimNow (robotic simulation) and suture-pads (dry-lab). Participants were randomized to start with robotic simulation (intervention group, n = 10) or suture-pads (control group, n = 10). After the first and second training, all participants completed a robotic hepaticojejunostomy (HJ) in biotissue. Primary endpoint was the objective structured assessment of technical skill (OSATS) score during HJ, scored by two blinded raters. Secondary endpoints were force measurements and a qualitative analysis. After training, participants were surveyed regarding their preferences.ResultsOverall, 20 robotic novices completed both training sessions and performed 40 robotic HJs. After both trainings, OSATS was scored higher in the robotic simulation-first group (3.3 ± 0.9 vs 2.5 ± 0.8; p = 0.049), whereas the median maximum force (N) (5.0 [3.2–8.0] vs 3.8 [2.3–12.8]; p = 0.739) did not differ significantly between the groups. In the survey, 17/20 (85%) participants recommended to include robotic simulation training, 14/20 (70%) participants preferred to start with robotic simulation, and 20/20 (100%) to include suture-pad training.ConclusionSurgical performance during robotic HJ in robotic novices was significantly better after robotic simulation-first training followed by suture-pad training. A robotic suturing curriculum including both robotic simulation and dry-lab suturing should ideally start with robotic simulation.

Real-Time multifaceted artificial intelligence vs In-Person instruction in teaching surgical technical skills: a randomized controlled trial

Trainees develop surgical technical skills by learning from experts who provide context for successful task completion, identify potential risks, and guide correct instrument handling. This expert-guided training faces significant limitations in objectively assessing skills in real-time and tracking learning. It is unknown whether AI systems can effectively replicate nuanced real-time feedback, risk identification, and guidance in mastering surgical technical skills that expert instructors offer. This randomized controlled trial compared real-time AI feedback to in-person expert instruction. Ninety-seven medical trainees completed a 90-min simulation training with five practice tumor resections followed by a realistic brain tumor resection. They were randomly assigned into 1-real-time AI feedback, 2-in-person expert instruction, and 3-no real-time feedback. Performance was assessed using a composite-score and Objective Structured Assessment of Technical Skills rating, rated by blinded experts. Training with real-time AI feedback (n = 33) resulted in significantly better performance outcomes compared to no real-time feedback (n = 32) and in-person instruction (n = 32), .266, [95% CI .107 .425], p < .001; .332, [95% CI .173 .491], p = .005, respectively. Learning from AI resulted in similar OSATS ratings (4.30 vs 4.11, p = 1) compared to in-person training with expert instruction. Intelligent systems may refine the way operating skills are taught, providing tailored, quantifiable feedback and actionable instructions in real-time.

Association of Crowd-Sourced Assessment of Technical Skills and Outcomes of Robotic-assisted Radical Prostatectomy

ObjectiveTo investigate if use of the Crowd-Sourced Assessment of Technical Skills (CSATSTM) platform and video peer review with constructive feedback is associated with improvement in technical skill and patient outcomes for robotic-assisted laparoscopic prostatectomy (RALP). MethodsFive fellowship-trained urologists voluntarily submitted RALP cases for CSATSTM Global Evaluative Assessment of Robotic Skills (GEARS) scoring and expert narrative review between April 15, 2022 - April 30, 2023. Surgeon-selected and randomly-selected cases were reviewed. Surgeons underwent local peer review of videos with constructive feedback. Change in GEARS scores and frequency of post-operative outcomes over the 12-month periods before and during the study were analyzed in logistic regression models. Bias was assessed with sensitivity analysis comparing surgeon-selected to randomly-selected cases. ResultsGEARS scores for randomly-selected versus surgeon-selected cases did not differ significantly. Overall GEARS score correlated positively with annual surgical RALP volume (r=0.39, p=0.003) and negatively with years in practice (r=-0.34, p=0.01). After adjusting for confounders, there was no significant improvement in overall GEARS Score (0.01±0.02/month, p=0.48); but likelihood of sepsis (Odds Ratio 0.07, 95% CI 0.01-1.00, p=0.05) and pelvic fluid collection (Odds Ratio 0.09, 95% CI 0.01-0.99, p=0.049) were significantly decreased during the intervention period (n=165) compared to the prior 12 months (n=144). No outcome increased in likelihood (p>0.05). ConclusionsIntegration of CSATSTM and local video peer review is associated with significant improvement in patient outcomes after RALP, despite no significant change in surgeons’ GEARS scores. This is the first study demonstrating improvement in patient RALP outcomes after implementation of such a paradigm in practicing surgeons.

Implementation and Validation of a High-fidelity Simulation Model for Surgical Resident Training: Management of Acute Intraoperative Hemorrhage during Robot-assisted Surgery

ObjectivesTo improve our previous simulation-based training module by using sustainable material to mold an anatomically accurate terrain and reproducing major vascular injuries encountered during robot-assisted nephrectomy. MethodsThe simulator was built with a pump, gauge, and valve linked via silicone tubing. Artificial blood was made from cornstarch, water, and red dye, and pumped through 3D-Med artificial vessels with the dimensions of an average renal artery. Silicone was used to emulate the pliability of organic tissue and mold an anatomically accurate terrain. Eight urologic residents participated in the pilot simulation. We employed validated assessment tools including Non-Technical Skills for Surgeons and Objective Structured Assessment of Technical Skills forms to guide debrief sessions moderated by an expert physician after individual performance evaluations. ResultsThe apparatus demonstrated high reproducibility across all simulation scenarios, enhancing resident problem-solving skills. Residents’ pre-simulation surveys revealed significant concern regarding their acute hemorrhage management. Residents’ post-simulation survey demonstrated average realism scores increased from 4.375 to 4.75. Residents also felt the simulator enhanced learning, offering valuable practice and knowledge applicable to their surgical specialty. ConclusionsThe management of acute hemorrhage during robot-assisted surgery remains a space for additional surgical education and training. Our simulation successfully provided a reliable, reproducible training for residents to practice their technical and non-technical skills in managing acute hemorrhage.

Educational value of a novel telestration device for surgical coaching-a randomized controlled trial.

Communication is fundamental to effective surgical coaching. This can be challenging for training during image-guided procedures where coaches and trainees need to articulate technical details on a monitor. Telestration devices that annotate on monitors remotely could potentially overcome these limitations and enhance the coaching experience. This study aims to evaluate the value of a novel telestration device in surgical coaching. A randomized-controlled trial was designed. All participants watched a video demonstrating the task followed by a baseline performance assessment and randomization into either control group (conventional verbal coaching without telestration) or telestration group (verbal coaching with telestration). Coaching for a simulated laparoscopic small bowel anastomosis on a dry lab model was done by a faculty surgeon. Following the coaching session, participants underwent a post-coaching performance assessment of the same task. Assessments were recorded and rated by blinded reviewers using a modified Global Rating Scale of the Objective Structured Assessment of Technical Skills (OSATS). Coaching sessions were also recorded and compared in terms of mentoring moments; guidance misinterpretations, questions/clarifications by trainees, and task completion time. A 5-point Likert scale was administered to obtain feedback. Twenty-four residents participated (control group 13, telestration group 11). Improvements in some elements of the OSATS scale were noted in the Telestration arm but there was no statistical significance in the overall score between the two groups. Mentoring moments were more in the telestration Group. Amongst the telestration Group, 55% felt comfortable that they could perform this task independently, compared to only 8% amongst the control group and 82% would recommend the use of telestration tools here. There is demonstrated educational value of this novel telestration device mainly in the non-technical aspects of the interaction by enhancing the coaching experience with improvement in communication and greater mentoring moments between coach and trainee.

The influence of virtual reality simulation on surgical residents’ heart rate during an assessment of arthroscopic technical skills: A prospective, paired observational study

HypothesisTo demonstrate that a virtual reality (VR) simulation training program reduces heart rate variability during an assessment of surgical trainees’ technical skills in arthroscopy. Study designProspective observational matched study. Materials & methodsThirty-six orthopaedic surgery residents, new to arthroscopy, received standard training in arthroscopic knee surgery, supplemented by additional monthly training for 6months on a VR simulator for 16 of them. At inclusion, the 2 groups (VR and NON-VR) answered a questionnaire and performed a meniscectomy on a VR simulator. After 6months of training, two independent trainers blinded to the inclusion arms evaluated the technical skills of the two groups during meniscectomies on a model and on an anatomical subject. Heart rate variability (HRV) was measured using a wireless heart rate monitor during baseline, VR training, and assessment. ResultsAfter removing incomplete data, the analysis focused on 10 VR residents matched at inclusion with 10 NON-VR residents. The VR group had a significantly lower heart rate at the final assessment (p=0.02) and lower overall HRV (p=0.05). The low/high frequency ratio (LF/HF) was not significantly different between the groups (1.84 vs 2.05, p=0.66) but the before-after training comparison showed a greater decrease in this ratio in the VR group compared to the NON-VR group −0.76 (−41%) vs −0.08 (−4%). ConclusionThis study demonstrates a significant difference in heart rate variability between trained residents versus untrained residents during the final assessment of their technical skills at 6months. It appears that improving stress management should be an integral part of training programs in arthroscopic surgery. Clinical InterestVR simulators in arthroscopy could improve non-technical skills such as heart rate variability, from the perspective of accountability. Level of evidenceIII.

The use of household items to support online surgical knot-tying skills training: a mixed methods study

BackgroundThis study investigated the perceptions and performance of medical students regarding their engagement and learning of a knot-tying skill presented in an online demonstration format due to the emergency remote measures that accompanied COVID-19 restrictions.MethodsFinal-year undergraduate medical students were invited to view an online demonstration of a one-handed knot-tying skill and practice the skill using common household items. They recorded their attempts and uploaded them onto the Flipgrid application. Completed attempts were scored using an adapted Objective Structured Assessment of Technical Skill (OSATS) validated tool. We used a mixed-methods sequential explanatory design; data regarding students’ engagement was gathered via a short questionnaire, and a Focus Group Discussion (FGD) was conducted to understand their learning experiences better. Descriptive statistics such as proportions and percentages were used to summarize categorical variables and median for continuous variables. Each video attempt was scored independently by two surgeons; reliability was determined using intraclass correlation; statistical tests were conducted at a 5% level of significance. Responses to open-ended survey questions and qualitative data from the FGDs were analysed using thematic analysis.ResultsSeventy-one students participated in the exercise. Most students (91.5%) expressed confidence in their ability to perform the skill and were able to follow the steps in the video demonstration (83.1%). Median number of times needed to practice before video submission was 7.0 (Interquartile range [IQR] 5.0–10.0). Using the adapted OSATS tool; median scores on student attempts were 19.0 out of 21 (IQR: 17.0–20.0) for Assessor 1 and 18.0 out of 21 (IQR: 17.0–20.0) for Assessor 2, and overall scores showed good reliability between assessors based on intraclass correlation (0.86, 95% CI 0.79–0.90, p < 0.001). Qualitative insights from the students’ experiences in learning the skill were generally positive; it was a practical, experiential learning process and they valued the social aspects of learning via Flipgrid. Challenges expressed related to the need for in-person training and formal feedback on how to improve their technique. Suggestions to improve their learning included a request for an interactive session with immediate feedback on attempts, and being able to practice with a friend who would assist with videoing.ConclusionBasic knot-tying can be taught with acceptable efficiency and student satisfaction using online methods with items available at home.

Vascular Anastomoses and Dissection: A Six-Part Simulation Curriculum for Surgical Residents.

As surgical technologies grow, so too do demands on surgical trainees to master increasing numbers of skill sets. With the rise of endovascular surgery, trainees have fewer opportunities to practice open vascular techniques in the operating room. Simulation can bridge this gap. However, existing published open vascular simulation curricula are basic or based on expensive models. We iteratively developed an open vascular skills curriculum for second-year surgery residents comprising six 2-hour sessions. We refined the curriculum based on feedback from learners and faculty. The curriculum required skilled facilitators, vascular instruments, and tissue models. We evaluated the latest iteration with a survey and by assessing participants' technical skills using the Objective Structured Assessment of Technical Skills (OSATS) form. Over the past 10 years, 101 residents have participated in the curriculum. Nine of 13 residents who participated in the latest curricular iteration completed the survey. All respondents rated the sessions as excellent and strongly agreed that they had improved their abilities to perform anastomoses with tissue and prosthetic. Facilitators completed 18 OSATS forms for residents in the fifth and sixth sessions of the latest iteration. Residents scored well overall, with a median 26.5 (interquartile range: 24-29) out of a possible score of 35, with highest scores on knowledge of instruments. This simulation-based curriculum facilitates open vascular surgical skill acquisition among surgery residents. The curriculum allows residents to acquire critical vascular skills that are challenging to learn in an increasingly demanding operative setting.

How to integrate robotic training in surgical residency? An example of a 2-week robotic rotation.

General surgery residents should be proficiently trained in robotic surgery. However, there is currently no standardized robotic training curriculum. We aimed to evaluate two approaches to a robotic curriculum and how implementing a virtual reality (VR) simulation curriculum improves trainee robotic performance. From 2019 to 2022, two models of a robotic training curriculum were examined: an in-unit rotation (IUR) and a 2-week curriculum (2WR). The VR curriculum was completed using the da Vinci® Skill Simulator. The curriculum used a pre/post-test design. Residents completed a pre-test that consisted of 4 VR exercises (graded 0-100%) and 3 inanimate box trainer exercises (graded using modified Objective Structured Assessment of Technical Skills). Then, residents completed a VR curriculum of 23 modules. Following the curriculum, residents were given a post-test with the same pre-test exercises. Time necessary to complete the curriculum and compliance were recorded. Of the 11 residents who participated in the IUR, 4 completed the VR curriculum. Comparatively, 100% (n = 23) of residents in the 2WR completed the curriculum. Average time to complete the VR curriculum was 3.8h. After completion of the 2WR curriculum, resident performance improved from pre-test to post-test: VR test scores increased (160% vs 223%, p < 0.001), OSATS scores increased (15.0 vs 21.0, p < 0.001), and time to complete inanimate exercises decreased (1083 vs 756s, p = 0.001). Residents who mastered all modules had higher post-test VR scores (241% vs 214%, p = 0.024). General surgery residents demonstrated improved compliance with the 2WR. The VR curriculum improved resident robotic performance in both virtual and inanimate domains.

Role of surgical simulation on self-reported confidence level on cardiothoracic surgical trainees.

Simulated self-practice using simulation models could improve fine motor skills and self confidence in surgical trainees. The purpose of this study is to evaluate on self-reported confidence level in cardiothoracic surgical trainees by using surgical simulation models. We conducted a cross-sectional study on all surgeons (n=10) involved in MIS simulation training. All surgeons are required to perform on three minimally invasive surgery (MIS) procedures (Mitral Valve Repair, Mitral Valve Replacement and Aortic Valve Replacement). A questionnaire was designed based on two existing scales related to self-confidence, the surgical self-efficacy scale [SSES] and the perceived competency scale [PCS]. We assessed their self-confidence (before and after training) in the use of simulation in MIS procedures using rating scales 1-5. The mean score was calculated for each domain and used as the predictor variable. We also developed six questions (PCS) using Objective Structured Assessment of Technical Skills (OSAT) related to each domain and asked participants how confident they were after performing each MICS procedure. The mean score was 4.7 for all assessed domains, except "knowledge" (3.8). Surgeons who had performed one or more MIS procedures had higher scores (P<0.05). There was no correlation between the number of MIS procedures performed and self-confidence scores. The results indicate that the cardiac surgery training based on MIS simulation improves trainees and consultants in terms of the level of self-confidence. Although surgeons generally have high levels of self-confidence after simulation training in MIS cardiac procedures, there is still room for improvement with respect to technical skills related to the procedure itself and its results.

Technical skills assessment: The expert versus the algorithm.

Technical skills assessment: The expert versus the algorithm.

Structured feedback and operative video debriefing with critical view of safety annotation in training of laparoscopic cholecystectomy: a randomized controlled study

BackgroundThe learning curve in minimally invasive surgery (MIS) is lengthened compared to open surgery. It has been reported that structured feedback and training in teams of two trainees improves MIS training and MIS performance. Annotation of surgical images and videos may prove beneficial for surgical training. This study investigated whether structured feedback and video debriefing, including annotation of critical view of safety (CVS), have beneficial learning effects in a predefined, multi-modal MIS training curriculum in teams of two trainees.MethodsThis randomized-controlled single-center study included medical students without MIS experience (n = 80). The participants first completed a standardized and structured multi-modal MIS training curriculum. They were then randomly divided into two groups (n = 40 each), and four laparoscopic cholecystectomies (LCs) were performed on ex-vivo porcine livers each. Students in the intervention group received structured feedback after each LC, consisting of LC performance evaluations through tutor-trainee joint video debriefing and CVS video annotation. Performance was evaluated using global and LC-specific Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores.ResultsThe participants in the intervention group had higher global and LC-specific OSATS as well as global and LC-specific GOALS scores than the participants in the control group (25.5 ± 7.3 vs. 23.4 ± 5.1, p = 0.003; 47.6 ± 12.9 vs. 36 ± 12.8, p < 0.001; 17.5 ± 4.4 vs. 16 ± 3.8, p < 0.001; 6.6 ± 2.3 vs. 5.9 ± 2.1, p = 0.005). The intervention group achieved CVS more often than the control group (1. LC: 20 vs. 10 participants, p = 0.037, 2. LC: 24 vs. 8, p = 0.001, 3. LC: 31 vs. 8, p < 0.001, 4. LC: 31 vs. 10, p < 0.001).ConclusionsStructured feedback and video debriefing with CVS annotation improves CVS achievement and ex-vivo porcine LC training performance based on OSATS and GOALS scores.

428 Artificial Intelligence Training Versus In-person Expert Training in Teaching Simulated Tumor Resection Skills - A Cross-Over Randomized Controlled Trial

INTRODUCTION: Neurosurgical simulations equipped with artificial intelligence systems provide an objective quantitative assessment of surgical technical skills and tailored intelligent feedback. These systems allow for repeated practice of complex skills such as subpial brain tumor resection. However, more work is needed to assess the utility of intelligent systems in teaching tumor resection skills, compared to traditional human instruction. METHODS: Twenty-five trainees who are currently enrolled in four Canadian medical schools participated in two training sessions, during which they completed a simulated subpial tumor resection five times. Participants were randomly assigned to two feedback groups: (1) real-time intelligent instruction, and (2) in-person human instruction. They were then assigned to the other feedback group in the second (cross-over) session. A composite-score was given by the intelligent system to assess technical skills during each task. RESULTS: Trainees who received real-time intelligent instruction significantly improved their composite-score in the first and second training sessions (p = .017, p = .005, respectively). Trainees who received in-person human instruction in the first training session had no statistically significant changes in their composite-score from the first to the last task repetition (p = .119). The composite-score decreased significantly in the second training session with in-person human instruction (p = .004). CONCLUSIONS: Real-time intelligent instruction provides an effective way of teaching simulated brain tumor resection skills. Artificial intelligence systems may be a useful addition to current surgical training curricula, providing objective, and tailored assessment and teaching of surgical technical skills in risk-free realistically simulated patient cases. Educators can benefit from these systems to guide their supervision, while trainees can use them for self-guided skill acquisition. Further research may assess the generalizability of these findings to other surgical procedures.

Testing the validity of three acute care assessment tools for assessing residents' performance during in situ simulation: the ACAT-SimSit study.

The assessment of technical and nontechnical skills in emergency medicine requires reliable and usable tools. Three Acute Care Assessment Tools (ACATs) have been developed to assess medical learners in their management of cardiac arrest (ACAT-CA), coma (ACAT-coma) and acute respiratory failure (ACAT-ARF). This study aims to analyze the reliability and usability of the three ACATs when used for in situ (bedside) simulation. This prospective multicenter validation study tested ACATs using interprofessional in situ simulations in seven emergency departments and invited training residents to participate in them. Each session was rated by two independent raters using ACAT. Intraclass correlation coefficients (ICC) were used to assess interrater reliability, and Cronbach's alpha coefficient was used to assess internal consistency for each ACAT. The correlation between ACATs' scores and the learners' level of performance was also assessed. Finally, a questionnaire and two focus groups were used to assess the usability of the ACATs. A total of 104 in situ simulation sessions, including 85 residents, were evaluated by 37 raters. The ICC for ACAT-CA, ACAT-coma and ACAT-ARF were 0.95 [95% confidence interval (CI), 0.93-0.98], 0.89 (95% CI, 0.77-0.95) and 0.92 (95%CI 0.83-0.96), respectively. The Cronbach's alphas were 0.79, 0.80 and 0.73, respectively. The ACAT-CA and ARF showed good construct validity, as third-year residents obtained significantly higher scores than first-year residents ( P < 0.001; P < 0.019). The raters supported the usability of the tools, even though they expressed concerns regarding the use of simulations in a summative way. This study reported that the three ACATs showed good external validity and usability.

Hip Arthroscopy Simulator Training With Immersive Virtual Reality Has Similar Effectiveness to Nonimmersive Virtual Reality

PurposeThe purpose of this study was to (1) compare the efficacy of immersive virtual reality (iVR) to non-immersive virtual reality (non-iVR) training in hip arthroscopy on procedural and knowledge-based skills acquisition and (2) evaluate the relative cost of each platform. MethodsFourteen orthopaedic surgery residents were randomized to simulation training utilizing an iVR Hip Arthroscopy Simulator (n=7; PrecisionOS, Vancouver, Canada) or non-iVR simulator (n=7; ArthroS Hip VR; VirtaMed, Zurich, Switzerland). After training, performance was assessed on a cadaver by four expert hip arthroscopists through arthroscopic video review of a diagnostic hip arthroscopy. Performance was assessed using the Objective Structured Assessment of Technical Skills (OSATS) and Arthroscopic Surgery Skill Evaluation Tool (ASSET) scores. A cost analysis was performed using the transfer effectiveness ratio (TER) and a direct cost comparison of iVR to non-iVR. ResultsDemographic characteristics did not differ between treatment arms, by training level, hip arthroscopy experience, or prior simulator use. No significant differences were observed in OSATS and ASSET scores between iVR and non-iVR cohorts (OSATS: iVR 19.6±4.4, non-iVR 21.0±4.1, p=0.55; ASSET: iVR 23.7±4.5, non-iVR 25.8±4.8, p=0.43). The absolute TER was 0.06 and there was a 132-fold cost difference of iVR to non-iVR. ConclusionsHip arthroscopy simulator training with iVR had similar performance results to non-iVR for technical skill and procedural knowledge acquisition after expert arthroscopic video assessment. The iVR platform had similar effectiveness in transfer-of-skill compared to non-iVR with a 132-times cost differential.

Performance in Irish Selection and Future Performance in Surgical Training

Selection processes for surgical training should aim to identify candidates who will become competent independent practitioners and should aspire to high standards of reliability and validity. To determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training, examined via rate of progression to Higher Specialist Training (HST), attrition rates, and performance as assessed through a multimodal framework of workplace-based and simulation-based assessments. This retrospective observational cohort study included data from all successful applicants to the Royal College of Surgeons in Ireland (RCSI) national Core Surgical Training (CST) program. Participants included all trainees recruited to dedicated postgraduate surgical training from 2016 to 2020. These data were analyzed from July 11, 2016, through July 10, 2022. Selection decisions were based on a composite score that was derived from technical aptitude assessments, undergraduate academic performance, and a 4-station multiple mini-interview. Assessment data, attrition rates, and rates of progression to HST were recorded for each trainee. CST performance was assessed using workplace-based and simulation-based technical and nontechnical skill assessments. Potential associations between selection and assessment measures were explored using Pearson correlation, logistic regression, and multiple linear-regression analyses. Data were available for 303 trainees. Composite scores were positively associated with progression to HST (odds ratio [OR], 1.09; 95% CI, 1.05-1.13). There was a weak positive correlation, ranging from 0.23 to 0.34, between scores and performance across all CST assessments. Multivariable linear regression analysis showed technical aptitude scores at application were associated with future operative performance assessment scores, both in the workplace (β = 0.31; 95% CI, 0.14-0.48) and simulated environments (β = 0.57; 95% CI, 0.33-0.81). There was evidence that the interpersonal skills interview station was associated with future performance in simulated communication skill assessments (β = 0.55; 95% CI, 0.22-0.87). In this study, performance at the time of Irish national selection, measured across technical and nontechnical domains in a multimodal fashion, was associated with future performance in the workplace and in simulated environments. Future studies will be required to explore the consequential validity of selection, including potential unintended effects of selection and ranking on candidate performance.

Comparing the impact of three-dimensional digital visualization technology versus traditional microscopy on microsurgeons in microsurgery: a prospective self-controlled study.

Emerging three-dimensional digital visualization technology (DVT) provides more advantages than traditional microscopy in microsurgery; however, its impact on microsurgeons' visual and nervous systems and delicate microsurgery is still unclear, which hinders the wider implementation of DVT in digital visualization for microsurgery. Forty-two microsurgeons from the Zhongshan Ophthalmic Center were enrolled in this prospective self-controlled study. Each microsurgeon consecutively performed 30min conjunctival sutures using a three-dimensional digital display and a microscope, respectively. Visual function, autonomic nerve activity, and subjective symptoms were evaluated before and immediately after the operation. Visual functions, including accommodative lag, accommodative amplitude, near point of convergence and contrast sensitivity function (CSF), were measured by an expert optometrist. Heart rate variability was recorded by a wearable device for monitoring autonomic nervous activity. Subjective symptoms were evaluated by questionnaires. Microsurgical performance was assessed by the video-based Objective Structured Assessment of Technical Skill (OSATS) tool. Accommodative lag decreased from 0.63 (0.18) diopters (D) to 0.55 (0.16) D ( P =0.014), area under the log contrast sensitivity function increased from 1.49 (0.15) to 1.52 (0.14) ( P =0.037), and heart rate variability decreased from 36.00 (13.54) milliseconds (ms) to 32.26 (12.35) ms ( P =0.004) after using the DVT, but the changes showed no differences compared to traditional microscopy ( P >0.05). No statistical significance was observed for global OSATS scores between the two rounds of operations [mean difference, 0.05 (95% CI: -1.17 to 1.08) points; P =0.95]. Subjective symptoms were quite mild after using both techniques. The impact of DVT-based procedures on microsurgeons includes enhanced accommodation and sympathetic activity, but the changes and surgical performance are not significantly different from those of microscopy-based microsurgery. Our findings indicate that short-term use of DVT is reliable for microsurgery and the long-term effect of using DVT deserve more consideration.

Reducing trainee mistakes. Better performance with changing to a high-fidelity simulation system?

Postpartum hemorrhage is a significant cause of both maternal morbidity and mortality worldwide and is increasing in incidence. This study aimed to assess improvement and identify shortcomings in trainee performance in different simulation systems in the management of postpartum hemorrhage. To perform a pilot study evaluating and comparing high- and low-fidelity simulation models, assessing improvement in repeated performance with high-fidelity mode and identifying mistakes made assessed using Objective Structured Assessment of Technical Skills and thereby exploring what aspects of emergency management of postpartum hemorrhage should be prioritized in teaching settings and assessing what simulation setup is most effective in achieving competence. This was a prospective randomized, single-blinded, single-institution trial in a population of 17 junior obstetrical trainees at the Charité University Hospital Obstetric Simulation Center in Berlin. Trainees were randomized into 2 groups, with either initial low-fidelity simulation or high-fidelity simulation, followed by repeated assessment of performance, using the high-fidelity model simulation system. Individual simulation sessions were video-recorded and transcribed, and the timing of interventions was documented. Strandardized Objective Structured Assessment of Technical Skills forms were used as a checklist for performance. There was a statistically significant general improvement in performance (P=.02; 24.7-27.2 of 31.0 points; average of 8.7%) in the second cycle of simulation assessment and a statistically significant training effect (P=.043; 24.4-28.4 of 31.0 points; average of 12.9%) in the group that underwent repeat simulation assessment from the initial low-fidelity system to the high-fidelity system compared with the group using the same high-fidelity setup (P=.276; 25.0-25.8 of 31.0; average of 2.4%). There was an improvement in the performance when trainees underwent a repeated cycle of simulation assessment changing from a low-fidelity system to a high-fidelity system. Simulation assessment can identify mistakes and learning gaps that are important for obstetrical trainees. This study found that trainees make the same mistakes, regardless of which simulation model was initially used.