Expert Instruction Research Articles

Study preferences and exam outcomes in medical education: insights from renal physiology

BackgroundEfficient learning strategies and resource utilization are critical in medical education, especially for complex subjects like renal physiology. This is increasingly important given the rise in chronic renal diseases and the decline in nephrology fellowships. However, the correlations between study time, perceived utility of learning resources, and academic performance are not well-explored, which led to this study.MethodsA cross-sectional survey was conducted with second-year medical students at the University of Bergen, Norway, to assess their preferred learning resources and study time dedicated to renal physiology. Responses were correlated with end-of-term exam scores.ResultsThe study revealed no significant correlation between time spent studying and overall academic performance, highlighting the importance of study quality over quantity. Preferences for active learning resources, such as Team-Based Learning, interactive lessons and formative assignments, were positively correlated with better academic performance. A notable correlation was found between students’ valuation of teachers’ professional competence and their total academic scores. Conversely, perceived difficulty across the curriculum and reliance on self-found online resources in renal physiology correlated negatively with academic performance. ‘The Renal Pod’, a locally produced renal physiology podcast, was popular across grades. Interestingly, students who listened to all episodes once achieved higher exam scores compared to those who listened to only some episodes, reflecting a strategic approach to podcast use. Textbooks, while less popular, did not correlate with higher exam scores. Despite the specific focus on renal physiology, learning preferences are systematically correlated with broader academic outcomes, reflecting the interconnected nature of medical education.ConclusionThe study suggests that the quality and strategic approaches to learning significantly impact academic performance. Successful learners tend to be proactive, engaged, and strategic, valuing expert instruction and active participation. These findings support the integration of student-activating teaching methods and assignments that reward deep learning.

Real-Time multifaceted artificial intelligence vs In-Person instruction in teaching surgical technical skills: a randomized controlled trial

Trainees develop surgical technical skills by learning from experts who provide context for successful task completion, identify potential risks, and guide correct instrument handling. This expert-guided training faces significant limitations in objectively assessing skills in real-time and tracking learning. It is unknown whether AI systems can effectively replicate nuanced real-time feedback, risk identification, and guidance in mastering surgical technical skills that expert instructors offer. This randomized controlled trial compared real-time AI feedback to in-person expert instruction. Ninety-seven medical trainees completed a 90-min simulation training with five practice tumor resections followed by a realistic brain tumor resection. They were randomly assigned into 1-real-time AI feedback, 2-in-person expert instruction, and 3-no real-time feedback. Performance was assessed using a composite-score and Objective Structured Assessment of Technical Skills rating, rated by blinded experts. Training with real-time AI feedback (n = 33) resulted in significantly better performance outcomes compared to no real-time feedback (n = 32) and in-person instruction (n = 32), .266, [95% CI .107 .425], p < .001; .332, [95% CI .173 .491], p = .005, respectively. Learning from AI resulted in similar OSATS ratings (4.30 vs 4.11, p = 1) compared to in-person training with expert instruction. Intelligent systems may refine the way operating skills are taught, providing tailored, quantifiable feedback and actionable instructions in real-time.

Simulation based CPR Training of EMS Staff and its Utility in Managing Remote Patients at Tertiary Care Hospital of Central India

Abstract Introduction: AVBRH is 1500-bedded tertiary care hospital that caters to patients from areas within more than a 50 km radius. Emergency medical services (EMS) are established and frequently on the run. The EMS team has 10 basic life support (BLS) ambulances with 20 drivers and 20 paramedical staff. Daily, 3–5 highly critical patients who are in need of cardiopulmonary resuscitation (CPR) are transferred to the AVBRH hospital. The optimal medical care while transferring the patient to AVBRH hospital was unmonitored and without any benchmark. A low-risk simulation-based environment to practice can be valuable to maintain a wide array of skills. Effective and prompt delivery of expert instruction and guidance by a telesimulation is approved to be a beneficial approach in this setting. Training of EMS staff helps to improve patient care and satisfaction. Hence, we decided to undertake the simulation-based CPR training of the staff and correlate the outcome with existing mortality data of Medical record department (MRD). Objective: To optimize health care while transferring the patient to a tertiary hospital by BLS training and teleassisting. To see the impact of teaching and assisting on overall patient outcomes at the emergency department. Methodology: Forty EMS participants were enrolled and trained for 2 days. On day 1, they were trained in BLS, while on day 2, training in working as a team through simulation at school of virtual learning was done. They were trained to work as a team through simulated case scenarios with effective telecommunication while transferring the patient to the hospital. After training, these EMS staff were monitored and assisted according to the protocol of BLS by the emergency medicine department through telecommunication on the field for the next 3 months and reassessed again after the completion of 3 months. The following tabulated assessment format is used for EMS staff attending to the patient on the field. The prefield assessment on the same day of training and on postfield assessment after 3 months at school of virtual learning-simulation lab (SVL) was conducted. The data were gathered, tabulated, and analyzed for the percentage. Conclusion and Recommendation: Simulation-based teaching by a facilitator along with effective telesimulation creates an important link and gaps in the bridge between knowledge and technical skills. It has been found that overall patient care and performance of EMS staff was significantly improved; hence, recommendations for continuous assistance while transferring the patients are given. Training of the newly appointed EMS crew and the other available EMS services from the district should be aligned with assisted telecommunication services of AVBRH. Regular audit for quality improvement in EMS performance to be imparted.

Feeling the heat: undergraduate science students’ emotional management during classroom debates

ABSTRACT Addressing a need to prepare the next generation of scientists to effectively engage in adversarial science communication, the present study examines a group of undergraduate science students from a Canadian university who, after receiving expert instruction, participated in classroom debates about science controversies recently politicized in the Canadian social media (e.g. the flat Earth, genetically-modified foods, and human overpopulation). Our research questions were: (1) What emotions were experienced and how were these managed by students while participating in classroom debates? (2) How did students’ emotional management influence their debate performance? A video-based micro-ethnography revealed that more than half of the students (16/28) experienced feelings of stress and nervousness when engaging debaters with opposing/disagreeing views. Although some were able to manage these emotions, others were unable to feel relaxed, which negatively influenced their debate performance. These latter students’ initial confidence and preparation were undermined by their felt anxiety, leading to rhetorically weak and error-filled performances that went against their expectations. Highlighting the complexity of pedagogically promoting student development of communicative competence in adversarial social contexts, our findings reveal a need for science communication instructors to find ways to effectively prepare science students to manage their own emotions.

“Catastrophic”: A qualitative exploration of survivors experiences of expert instruction in private law child arrangements proceedings

ABSTRACTThis article explores the lived experiences of female survivors of domestic abuse when an expert had been instructed in private law family court proceedings in England and Wales. It considered survivors’ lived experiences from the moment that an expert was suggested to the expert report outcome. The study found that when the family court has made no findings of domestic abuse the court’s focus shifts from the perpetrator to a survivors’ behaviour. Survivors described that, despite their reservations, they considered the expert assessment to be the last opportunity for a professional to recognise the domestic abuse and change the court’s opinion. Survivors found that their natural brain’s response to trauma, due to abuse, was repackaged and (mis)diagnosed as disordered behaviour. Expert reports recommended psychological treatments that were often conditional to any future changes to child arrangements in their favour. Further, survivors’ lived experiences described how perpetrators were able to harness the legal mechanism of child removal and use it as a form of post-separation coercive control, often supported by the expert. Overall, survivors’ lived experiences raised serious professional and safeguarding concerns, as well as ethical issues about the role and impact of the expert in private law child arrangements proceedings.

Instructor Social Support in the Group Physical Activity Context: Older Participants' Perspectives.

Instructors in organized physical activity classes can be a source of social support through their relationships with participants, influence on participants' interactions with each other, and design of activities. Grounded in interpretive description, the objective of this study was to examine older adults' experiences of and their perspectives on group physical activity instructors' supportive behaviors. Observations of 16 group physical activity classes (N = 295) and focus groups or interviews with N = 38 class participants aged ≥ 55 (n = 29 women) were conducted at four municipal recreation facilities in a Canadian city. Five themes shed light on how instructors provided social support: (a)supporting autonomous engagement, (b)developing caring connections, (c)fostering trust through expert instruction, (d)managing conflict directly and effectively, and (e)creating a climate where people want to go. Instructor training should consider older adults' social support needs and help instructors embody behaviors that support continued physical activity participation, thereby contributing to healthy aging.

A synopsis of the SAGES ADOPT course for foregut surgery and its positive impact on surgeons' confidence levels.

The Acquisition of Data for Outcomes and Procedure Transfer (ADOPT) program was established by SAGES to develop and expand individual surgeon's comfort with specific, complex operations using hands-on teaching and longitudinal mentoring. The 2022-2023 Foregut (Dominating the Hiatus) section of the course focused on hiatal hernia dissection and gastric fundoplication techniques. Our aim was to describe the experience of surgeons who participated in the course. The hands-on component occurred in March 2022 at the SAGES annual meeting. Each expert mentor was matched to two participants. The mentors guided the surgeons through steps of a laparoscopic paraesophageal (PEH) hernia repair and fundoplication using a cadaveric model. Afterwards, monthly group webinars occurred and participants could receive individual coaching from their assigned mentor for a year. Each participant was given a pre-course survey with 3 and 12-month follow-up questionnaires. The majority of the 16 participants were employed in non-academic settings (87.5%). Years in practice ranged from 1 to 26, and 69% completed a fellowship. 100% completed the pre-course survey, and 53.8% responded to the 12-month post-course survey. Participant-reported effectiveness in performing a PEH hernia repair with fundoplication increased from 37.5% pre-course to 85.7% by the conclusion of the course. Confidence levels for the six core steps of the operation also increased: pre-course only 56-75% were confident with each step, this improved to 100% in four out of six steps. 85.7% said the course has changed their practice. Since inception, the ADOPT program has aimed to provide expert instruction for practicing surgeons to learn new techniques or improve their confidence in performing operations. The data for the 2022 ADOPT Foregut course shows that 1year of participation made a positive impact on these surgeons' practices. This helps to fill in the learning gap that occurs after formal surgical training ends.

828 Real-Time Artificial Intelligence Instruction in Comparison to Human Expert Instruction in Surgical Technical Skills Teaching – a Randomized Controlled Trial

Abstract Aim Surgical simulators equipped with artificial intelligence (AI) systems provide risk-free training on realistically simulated complex patient cases, objective performance assessment, and tailored error avoidance feedback. This study compared the efficacy of real-time AI instruction with human instructor-mediated training in surgical technical skills teaching. Method Medical students (n = 98) were randomly allocated into three feedback groups. Everyone repeated a virtual brain tumour resection five times. The first repetition was done without receiving feedback to assess their baseline performance. Students in Group 1 did not receive real-time feedback, and they were shown visual feedback only after each procedure based on expert benchmarks. Students in Group 2 were instructed in real-time by the AI system. After each task, the students were shown their error-video clips generated by this system alongside the expert-level demonstrations relating to each error. Students in Group 3 were instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated correction techniques. All participant data was scored using a composite-score. Statistical analysis was conducted to compare learning from the first to the last task repetition. Results Students in Group 2 and Group 3 significantly improved in the performance score by the third and second repetition, respectively (p&amp;lt;0.01, p = 0.01) compared to their baseline performance. Students in Group 2 achieved significantly higher scores than those in Group 3 in the fifth repetition (p&amp;lt;0.01). Conclusions Artificial intelligence systems may provide competency-based curricula in surgical training with efficient surgical technical skills teaching methodologies.

SP11.14 Evaluating the Efficacy and Cognitive Load of a Real-Time Intelligent Coaching System versus Human Expert Instruction in Surgical Technical Skills Training: A Randomized Controlled Study

Abstract Aims An artificial intelligence-driven tutoring system provides risk-free training on realistic simulated procedures and objective assessment to improve surgical technical skills. This study compared its teaching efficacy to human instructor training. Methods Ninety-eight medical students completed 5 simulated brain tumor resections, scored by the intelligent system. Participants were randomly sorted into 3 feedback groups: (1) no real-time feedback, (2) real-time intelligent instruction, (3) real-time human instruction. Each participant completed a base-line repetition without feedback. Group 1 received visual feedback based on expert benchmarks following each repetition. Group 2 was tutored by the intelligent system in real-time. After each repetition, the system provided the student’s error clips and correlating expert-level demonstrations. Group 3 was tutored by human instructors in real-time. After each repetition, instructors provided critiques and demonstrated correction techniques. Following the final repetition, each participant completed a questionnaire of Leppink’s Cognitive Load Index on a 5-point Likert scale. Performance scores were compared from the first to the last repetition of the task within and between-groups to assess learning. Results Compared with baseline, groups 2 and 3 significantly improved in performance by the third and second repetition, respectively (p&amp;lt;0.01 and p=0.01). Between-groups comparison revealed significantly higher scores and extrinsic load for group 2 than group 3 in the fifth repetition, respectively (p&amp;lt;0.01 and p&amp;lt;0.01). No between-group differences existed for intrinsic and germane load. Conclusion The intelligent system provided more efficient learning than human instruction. Such systems may enhance trainees’ learning and facilitate the shift towards standardized, competency-based surgical training.

Applying the Project ECHO Model to Support Implementation and Sustainment of Cognitive Behavioral Therapy for Psychosis.

Project Extension for Community Healthcare Outcomes (ECHO) is a teleconsultation model that leverages technology to sustain specialized interventions in underresourced settings. We present the application of the ECHO model to longitudinal training and consultation for community behavioral health providers learning to deliver cognitive behavioral therapy for psychosis, an evidence-based psychotherapy for individuals with psychotic disorders that has poorly penetrated the US mental health system. We analyzed within-group change over practitioners' 6-month ECHO participation cycle using the Expanded Outcomes Framework. We evaluated outcomes associated with participation, satisfaction, knowledge acquisition, performance, patient symptom severity, and functional impairment. In the first 3 years, the cognitive behavioral therapy for psychosis ECHO Clinics supported 150 providers from 12 community agencies. Forty percent did not complete the 6-month ECHO calendar, most commonly due to separation from their agency. Participants reported high degrees of satisfaction. Declarative and procedural knowledge increased over the 6-month period. Of the 24 providers who received a fidelity review, 87.5% met or exceeded the competency benchmark within the 6-month period. Clinical outcomes reflected reductions in hallucinations, negative symptoms, depression, mania, and functional impairment, but no reductions were detected in delusions, disorganized speech, or abnormal psychomotor behavior. ECHO Clinics offer a mode of providing continuous access to expert instruction, peer-to-peer consultation, and case-based learning that other workforce training models lack. Our evaluation suggests that the ECHO model supports continuous professional development for practitioners, most of whom had indicated inadequate preparation for their role. We observed improved learner and select patient outcomes.

LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity

AbstractCross‐task generalization is a significant outcome that defines mastery in natural language understanding. Humans show a remarkable aptitude for this, and can solve many different types of tasks, given definitions in the form of textual instructions and a small set of examples. Recent work with pre‐trained language models mimics this learning style: users can define and exemplify a task for the model to attempt as a series of natural language prompts or instructions. While prompting approaches have led to higher cross‐task generalization compared to traditional supervised learning, analyzing ‘bias’ in the task instructions given to the model is a difficult problem, and has thus been relatively unexplored. For instance, are we truly modeling a task, or are we modeling a user's instructions? To help investigate this, we develop LINGO, a novel visual analytics interface that supports an effective, task‐driven workflow to (1) help identify bias in natural language task instructions, (2) alter (or create) task instructions to reduce bias, and (3) evaluate pre‐trained model performance on debiased task instructions. To robustly evaluate LINGO, we conduct a user study with both novice and expert instruction creators, over a dataset of 1,616 linguistic tasks and their natural language instructions, spanning 55 different languages. For both user groups, LINGO promotes the creation of more difficult tasks for pre‐trained models, that contain higher linguistic diversity and lower instruction bias. We additionally discuss how the insights learned in developing and evaluating LINGO can aid in the design of future dashboards that aim to minimize the effort involved in prompt creation across multiple domains.

O022 Real-time artificial intelligence instructor vs expert instruction in teaching of expert level tumour resection skills – a randomized controlled trial

Abstract Introduction Competency-based approach in surgical training still lacks objective quantifiable methodologies to assess surgical technical skills and train residents, a limitation that can be addressed with the adaptation of artificial intelligence (AI). In this randomized controlled, the efficacy of learning by a real-time intelligent instruction system was compared to learning with in-person human instructor-mediated training. Methods The study was ethics approved. Ninety-eight medical students performed five virtually simulated brain tumour resections, randomly allocated into three feedback groups: (1) no-real-time feedback, (2) real-time intelligent instruction, and (3) in-person human instruction. The first task was considered as baseline performance, done with no feedback. Group-1 received expert benchmark feedback only after each procedure. Group-2 was instructed in real-time by the AI system. After each task, the students were shown their error-video clips generated by this system alongside the expert-level demonstrations relating to each error. Group-3 was instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated how to expertly perform the tumour resections. Participant data in all tasks were scored by the AI system to assess learning. Results Students in Group-2 and Group-3 significantly improved their performance score by the third and second task, respectively (p&amp;lt;0.01, p=0.01), compared to the baseline performance. Group-2 achieved significantly higher scores than Group-3 in the final/fifth task (p&amp;lt;0.01). Conclusion AI-powered systems may increase efficiency in learning by providing objective, and action-oriented real-time feedback. Such systems may aid the shift towards competency-based surgical curricula.

C-CASE 2022: Competence to Excellence01. The Queen Bee phenomenon in Canadian surgical subspecialties: an evaluation of gender biases in the resident training environment02. Barriers to surgical peer coaching — What have we learned, and where do we go from here?03. Shared decision-making and evidence-based medicine: Pivotal or trivial to patient care in orthopedic trauma?04. Immersive virtual reality and cadaveric bone

# 01. The Queen Bee phenomenon in Canadian surgical subspecialties: an evaluation of gender biases in the resident training environment {#article-title-2} The Queen Bee phenomenon describes the behavioural response that occurs when women achieve success in a male-dominated environment, and in this

Learning of Intracorporal Knot Tying in Minimally Invasive Surgery by Video or Expert Instruction.

Minimally invasive surgery skill laboratories are indispensable in training, especially for complex procedural skills such as intracorporal suturing and knot tying (ICKT). However, maintaining a laboratory is expensive, and specially trained teachers are in short supply. During the COVID-19 pandemic, in-person instruction has reduced to almost zero, while model learning via video instruction (VID) has become an integral part of medical education. The aim of this study was to compare the learning effectiveness and efficiency of ICKT by laparoscopically inexperienced medical students through video versus direct expert instruction. A secondary analysis of two randomized controlled trials was performed. We drew data from students who were trained in ICKT with expert instruction (EXP, n = 30) and from students who were trained via VID, n = 30). A laparoscopic box trainer including laparoscope was used for ICKT. Objective Structured Assessment of Technical Skills (OSATS), knot quality, and total ICKT time were the parameters for the assessment in this study. Proficiency criteria were also defined for these parameters. Students in the EXP group performed significantly better in OSATS-procedure-specific checklist (PSC) and knot quality compared with students in the VID group, with no difference in task time. Of the students who reached the proficiency criteria for OSATS-PSC and knot quality, those in the EXP group required fewer attempts to do so than those in the VID group. Students in both groups improved significantly in all parameters over the first hour of evaluation. For the laparoscopically inexperienced, training in ICKT through expert instruction presents an advantage compared with video-based self-study in the form of faster understanding of the procedure and the associated consistent implementation of good knot quality. Both teaching methods significantly improved participants' ICKT skills.

Use of surgical video-based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study.

Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery. Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined-instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions-on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials. In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001). Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

158 Virtual Interactive Surgical Skills Classroom: A Parallel-Designed, Non-Inferiority, Adjudicator-Blinded, Randomised Controlled Trial (VIRTUAL)

Abstract Introduction Virtual classroom training (VCT) is a novel educational method that permits accessible, distanced interactive expert instruction. We aimed to evaluate the efficacy of VCT in comparison to face-to-face training (FFT) and non-interactive computer-based learning (CBL) for basic surgical skills training. Method 72 participants recruited from five London medical schools underwent stratified block randomisation into three equal intervention groups based on subjective and objective suturing experience. VCT was delivered via the BARCO weConnect platform and FFT was provided by expert instructors. Optimal student-to-teacher ratio was used, 12:1 for VCT and 4:1 for FFT. The assessed task was interrupted suturing with hand-tied knots. The primary outcome was post-intervention Objective Structured Assessment of Technical Skills (OSATS) score, adjudicated by two blinded experts and adjusted for baseline proficiency. Results VCT was non-inferior to FFT (adjusted difference 0.44, 95% CI: -0.54 to 1.75, delta 0.675), VCT was superior to CBL (adjusted difference 1.69, 95% CI 0.41 to 2.96) and FFT was superior to CBL (adjusted difference 1.25, 95% CI 0.20 to 2.29). FFT alone was associated with student travel expenses (mean £4.88, SD 3.70). Instructor hours used per student for VCT and FFT were 0.25 and 0.75, respectively. Conclusions VCT has a similar educational benefit to FFT and is a suitable modality of high-quality surgical skills education. VCT provides greater accessibility and resource efficiency compared to FFT. VCT satisfies the requirement for social distancing during the COVID-19 pandemic and is better than non-interactive CBL. VCT has the potential to improve global availability and accessibility of surgical skills training.

510 Artificial Intelligence Tutoring Compared with Expert Instruction in Neurosurgical Simulation Training: A Randomized Controlled Trial

INTRODUCTION: Resection of subpial tumors adjacent to critical brain areas is one of the most high-stakes surgical procedures that lacks opportunities for safe deliberate practice. Furthermore, performance-based assessment in neurosurgical apprenticeship is inefficient and vulnerable to subjectivity. Utilizing Artificial Intelligence (AI) to classify neurosurgical psychomotor expertise, we developed the first AI-powered tutor in neurosurgical simulation training, known as the Virtual Operative Assistant (VOA), to augment technical skills acquisition training by providing intelligent feedback. METHODS: A multi-institutional randomized controlled trial compared VOA’s automated audiovisual metric-based feedback vs remote verbal debriefing with expert instruction and no-feedback controls. Medical students performed six simulated subpial brain tumor resections: five practice attempts followed with feedback and one complex realistic attempt evaluated skill retention and transfer. A deep learning model, Intelligent Continuous Expertise Monitoring System (ICEMS Expertise Score), and blinded Objective Structured Assessment of Technical Skills (OSATS) evaluated performance. Participants reported emotions before, during and after training and completed a cognitive load questionnaire following training. RESULTS: Seventy medical students from four institutions were randomly assigned to VOA (n=23), Instructor (n=24), and Control (n=23) Groups. 350 practice attempts were assessed by ICEMS, and 70 realistic attempts were evaluated by ICEMS and OSATS. During practice, VOA training resulted in a significant improvement of participants’ Expertise Scores that was on average 0.66 (95% CI 0.55-0.77) and 0.65 (95% CI 0.54-0.77) points higher than Instructor and Control Groups (p&lt;.001). Realistic attempt’s average Expertise Score was significantly higher in the VOA Group compared to Instructor and Control Groups (mean difference 0.53; 0.49, respectively, p&lt;.001). VOA and Instructor Group’s realistic attempt OSATS ratings were not significantly different. OSATS ratings demonstrated that VOA feedback resulted in significantly higher Respect for Tissue and Economy of Movement compared to Control while expert instruction significantly improved Instrument Handling compared to Control. There was no significant between-groups difference in cognitive load, positive-, and negative-activating emotions. CONCLUSION: VOA’s quantitative automated benchmark feedback demonstrated superior performance outcome, improved skill transfer, with equivalent OSATS ratings and similar cognitive and affective responses compared to remote expert instruction.

Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students

To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches. To determine how VOA and remote expert instruction compare in learners' skill acquisition, affective, and cognitive outcomes during surgical simulation training. This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021. The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback. The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, -1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports. A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found. In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training. ClinicalTrials.gov Identifier: NCT04700384.

EP.TU.566Virtual Interactive surgical skills classroom – An Ongoing Randomized Controlled Trial

IntroductionHigh costs and inaccessibility are significant barriers to face-to-face basic surgical skills (BSS) training. Virtual classrooms enable the combination of computer-based learning with interactive expert instruction. They may optimise resources and increase accessibility, facilitating larger-scale training with a similar educational benefit. We aim to evaluate the efficacy of virtual BSS classroom training compared to both non-interactive video and face-to-face teaching.Method72 medical students will be randomly assigned to three equal intervention groups based on year group and surgical skill confidence. Interventions will be implemented following an instructional video. Group A will practice independently, Group B will receive face-to-face training, and Group C will attend a virtual classroom. Participants will be recorded placing three interrupted sutures with hand tied knots pre- and post-intervention, and Objective Structured Assessment of Technical Skills (OSATS) will be blind marked by two experts. Change in confidence, time to completion and a granular performance score will also be measured. Each intervention’s feasibility and accessibility will be assessed.ResultsData collection will be completed in February 2021. Significant improvement in OSATS within groups will be indicative of intervention quality. Difference in improvement between groups will determine the relative performance of the interventions.ConclusionTo our knowledge, this will be the largest randomised control trial investigating virtual BSS classroom training. It will serve as a comprehensive appraisal of the virtual classroom’s suitability as an alternative to face-to-face training. The findings will assist the development of further resource-efficient training programs during the COVID-19 pandemic and afterwards.

81 VIRTUAL: Virtual Interactive Surgical Skills Classroom – An Ongoing Randomized Controlled Trial

Abstract Introduction High costs and inaccessibility are significant barriers to face-to-face basic surgical skills (BSS) training. Virtual classrooms enable the combination of computer-based learning with interactive expert instruction. They may optimise resources and increase accessibility, facilitating larger-scale training with a similar educational benefit. We aim to evaluate the efficacy of virtual BSS classroom training compared to both non-interactive video and face-to-face teaching. Method 72 medical students will be randomly assigned to three equal intervention groups based on year group and surgical skill confidence. Interventions will be implemented following an instructional video. Group A will practice independently, Group B will receive face-to-face training, and Group C will attend a virtual classroom. Participants will be recorded placing three interrupted sutures with hand tied knots pre- and post-intervention, and Objective Structured Assessment of Technical Skills (OSATS) will be blind marked by two experts. Change in confidence, time to completion and a granular performance score will also be measured. Each intervention’s feasibility and accessibility will be assessed. Results Data collection will be completed in January 2021. Significant improvement in OSATS within groups will be indicative of intervention quality. Difference in improvement between groups will determine the relative performance of the interventions. Conclusions To our knowledge, this will be the largest randomised control trial investigating virtual BSS classroom training. It will serve as a comprehensive appraisal of the virtual classroom’s suitability as an alternative to face-to-face training. The findings will assist the development and implementation of further resource-efficient training programs during the COVID-19 pandemic and in the future.