Expert Benchmark Research Articles

Evaluation of different approaches to define expert benchmark scores for new robotic training simulators based on the Medtronic HUGO™ RAS surgical robot experience.

New robot-assisted surgery platforms being developed will be required to have proficiency-based simulation training available. Scoring methodologies and performance feedback for trainees are currently not consistent across all robotic simulator platforms. Also, there are virtually no prior publications on how VR simulation passing benchmarks have been established. This paper compares methods evaluated to determine the proficiency-based scoring thresholds (a.k.a. benchmarks) for the new Medtronic Hugo™ RAS robotic simulator. Nine experienced robotic surgeons from multiple disciplines performed the 49 skills exercises 5 times each. The data were analyzed in 3 different ways: (1) include all data collected, (2) exclude first sessions, (3) exclude outliers. Eliminating the first session discounts becoming familiar with the exercise. Discounting outliers allows removal of potentially erroneous data that may be due to technical issues, unexpected distractions, etc. Outliers were identified using a common statistical technique involving the interquartile range of the data. Using each method above, mean and standard deviations were calculated, and the benchmark was set at a value of 1 standard deviation above the mean. In comparison to including all the data, when outliers are excluded, fewer data points are removed than just excluding first sessions, and the metric benchmarks are made more difficult by an average of 11%. When first sessions are excluded, the metric benchmarks are made easier by an average of about 2%. In comparison with benchmarks calculated using all data points, excluding outliers resulted in the biggest change making the benchmarks more challenging. We determined that this method provided the best representation of the data. These benchmarks should be validated with future clinical training studies.

Pediatric ECG-Based Deep Learning to Predict Left Ventricular Dysfunction and Remodeling.

Artificial intelligence-enhanced ECG analysis shows promise to detect ventricular dysfunction and remodeling in adult populations. However, its application to pediatric populations remains underexplored. A convolutional neural network was trained on paired ECG-echocardiograms (≤2 days apart) from patients ≤18 years of age without major congenital heart disease to detect human expert-classified greater than mild left ventricular (LV) dysfunction, hypertrophy, and dilation (individually and as a composite outcome). Model performance was evaluated on single ECG-echocardiogram pairs per patient at Boston Children's Hospital and externally at Mount Sinai Hospital using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). The training cohort comprised 92 377 ECG-echocardiogram pairs (46 261 patients; median age, 8.2 years). Test groups included internal testing (12 631 patients; median age, 8.8 years; 4.6% composite outcomes), emergency department (2830 patients; median age, 7.7 years; 10.0% composite outcomes), and external validation (5088 patients; median age, 4.3 years; 6.1% composite outcomes) cohorts. Model performance was similar on internal test and emergency department cohorts, with model predictions of LV hypertrophy outperforming the pediatric cardiologist expert benchmark. Adding age and sex to the model added no benefit to model performance. When using quantitative outcome cutoffs, model performance was similar between internal testing (composite outcome: AUROC, 0.88, AUPRC, 0.43; LV dysfunction: AUROC, 0.92, AUPRC, 0.23; LV hypertrophy: AUROC, 0.88, AUPRC, 0.28; LV dilation: AUROC, 0.91, AUPRC, 0.47) and external validation (composite outcome: AUROC, 0.86, AUPRC, 0.39; LV dysfunction: AUROC, 0.94, AUPRC, 0.32; LV hypertrophy: AUROC, 0.84, AUPRC, 0.25; LV dilation: AUROC, 0.87, AUPRC, 0.33), with composite outcome negative predictive values of 99.0% and 99.2%, respectively. Saliency mapping highlighted ECG components that influenced model predictions (precordial QRS complexes for all outcomes; T waves for LV dysfunction). High-risk ECG features include lateral T-wave inversion (LV dysfunction), deep S waves in V1 and V2 and tall R waves in V6 (LV hypertrophy), and tall R waves in V4 through V6 (LV dilation). This externally validated algorithm shows promise to inexpensively screen for LV dysfunction and remodeling in children, which may facilitate improved access to care by democratizing the expertise of pediatric cardiologists.

Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems– A Four-Arm Randomized Control Trial

To explore optimal feedback methodologies to enhance trainee skill acquisition in simulated surgical bimanual skills learning during brain tumor resections. (1) Providing feedback results in better learning outcomes in teaching surgical technical skill when compared to practice alone with no tailored performance feedback. (2) Providing more visual and visuospatial feedback results in better learning outcomes when compared to providing numerical feedback. A prospective 4-parallel-arm randomized controlled trial. Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Canada. Medical students (n = 120) from 4 Quebec medical schools. Participants completed a virtually simulated tumor resection task 5 times while receiving 1 of 4 feedback based on their group allocation: (1) practice-alone without feedback, (2) numerical feedback, (3) visual feedback, and (4) visuospatial feedback. Outcome measures were participants' scores on 14-performance metrics and the number of expert benchmarks achieved during each task. There were no significant differences in the first task which determined baseline performance. A statistically significant interaction between feedback allocation and task repetition was found on the number of benchmarks achieved, F (10.558, 408.257)=3.220, p < 0.001. Participants in all feedback groups significantly improved their performance compared to baseline. The visual feedback group achieved significantly higher number of benchmarks than the practice-alone group by the third repetition of the task, p = 0.005, 95%CI [0.42 3.25]. Visual feedback and visuospatial feedback improved performance significantly by the second repetition of the task, p = 0.016, 95%CI [0.19 2.71] and p = 0.003, 95%CI [0.4 2.57], respectively. Simulations with autonomous visual computer assistance may be effective pedagogical tools in teaching bimanual operative skills via visual and visuospatial feedback information delivery.

AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training

To better elucidate the role of artificial intelligence (AI) in surgical skills training requires investigations in the potential existence of a hidden curriculum. To assess the pedagogical value of AI-selected technical competencies and their extended effects in surgical simulation training. This cohort study was a follow-up of a randomized clinical trial conducted at the Neurosurgical Simulation and Artificial Intelligence Learning Centre at the Montreal Neurological Institute, McGill University, Montreal, Canada. Surgical performance metrics of medical students exposed to an AI-enhanced training curriculum were compared with a control group of participants who received no feedback and with expert benchmarks. Cross-sectional data were collected from January to April 2021 from medical students and from March 2015 to May 2016 from experts. This follow-up secondary analysis was conducted from June to September 2022. Participants included medical students (undergraduate year 0-2) in the intervention cohorts and neurosurgeons to establish expertise benchmarks. Performance assessment and personalized feedback by an intelligent tutor on 4 AI-selected learning objectives during simulation training. Outcomes of interest were unintended performance outcomes, measured by significant within-participant difference from baseline in 270 performance metrics in the intervention cohort that was not observed in the control cohort. A total of 46 medical students (median [range] age, 22 [18-27] years; 27 [59%] women) and 14 surgeons (median [range] age, 45 [35-59] years; 14 [100%] men) were included in this study, and no participant was lost to follow-up. Feedback on 4 AI-selected technical competencies was associated with additional performance change in 32 metrics over the entire procedure and 20 metrics during tumor removal that was not observed in the control group. Participants exposed to the AI-enhanced curriculum demonstrated significant improvement in safety metrics, such as reducing the rate of healthy tissue removal (mean difference, -7.05 × 10-5 [95% CI, -1.09 × 10-4 to -3.14 × 10-5] mm3 per 20 ms; P < .001) and maintaining a focused bimanual control of the operative field (mean difference in maximum instrument divergence, -4.99 [95% CI, -8.48 to -1.49] mm, P = .006) compared with the control group. However, negative unintended effects were also observed. These included a significantly lower velocity and acceleration in the dominant hand (velocity: mean difference, -0.13 [95% CI, -0.17 to -0.09] mm per 20 ms; P < .001; acceleration: mean difference, -2.25 × 10-2 [95% CI, -3.20 × 10-2 to -1.31 × 10-2] mm per 20 ms2; P < .001) and a significant reduction in the rate of tumor removal (mean difference, -4.85 × 10-5 [95% CI, -7.22 × 10-5 to -2.48 × 10-5] mm3 per 20 ms; P < .001) compared with control. These unintended outcomes diverged students' movement and efficiency performance metrics away from the expertise benchmarks. In this cohort study of medical students, an AI-enhanced curriculum for bimanual surgical skills resulted in unintended changes that improved performance in safety but negatively affected some efficiency metrics. Incorporating AI in course design requires ongoing assessment to maintain transparency and foster evidence-based learning objectives.

828 Real-Time Artificial Intelligence Instruction in Comparison to Human Expert Instruction in Surgical Technical Skills Teaching – a Randomized Controlled Trial

Abstract Aim Surgical simulators equipped with artificial intelligence (AI) systems provide risk-free training on realistically simulated complex patient cases, objective performance assessment, and tailored error avoidance feedback. This study compared the efficacy of real-time AI instruction with human instructor-mediated training in surgical technical skills teaching. Method Medical students (n = 98) were randomly allocated into three feedback groups. Everyone repeated a virtual brain tumour resection five times. The first repetition was done without receiving feedback to assess their baseline performance. Students in Group 1 did not receive real-time feedback, and they were shown visual feedback only after each procedure based on expert benchmarks. Students in Group 2 were instructed in real-time by the AI system. After each task, the students were shown their error-video clips generated by this system alongside the expert-level demonstrations relating to each error. Students in Group 3 were instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated correction techniques. All participant data was scored using a composite-score. Statistical analysis was conducted to compare learning from the first to the last task repetition. Results Students in Group 2 and Group 3 significantly improved in the performance score by the third and second repetition, respectively (p&amp;lt;0.01, p = 0.01) compared to their baseline performance. Students in Group 2 achieved significantly higher scores than those in Group 3 in the fifth repetition (p&amp;lt;0.01). Conclusions Artificial intelligence systems may provide competency-based curricula in surgical training with efficient surgical technical skills teaching methodologies.

SP11.14 Evaluating the Efficacy and Cognitive Load of a Real-Time Intelligent Coaching System versus Human Expert Instruction in Surgical Technical Skills Training: A Randomized Controlled Study

Abstract Aims An artificial intelligence-driven tutoring system provides risk-free training on realistic simulated procedures and objective assessment to improve surgical technical skills. This study compared its teaching efficacy to human instructor training. Methods Ninety-eight medical students completed 5 simulated brain tumor resections, scored by the intelligent system. Participants were randomly sorted into 3 feedback groups: (1) no real-time feedback, (2) real-time intelligent instruction, (3) real-time human instruction. Each participant completed a base-line repetition without feedback. Group 1 received visual feedback based on expert benchmarks following each repetition. Group 2 was tutored by the intelligent system in real-time. After each repetition, the system provided the student’s error clips and correlating expert-level demonstrations. Group 3 was tutored by human instructors in real-time. After each repetition, instructors provided critiques and demonstrated correction techniques. Following the final repetition, each participant completed a questionnaire of Leppink’s Cognitive Load Index on a 5-point Likert scale. Performance scores were compared from the first to the last repetition of the task within and between-groups to assess learning. Results Compared with baseline, groups 2 and 3 significantly improved in performance by the third and second repetition, respectively (p&amp;lt;0.01 and p=0.01). Between-groups comparison revealed significantly higher scores and extrinsic load for group 2 than group 3 in the fifth repetition, respectively (p&amp;lt;0.01 and p&amp;lt;0.01). No between-group differences existed for intrinsic and germane load. Conclusion The intelligent system provided more efficient learning than human instruction. Such systems may enhance trainees’ learning and facilitate the shift towards standardized, competency-based surgical training.

COMPARING THE EFFICIENCY OF A REAL-TIME ARTIFICIAL INTELLIGENCE INSTRUCTOR TO HUMAN EXPERT INSTRUCTORS IN SIMULATED SURGICAL TECHNICAL SKILLS TRAINING– A RANDOMIZED CONTROLLED TRIAL

Abstract BTFC travel award recipient Artificial intelligence systems provide risk-free training on realistically simulated patient cases and objective assessment of surgical technical skills. This randomized controlled study compared a real-time intelligent tutoring system in technical skills learning with human expert instructor-mediated training. METHODS: Ninety-eight medical students performed six simulated brain tumor resections. Participants were randomly allocated into (1)no-real-time feedback, (2)real-time intelligent instruction, and (3)in- person human instruction. All students performed the first repetition without receiving feedback (baseline). Group-1 received visual feedback only after each procedure based on expert benchmarks. Group-2 was instructed by the intelligent system in real-time. After each task, the students were shown their error-video clips generated by this system alongside expert-level demonstrations on how to improve. Group-3 was instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated correction techniques. Participant performance was scored by the intelligent system and also by blinded experts using OSATS scores. The performance score was compared within groups and between groups to compare learning. RESULTS: Compared to baseline performance, Group-2 and Group-3 significantly improved in the performance score by the third and second repetition, respectively (p&amp;lt;0.01, p=0.01). The between-groups comparison demonstrated that Group-2 scored significantly higher than Group-3 in the fifth repetition (p&amp;lt;0.01). Group-2 achieved significantly higher OSATS scores than Group-1 in the sixth task. CONCLUSIONS: Artificial intelligence may facilitate trainee learning by providing equally or more efficient learning when compared to human instruction. These systems may aid in developing competency-based standardized curricula in surgical training.

O022 Real-time artificial intelligence instructor vs expert instruction in teaching of expert level tumour resection skills – a randomized controlled trial

Abstract Introduction Competency-based approach in surgical training still lacks objective quantifiable methodologies to assess surgical technical skills and train residents, a limitation that can be addressed with the adaptation of artificial intelligence (AI). In this randomized controlled, the efficacy of learning by a real-time intelligent instruction system was compared to learning with in-person human instructor-mediated training. Methods The study was ethics approved. Ninety-eight medical students performed five virtually simulated brain tumour resections, randomly allocated into three feedback groups: (1) no-real-time feedback, (2) real-time intelligent instruction, and (3) in-person human instruction. The first task was considered as baseline performance, done with no feedback. Group-1 received expert benchmark feedback only after each procedure. Group-2 was instructed in real-time by the AI system. After each task, the students were shown their error-video clips generated by this system alongside the expert-level demonstrations relating to each error. Group-3 was instructed by human instructors during the tasks. After each task, instructors summarized the areas of improvement and demonstrated how to expertly perform the tumour resections. Participant data in all tasks were scored by the AI system to assess learning. Results Students in Group-2 and Group-3 significantly improved their performance score by the third and second task, respectively (p&amp;lt;0.01, p=0.01), compared to the baseline performance. Group-2 achieved significantly higher scores than Group-3 in the final/fifth task (p&amp;lt;0.01). Conclusion AI-powered systems may increase efficiency in learning by providing objective, and action-oriented real-time feedback. Such systems may aid the shift towards competency-based surgical curricula.

Benchmarking saliency methods for chest X-ray interpretation

Saliency methods, which produce heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. However, rigorous investigation of the accuracy and reliability of these strategies is necessary before they are integrated into the clinical setting. In this work, we quantitatively evaluate seven saliency methods, including Grad-CAM, across multiple neural network architectures using two evaluation metrics. We establish the first human benchmark for chest X-ray segmentation in a multilabel classification set-up, and examine under what clinical conditions saliency maps might be more prone to failure in localizing important pathologies compared with a human expert benchmark. We find that (1) while Grad-CAM generally localized pathologies better than the other evaluated saliency methods, all seven performed significantly worse compared with the human benchmark, (2) the gap in localization performance between Grad-CAM and the human benchmark was largest for pathologies that were smaller in size and had shapes that were more complex, and (3) model confidence was positively correlated with Grad-CAM localization performance. Our work demonstrates that several important limitations of saliency methods must be addressed before we can rely on them for deep learning explainability in medical imaging.

Metric-Based Simulation Training to Proficiency for Endovascular Thrombectomy in Ischemic Stroke.

Ischemic stroke is one of the leading causes of death and long-term disability in the West. Mechanical revascularization techniques are considered the standard of care for large vessel occlusive stroke. Traditional apprenticeship models involve doctors training their skills on patients. Simulation platforms have long been recognized as an alternative to this. There has however been very little robust assessment of the training outcomes achieved on some of these platforms. At best, these simulations increase understanding of the procedural process and may help improve some technical skills; at worst they may instill bad habits and poor technique. The prerequisite of any simulation process must be to teach what to do, with which devices, in the correct sequence as well as what not to do. It should provide valid metric-based feedback to the trainee that is objective, transparent, and fair for formative and summative performance feedback. It should deliver a training program that measures the performance progress of trainees against expert benchmarks—benchmarks that represent an evidence-based peer-reviewed standard. In this paper, we present a perspective for PBP training for thrombectomy based on our experience with the process of procedure characterization, metric validation, and early experience of using this approach for proficiency training. Patient outcomes are not only determined by optimal performance in the Angio Suite but also by an efficient patient procedure pathway. There will be value in utilizing the PBP training standard not only for the procedure itself but also for the constituent elements of the stroke pathway to further improve treatment outcomes for ischemic stroke patients.

MP19-10 DETERMINING THE EFFECTIVENESS OF A DELIBERATE PRACTICE TRAINING PLATFORM IN A VIRTUAL REALITY (VR) URETHROVESICAL ANASTOMOSIS CURRICULUM

You have accessJournal of UrologyCME1 May 2022MP19-10 DETERMINING THE EFFECTIVENESS OF A DELIBERATE PRACTICE TRAINING PLATFORM IN A VIRTUAL REALITY (VR) URETHROVESICAL ANASTOMOSIS CURRICULUM Kit Yuen, Alexis Steinmetz, Rachel Melnyk, Nathan Schuler, Galen Cheng, Dennis Phan, and Ahmed Ghazi Kit YuenKit Yuen More articles by this author , Alexis SteinmetzAlexis Steinmetz More articles by this author , Rachel MelnykRachel Melnyk More articles by this author , Nathan SchulerNathan Schuler More articles by this author , Galen ChengGalen Cheng More articles by this author , Dennis PhanDennis Phan More articles by this author , and Ahmed GhaziAhmed Ghazi More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000002552.10AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Effective and independent VR training is essential to optimize robotic training. Our study sought to evaluate a deliberate practice (DP) framework incorporated into a VR urethrovesical anastomosis (UVA) curriculum to improve skill transference. METHODS: Ten medical students with no robotic surgery experience were recruited for a 4-phase curriculum. Phase 1 Pre-Learning – 7 basic VR exercises on SimNow DaVinci Simulator, completed to competency standards (>90%). Phase 2 Pre-test – UVA on a validated realistic hydrogel model and a blinded reviewers graded RACE scores. Phase 3 Training – Trainees randomly allocated to the control group (n=4) performed repetitions of the VR procedural task “Guided UVA” and received a report that identified deficient areas after each VR attempt. The DP group (n=6) was additionally assigned specific basic modules targeting these deficiencies to be completed between UVA VR attempts (Figure 1). Both groups repeated the VR UVA to proficiency (2 SEM from expert mean). Phase 4 Post-test – A second hydrogel model was performed and scored. Performance differences between pre- and post-test established the effect of the VR training curriculum. RESULTS: DP group reached expert benchmarks in 7.5 UVA simulation attempts with 4.9 hours of total sim time while the control group averaged 8.75 attempts over 2.8 hours. The DP group progressed faster to proficiency than the control group in the subcategories of needle handling (3.5 vs 6.75 attempts) and complications (3.5 vs 6.25) but progressed similarly in economy benchmarks (7 vs 7.25). This translated into significant increases in pre-post hydrogel model performance in the DP group with RACE scores of 13.5 to 23 (p<0.001) and every subscore except tissue approximation. The control group did not experience a significant change in pre-post RACE scores (14.5 to 21; p=0.08) or any subscores. CONCLUSIONS: A focused approach addressing specific skill deficiencies using the DP framework resulted in significant improvement in hydrogel UVA performance. This suggests that surgical skills acquired during DP are not task specific and improve both overall skills acquisition and transfer to performance improvement on a realistic hydrogel model. Source of Funding: NA © 2022 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 207Issue Supplement 5May 2022Page: e310 Advertisement Copyright & Permissions© 2022 by American Urological Association Education and Research, Inc.MetricsAuthor Information Kit Yuen More articles by this author Alexis Steinmetz More articles by this author Rachel Melnyk More articles by this author Nathan Schuler More articles by this author Galen Cheng More articles by this author Dennis Phan More articles by this author Ahmed Ghazi More articles by this author Expand All Advertisement PDF DownloadLoading ...

EP.TU.657Scoping the role and content of a simulator-based endosim curriculum

Abstract Background Endoscopy training requires simultaneous acquisition of practical skill and knowledge. Virtual reality Simulators, such as EndoSim (SurgicalScience), offer the opportunity to deconstruct a skill into fundamental components (1), to allow repetitive practice, and achievement of task-specific objectives. Trainees with the least endoscopy experience benefit most from early simulated training (2,3), supporting the introduction of a simulation curriculum into SPRINT: Structured PRogramme for INduction and Training, an existing initiative to improve endoscopy training delivery in Wales. (4) The aim of this study was to design a pilot simulator curriculum for the EndoSim (Surgical Science, Gothemberg) simulator. Methods A focus group completed all EndoSim modules. Each EndoSim exercise was cross-examined against the relevant DOPS tool “Direct Observation of Procedural Skill” used by the Joint Advisory Group for Endoscopy Training and Certification. Exercises were chosen that represented each DOPS domain to teach basic skills in endoscopy scope handling. Results 12 exercises were chosen. These exercises addressed the insertion and withdrawal, and visualisation components of the JAG DOPS tool. Pre-procedural skills, management of findings, post-procedural skills and endoscopic non-technical Skills (ENTS) are beyond the scope of this simulator and require additional taught sessions to provide the context for current simulation training. Discussion This is the first step in developing and refining appropriate exercises to inform the proposed curriculum. The next step will be validating the chosen exercises against expert benchmark performance.

Self-assessment of Competence in Endoscopy: Challenges and Insights.

BackgroundEndoscopists use self-assessment to monitor the development and maintenance of their skills. The accuracy of these self-assessments, which reflects how closely one’s own rating corresponds to an external rating, is unclear.MethodsIn this narrative review, we critically examine the current literature on self-assessment in gastrointestinal endoscopy with the aim of informing training and practice and identifying opportunities to improve the methodological rigor of future studies.ResultsIn the seven included studies, the evidence regarding self-assessment accuracy was mixed. When stratified by experience level, however, novice endoscopists were least accurate in their self-assessments and tended to overestimate their performance. Studies examining the utility of video-based interventions using observation of expert benchmark performances show promise as a mechanism to improve self-assessment accuracy among novices.ConclusionsBased on the results of this review, we highlight problematic areas, identify opportunities to improve the methodological rigor of future studies on endoscopic self-assessment and outline potential avenues for further exploration.

Scoping Evidence Review on Training and Skills Assessment for Open Emergency Surgery

Scope evidence on technical performance metrics for open emergency surgery. Identify surgical performance metrics and procedures used in trauma training courses. Structured literature searches of electronic databases were conducted from January 2010 to December 2019 to identify systematic reviews of tools to measure surgical skills employed in vascular or trauma surgery evaluation and training. Faculty of Shock Trauma Anesthesiology Research Center, University of Maryland School of Medicine, Uniformed Services University of Health Sciences, Bethesda, Maryland and Implementation Science, King's College, London. The evidence from 21 systematic reviews including over 54,000 subjects enrolled into over 840 eligible studies, identified that the Objective Structured Assessment of Technical Skill was used for elective surgery not for emergency trauma and vascular control surgery procedures. The Individual Procedure Score (IPS), used to evaluate emergency trauma procedures performed before and after training, distinguished performance of residents from experts and practicing surgeons. IPS predicted surgeons who make critical errors and need remediation interventions. No metrics showed Kirkpatrick's Level 4 evidence of technical skills training benefit to emergency surgery outcomes. Expert benchmarks, errors, complication rates, task completion time, task-specific checklists, global rating scales, Objective Structured Assessment of Technical Skills, and IPS were found to identify surgeons, at all levels of seniority, who are in need of remediation of technical skills for open surgical hemorrhage control. Large-scale, multicenter studies are needed to evaluate any benefit of trauma technical skills training on patient outcomes.

Student Teachers’ Knowledge to Enable Problem-Solving for Sustainable Development

Education is a central strategy in terms of sustainable development (SD) and can contribute to solving global challenges like biodiversity loss and climate change. Content knowledge represents one base for teaching education for sustainable development (ESD). Therefore, identifying teaching and learning prerequisites regarding SD challenges in teacher education is crucial. The focus of the paper was to assess and learn more about student teachers’ procedural knowledge regarding issues of biodiversity and climate change, by using an expert benchmark. The aims of the study are to describe and identify (i) differences between students’ and experts’ effectiveness estimations, (ii) differences in bachelor and master students’ procedural knowledge, and (iii) differences between procedural knowledge of students studying different ESD-relevant subjects. Student teachers at eight German universities (n = 236) evaluated the effectiveness of solution strategies to SD challenges. The results showed high deviations in the effectiveness estimations of experts and students and, therefore, differing procedural knowledge. The lack of student teachers’ interdisciplinary knowledge to reduce biodiversity loss and climate change seemed to be largely independent of their study program and ESD-relevant subject. One reason for this may be the generally low number of ESD-relevant courses they attended. This study suggests further longitudinal research in order to make clear statements about changes in SD-related knowledge during teacher education.

The Effect of Expert Feedback on Antibiotic Prescribing in Pediatrics: Experimental Evidence.

Background. Inappropriate prescribing of antibiotics, which is common in pediatric care, is a key driver of antimicrobial resistance. To mitigate the development of resistance, antibiotic stewardship programs often suggest the inclusion of feedback targeted at individual providers. Empirically, however, it is not well understood how feedback affects individual physicians’ antibiotic prescribing decisions. Also, the question of how physicians’ characteristics, such as clinical experience, relate to antibiotic prescribing decisions and to responses to feedback is largely unexplored. Objective. To analyze the causal effect of descriptive expert feedback (and individual characteristics) on physicians’ antibiotic prescribing decisions in pediatrics. Design. We employed a randomized, controlled framed field experiment, in which German pediatricians (n=73) decided on the length of first-line antibiotic treatment for routine pediatric cases. In the intervention group (n=39), pediatricians received descriptive feedback in form of an expert benchmark, which allowed them to compare their own prescribing decisions with expert recommendations. The recommendations were elicited in a survey of pediatric department directors (n=20), who stated the length of antibiotic therapies they would choose for the routine cases. Pediatricians’ characteristics were elicited in a comprehensive questionnaire. Results. Providing pediatricians with expert feedback significantly reduced the length of antibiotic therapies by 10% on average. Also, the deviation of pediatricians’ decisions from experts’ recommendations significantly decreased. Antibiotic therapy decisions were significantly related to pediatricians’ clinical experience, risk attitudes, and personality traits. The effect of feedback was significantly associated with physicians’ experience. Conclusion. Our results indicate that descriptive expert feedback can be an effective means to guide pediatricians, especially those who are inexperienced, toward more appropriate antibiotic prescribing. Therefore, it seems to be suitable for inclusion in antibiotic stewardship programs.

Towards Expert-Based Speed–Precision Control in Early Simulator Training for Novice Surgeons

Simulator training for image-guided surgical interventions would benefit from intelligent systems that detect the evolution of task performance, and take control of individual speed–precision strategies by providing effective automatic performance feedback. At the earliest training stages, novices frequently focus on getting faster at the task. This may, as shown here, compromise the evolution of their precision scores, sometimes irreparably, if it is not controlled for as early as possible. Artificial intelligence could help make sure that a trainee reaches her/his optimal individual speed–accuracy trade-off by monitoring individual performance criteria, detecting critical trends at any given moment in time, and alerting the trainee as early as necessary when to slow down and focus on precision, or when to focus on getting faster. It is suggested that, for effective benchmarking, individual training statistics of novices are compared with the statistics of an expert surgeon. The speed–accuracy functions of novices trained in a large number of experimental sessions reveal differences in individual speed–precision strategies, and clarify why such strategies should be automatically detected and controlled for before further training on specific surgical task models, or clinical models, may be envisaged. How expert benchmark statistics may be exploited for automatic performance control is explained.

Validation of a virtual reality laparoscopic appendicectomy simulator: a novel process using cognitive task analysis.

Virtual reality (VR) simulation is a vital component of surgical training with demonstrated improvements in surgical quality and clinical outcome. To validate the LAP Mentor (Simbionix™) laparoscopic appendicectomy (LA) VR simulator with inclusion of a novel tool, Cognitive Task Analysis (CTA). Thirty-two novices and nine experienced surgeons performed two simulated LAs. An expert-consensus questionnaire guided face validity assessment. Content validity was assessed using CTA-derived questions encompassing eight operative steps and four decision points. Construct validity was evaluated using dexterity metrics, masked assessment of surgical quality using the OSATS global rating scale, and mental workload from two validated tools: the NASA-TLX and SMEQ. Ten novices performed eight further LAs for learning curve assessment. Face validity was demonstrated across all domains. Considering content validity, the essential technical and non-technical steps were evident. The experienced group performed the procedure quicker (median time 361 vs. 538s, P= 0.0039) with fewer total movements (426 vs. 641, P< 0.0001) and shorter idle time (131 vs. 199s, P= 0.0006). This correlated with higher OSATS scores (median 33.5 vs. 22.2, P< 0.0001) and lower mental demand (NASA-TLX: 9.0 vs. 13.75, P= 0.012; SMEQ: 60 vs. 80, P= 0.0025), indicating construct validity. Learning curve data showed statistically significant improvements after the 7th session for procedure time, total movements and idle time, which correlated with reduction in mental demand. The LAP Mentor demonstrates face, content and construct validity for LA; thus, it can be used as an effective tool in surgical training. Task repetition leads to achievement of expert benchmarks.

Hand motion analysis for assessment of nursing competence in ultrasound-guided peripheral intravenous catheter placement.

Objective measures such as hand motion analysis are needed to assess competency in technical skills, including ultrasound-guided procedures. Ultrasound-guided peripheral intravenous catheter placement has many potential benefits and is a viable skill for nurses to learn. The objective of this study was to demonstrate the feasibility and validity of hand motion analysis for assessment of nursing competence in ultrasound-guided peripheral intravenous placement. We conducted a prospective cohort study at a tertiary children's hospital. Participants included a convenience sample of nurses with no ultrasound-guided peripheral intravenous experience and experts in ultrasound-guided peripheral intravenous placement. Nurses completed hand motion analysis before and after participating in a simulation-based ultrasound-guided peripheral intravenous placement training program. Experts also completed hand motion analysis to provide benchmark measurements. After training, nurses performed ultrasound-guided peripheral intravenous placement in clinical practice and self-reported details of attempts. A total of 21 nurses and 6 experts participated. Prior to the hands-on training session, experts performed significantly better in all hand motion analysis metrics and procedure time. After completion of the hands-on training session, the nurses showed significant improvement in all hand motion analysis metrics and procedure time. Few nurses achieved hand motion analysis metrics within the expert benchmark after completing the hands-on training session with the exception of angiocatheter motion smoothness. In total, 12 nurses self-reported 38 ultrasound-guided peripheral intravenous placement attempts in clinical practice with a success rate of 60.5%. We demonstrated the feasibility and construct validity of hand motion analysis as an objective assessment of nurse competence in ultrasound-guided peripheral intravenous placement. Nurses demonstrated rapid skill acquisition but did not achieve expert-level proficiency.

Individuals’ Perception and Processing of Risk Information: Exploratory Evidence from Germany

This paper presents a first exploratory study to analyze private investors’ perceptions regarding risk reporting. With the help of think-aloud protocols, we gain insight into how and to what extent readers perceive risk information of a German savings bank, which specific information cues are relevant for informed judgment and decision-making, and whether these assessments are correct in relation to an expert benchmark. The results reveal that, during the reading process, the perceived information broadly coincides with the structure of the risk report considered. Only little information is indeed relevant. Information about the risk exposure is more relevant than information about the risk management system for the participants’ judgments and decisions, but each of them play an important role. On average, the participants’ assessments are actually in line with the expert benchmark and can therefore be considered correct–even the information cues mentioned as relevant are similar. These results provide a first indication of preferable approaches to risk reporting from the addressees’ point of view.