Board 358 - Research AbstractVirtual-Reality Simulator Training is More Efficient than Apprenticeship Training in the First Part of the Learning Curve

Abstract

Introduction/Background Endosonography has proven superior to traditional Methods for obtaining tissue samples from the mediastinum and has now become the method of choice for staging lung cancer.1-3 However, the technique is highly operator dependent and focus on training and certification is essential before Results from expert centers can be generalized. Virtual-reality (VR) simulation are superior to no training but studies should instead compare VR training to traditional training.4 Outcomes should be measured on real patients (transfer studies) using assessment tools with established validity evidence from similar settings. Several procedures must be assessed by multiple raters in a blinded fashion to ensure reliability and reduce the risk of bias.5 Credible standards are necessary for summative assessment. The aims of this study were to determine whether simulator training could replace practicing on patients. Secondly, to explore a reliable assessment method and set a credible pass/fail standard. Methods A twelve item EndoBronchial UltraSound Assessment Tool (EBUSAT) was created by experts in endosonography and assessment. Each item was scored from 0 to 4 points giving a maximum score of 48 points. An international volunteer sample of 16 respiratory physicians without EBUS experience and a convenience sample of two EBUS experts were included in the study. The respiratory physicians attended an eight hour theoretical course and participated in two EBUS procedures on patients. Then they were randomized to a standardized, individual four hour training program on a virtual-reality EBUS simulator, administered by the same operator in both countries vs. a half-day, standard, hands-on training on patients supervised by one of two international EBUS experts. Each physician performed three consecutive EBUS-TBNA procedures. The endoscopic and the ultrasound videos were recorded and assessed blindly and independently by three EBUS experts using the EBUSAT. Ten consecutive procedures by EBUS experts were assessed simultaneously to gather validity evidence. Procedures performed by simulator-trained novices, apprenticeship-trained novices and endosonography experts respectively were compared using independent samples t-tests. Reliability was explored using Generalizability Theory (a two-facet design with participants crossed with procedures crossed with raters). The contrasting group’s method was used to set a credible pass/fail-standard (secondary objective). Results The simulator group performed significantly better than the standard training group; 24.2 points vs. 20.1 points, p=0.006. The EBUSAT scores were very reliable; generalizability coefficient=0.85. An acceptable Generalizability Coefficient for high-stakes assessment of 0.8 could be achieved by using three raters, assessing two procedures and two raters assessing three procedures, or one rater assessing six procedures. The experts scored significantly higher than the novices indicating that the EBUSAT possess construct validity; 35.3 points, p<0.001. A credible pass/fail standard could be established = 28.2 points. 4% of the procedures performed by the standard training group passed vs. 17% in the simulation group. Only one of the procedures performed by the experts failed to meet standards (90% passed). Conclusion EBUS training on a virtual-reality simulator is more efficient than standard apprenticeship training in the initial part of the learning curve but subsequent supervised training on patients are necessary to ensure basic competency. The EBUSAT can be used to assess performances in an unbiased, reliable and valid way. A credible standard set by an accepted method make it possible to perform high-stakes certification using the EBUSAT provided that the number of raters and procedures assessed ensures sufficient reliability. Focus on patient safety and competency based education makes virtual-reality training and assessment of competence very relevant issues in the implementation of endosonography.