Preservation and Incorporation of Valuable Endoscopic Innovations (PIVI) on the use of endoscopy simulators for training and assessing skill

Abstract

The PIVI (Preservation and Incorporation of Valuable endoscopic Innovations) initiative is an American Society for Gastrointestinal Endoscopy (ASGE) program whose objectives are to identify important clinical questions related to endoscopy and to establish a priori diagnostic and/or therapeutic thresholds for endoscopic technologies designed to resolve these clinical questions. Additionally, PIVIs may also outline the data and/or the research study design required for proving that an established threshold is met. Once endoscopic technologies meet an established PIVI threshold, those technologies are appropriate to incorporate into clinical practice, presuming the appropriate training in that endoscopic technology has been achieved. The ASGE encourages and supports the appropriate use of technologies that meet its established PIVI thresholds. The PIVI initiative was developed primarily to direct endoscopic technology development toward resolving important clinical issues in endoscopy. The PIVI initiative is also designed to minimize the possibility that potentially valuable innovations are prematurely abandoned due to lack of use and to avoid widespread use of an endoscopic technology before clinical studies documenting their effectiveness have been performed. The following document, or PIVI, is one of a series of statements defining the diagnostic or therapeutic threshold that must be met for a technique or device to become considered appropriate for incorporation into clinical practice. It is also meant to serve as a guide for researchers or those seeking to develop technologies that are designed to improve digestive health outcomes. An ad hoc committee under the auspices of the existing ASGE Technology and Standards of Practice Committees Chairs develops PIVIs. An expert in the subject area chairs the PIVI, with additional committee members chosen for their individual expertise. In preparing this document, evidence-based methodology was used, with a MEDLINE and PubMed literature search to identify pertinent clinical studies on the topic. PIVIs are ultimately submitted to the ASGE Governing Board for approval, as is done for all Technology and Standards of Practice documents. This document is provided solely for educational and informational purposes and to support incorporating these endoscopic technologies into clinical practice. It should not be construed as establishing a legal standard of care. This PIVI reviews the current literature on simulator use in endoscopy and assesses what data are required to support a wider adoption of their use for endoscopy training and skills assessment. Specifically, the following two questions are considered: 1How much benefit must be demonstrated from the use of simulators to justify widespread adoption into standard endoscopy training?2How reliable do simulator-based assessments need to be as a predictor of patient-based skills to justify their use in credentialing and recredentialing for endoscopy? Since the early days of flexible endoscopy, educators have recognized the potential for simulators to enhance the training of students to gain proficiency. What began with crude static models to provide familiarity with basic dials and endoscope handling has evolved in the past 15 years into a wide array of ex vivo animal tissue and computer virtual-reality simulators. The development and capabilities have been well chronicled in the literature, as have many efforts to demonstrate their usefulness, particularly in the area of training.1Gerson L.B. Evidence-based assessment of endoscopic simulators for training.Gastrointest Endosc Clin N Am. 2006; 16 (vii-viii): 489-509Abstract Full Text Full Text PDF PubMed Scopus (31) Google Scholar, 2Nelson D.B. Bosco J.J. Curtis W.D. et al.ASGE technology evaluation report Endoscopy simulators. May 1999. American Society for Gastrointestinal Endoscopy.Gastrointest Endosc. 1999; 50: 935-937Abstract Full Text Full Text PDF PubMed Scopus (19) Google Scholar The theoretical benefits of simulator training are intuitive. They can provide a student with a relaxed opportunity for repetitive practice of skills including those that might not be encountered with sufficient frequency during the course of a standard training program. Improving basic skills before actual patient experience could result in reduced patient discomfort.3Sedlack R.E. Kolars J.C. Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study.Am J Gastroenterol. 2004; 99: 33-37Crossref PubMed Scopus (225) Google Scholar, 4Inaba K. Recinos G. Teixeira P.G. et al.Complications and death at the start of the new academic year: is there a July phenomenon?.J Trauma. 2011; 68: 19-22Crossref Scopus (75) Google Scholar For certain higher risk procedures such as ERCP, there is the potential for reducing risk to patients undergoing procedures in which novices are participating. Manpower limitations of available endoscopic educators or cost considerations of the increased time that trainers must spend away from their clinical duties would support the use of simulation tools that might either shorten the learning curve or allow students to do more of their instruction independently. Although the use of simulators has become much more widespread, particularly via the use of ex vivo–based hands-on training courses by the ASGE at its national training facility at the Interactive Training and Technology Center in Oak Brook, Illinois, and at many regional courses throughout the world, there is no consensus to date on just how much of a role they should play in standard training. The question of how good simulators need to be to warrant their use depends on many variables. It begins with a consideration of what are the unmet needs that simulator use might address and a thorough review of their current capabilities. Comparisons of the efficacy of simulator-based education with standard methods alone can only be made after learning curves are established for standard instruction, by using objective measures that encompass technical and cognitive skill components of a particular procedure. Ultimately, the decision about whether to incorporate these technologies into a training program must rely on data regarding the magnitude of training benefits, any cost savings resulting from accelerated learning, the initial and ongoing expenses associated with the simulator work, and the local needs of the institution. The endpoint of endoscopic training is the acquisition of competency to perform procedures independently. Professional societies charged with educating future endoscopists, and the public at large, have a vested interest in ensuring that the individuals credentialed to perform endoscopy are able to provide high-quality care. Key to this need for quality assurance is the impetus to move from subjective assessments of trainees' skill to more objective and validated means of doing so before they are credentialed. Much effort has been devoted, and much more is still required, to define which specific skills are required to become competent in each procedure, to determine minimal standards of proficiency, and to devise ways to objectively assess whether an individual has met that threshold. Controversy over what constitutes sufficient training for a particular procedure and how many procedures trainees require to perform with supervision can be resolved if there emerges the following two items: 1A consensus as to what minimum level of clinical performance constitutes competence to perform the procedure independently in the community. Presumably this would derive from benchmarking data about clinical performance of the particular procedure by practicing endoscopists.2An assessment tool that can measure a trainees' skill and reliably predict whether the individual is able to perform procedures at that minimal level of acceptable competency defined above. Recently, investigators validated such a tool for measuring trainee performance in colonoscopy on actual cases and, from this, defined minimal competency benchmarks.5Sedlack R.E. Training to competency in colonoscopy: assessing and defining competency standards.Gastrointest Endosc. 2011; 74: 355-366Abstract Full Text Full Text PDF PubMed Scopus (98) Google Scholar The development of a simulator-based assessment tool that could similarly reliably predict competent performance would be of enormous value. It would allow an unbiased and reproducible measure for credentialing purposes and ensure patients that the individuals performing their endoscopy, regardless of specialty, have been trained to sufficient standards of quality. For an endoscopy simulator to be integrated into the standard instruction for a procedure, it must demonstrate a 25% or greater reduction in the median number of clinical cases required for the trainees to achieve the minimal competence parameters for that procedure. The principal way in which simulators can have a meaningful impact on training would be for them to lead to a significant acceleration of the learning curve to the achievement of competence.6Wexner S.D. Litwin D. Cohen J. et al.Principles of privileging and credentialing for endoscopy and colonoscopy.Gastrointest Endosc. 2002; 55: 145-148Abstract Full Text Full Text PDF PubMed Google Scholar, 7Wexner S.D. Eisen G.M. Simmang C. Principles of privileging and credentialing for endoscopy and colonoscopy.Surg Endosc. 2002; 16: 367-369Crossref PubMed Scopus (24) Google Scholar, 8Cohen J. Greenwald D. Training in endoscopy: a historical perspective.in: Cohen J. Successful training in endoscopy. Wiley-Blackwell, Hoboken (NJ)2011: 3-15Crossref Scopus (8) Google Scholar For colonoscopy, current simulators have demonstrated a benefit in skill acquisition for the first 20 to 80 cases performed by novices but no reduction in the median number of cases required to achieve technical and cognitive competency.3Sedlack R.E. Kolars J.C. Computer simulator training enhances the competency of gastroenterology fellows at colonoscopy: results of a pilot study.Am J Gastroenterol. 2004; 99: 33-37Crossref PubMed Scopus (225) Google Scholar, 9Cohen J. Cohen S.A. Vora K.C. et al.Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy.Gastrointest Endosc. 2006; 64: 361-368Abstract Full Text Full Text PDF PubMed Scopus (166) Google Scholar With improved realism of models and perhaps more rigorous simulator experience, the consensus of the PIVI committee was that some modest impact on the learning curve could realistically be achievable. A threshold was chosen that was thought to be both theoretically attainable and also sufficiently high to justify the expense and effort involved in purchasing simulators and incorporating them into the training program. This panel opined that given the expense and effort involved, a reduction in training times or procedure numbers of at least 25% would be required. A more modest 10% benefit was felt to be insufficient to justify the investment in simulation devices by training programs and, based on the results from the existing literature, a 50% reduction in training times/number of cases was thought to be unattainable by any simulator in the near future. Although a reduction in the learning curve of more than 25% is desirable, given both the data on current models and the anticipated expense required to develop simulators that could produce a greater impact on the rate of skill acquisition, current expert consensus arrived at the threshold of 25%. Regardless of the external pressures to satisfy difficult-to-meet training needs at a particular institution, a measurable benefit from simulator training must be observed and the curriculum (simulator plus traditional proctored human cases) must result in the trainee successfully reaching the minimal level of skill required to perform the procedure independently. An important distinction is to be made between the incorporation of simulators into a training program for specific purposes and reliance on them to conduct proper training. The modest enhancement of training shown thus far in the literature for colonoscopy computer simulators underscores the point that the 25% threshold refers to the use of simulators only as a complementary tool to standard proctored endoscopy education by dedicated teachers. To date, there has been scant evidence that patients materially benefit when their procedure is performed by a trainee with previous simulator experience. However, future investigation that could demonstrate a meaningful benefit to patients from simulator-augmented curricula, in terms of decreased adverse events, improved satisfaction, or better compliance with screening, may prompt reevaluation of the threshold criteria for using simulators. Simulator-based assessment tools must be procedure specific and predictive of independently defined minimal competence parameters from real procedures with a κ value of at least 0.70 for high-stakes assessment. It is the consensus of this PIVI that strong predictive correlation between simulation performance with actual procedures is necessary for validation and acceptance of a simulator's use in high-stakes assessment. In education studies, a positive correlation coefficient between 2 different assessment methods of 0.65 is generally accepted to demonstrate a reasonably strong predictive value; ideally, a coefficient of 0.70 or higher is suggested for high-stakes assessment. Correlation coefficients of 0.85 or higher are rarely achieved in education research.10Fraenkel J. Wallen N. Correlation research.in: How to design and evaluate research in education. 5th ed. McGraw Hill, New York (NY)2003: 336-363Google Scholar As such, a correlation coefficient between simulation scoring of a skill and the scores of the same skill obtained from clinical assessment of patient-based endoscopy would be expected to be at a minimum of 0.65 to be considered useful for assessment of any type and 0.70 or higher for high-stakes assessment. These goals are obtainable. The surgical literature reports the achievement of similar predictive values in the examinations of proficiency in laparoscopy correlated with competent levels of performance of real procedures.11Beyer L. Troyer J.D. Mancini J. et al.Impact of laparoscopy simulator training on the technical skills of future surgeons in the operating room: a prospective study.Am J Surg. 2011; 202: 265-272Abstract Full Text Full Text PDF PubMed Scopus (69) Google Scholar, 12Santos B.F. Reif T.J. Soper N.J. et al.Effect of training and instrument type on performance in single-incision laparoscopy: results of a randomized comparison using a surgical simulator.Surg Endosc. 2011; 25: 3798-3804Crossref PubMed Scopus (36) Google Scholar In particular, the Fundamentals of Laparoscopic Surgery program has been extensively validated and is now a requirement by the American Board of Surgery. Part of the process involved the development of a validated measure of intraoperative skill (Global Operative Assessment of Laparoscopic Skills [GOALS]) to use as an outcome measure.13Vassiliou M.C. Kaneva P.A. Poulose B.K. et al.Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy.Surg Endosc. 2010; 24: 1834-1841Crossref PubMed Scopus (129) Google Scholar The correlation coefficient for GOALS scores during laparoscopic cholecystectomy and performance on the Fundamentals of Laparoscopic Surgery (FLS) program simulator was 0.81.14Fried G.M. Feldman L.S. Vassiliou M.C. et al.Proving the value of simulation in laparoscopic surgery.Ann Surg. 2004; 240 (discussion 525-8): 518-525Crossref PubMed Scopus (654) Google Scholar It is expected that performance assessments on simulators will eventually play a role in the high-stakes assessment of competency (ie, credentialing/certification) and, as such, must have high predictive value for quality performance of the procedure of interest. The demonstrated ability to simply distinguish novice from expert on the model is necessary but not sufficient to guarantee that the evaluation tool will be of clinical utility. The ideal tests will have high interobserver agreement and be geared to finely discriminate trainees at various stages of learning. A comprehensive review of published trials using ex vivo and computer simulators was performed and circulated to the members of the PIVI committee. Particular attention was given to the use of simulators for training in diagnostic colonoscopy and hemostasis of GI bleeding, two areas of high clinical interest for which simulator applications have been more extensively investigated. Study methodologies, size, and findings were considered for their relevance to the central questions and the level of evidence that they constituted. Given the limited existing literature pertaining to some of the key questions, this PIVI additionally relied on the expert opinion of its members. For that reason, the PIVI committee comprised well-recognized investigators and thought leaders in the field of simulators and endoscopy education and credentialing. Participants included both gastroenterologists and surgical endoscopists with significant experience in the field of simulators and education. The current literature summarizing the use of simulators for endoscopic training is delineated in the full Web version of the PIVI (available at http://www.asge.org/publications/publications.aspx?id=11958). Despite their varied capabilities and promising potential, use of the current endoscopic simulators appears to help primarily with early learning curves for endoscopic procedures. However, to date, simulator use has not yet led to an accelerated achievement of competency benchmarks or improved outcomes for patients. The highest levels of evidence demonstrate benefit in the early phase of colonoscopy training, without an ultimate shortening of the learning curve, as well as improvement in hemostasis skills after a series of intensive ex vivo hands-on workshops.9Cohen J. Cohen S.A. Vora K.C. et al.Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy.Gastrointest Endosc. 2006; 64: 361-368Abstract Full Text Full Text PDF PubMed Scopus (166) Google Scholar, 15Hochberger J. Matthes K. Maiss J. et al.Training with the compactEASIE biologic endoscopy simulator significantly improves hemostatic technical skill of gastroenterology fellows: a randomized controlled comparison with clinical endoscopy training alone.Gastrointest Endosc. 2005; 61: 204-215Abstract Full Text Full Text PDF PubMed Scopus (120) Google Scholar, 16Haycock A.V. Youd P. Bassett P. et al.Simulator training improves practical skills in therapeutic GI endoscopy: results from a randomized, blinded, controlled study.Gastrointest Endosc. 2009; 70: 835-845Abstract Full Text Full Text PDF PubMed Scopus (53) Google Scholar Individuals participating in 3 simulator hemostasis sessions performed better than those who did not attend these workshops, both on simulator skill assessment and on actual hemostasis rates on procedures performed during the 7-month study period.17Maiss J. Prat F. Wiesnet J. et al.The complementary Erlangen active simulator for interventional endoscopy training is superior to solely clinical education in endoscopic hemostasis–the French training project: a prospective trial.Eur J Gastroenterol Hepatol. 2006; 18: 1217-1225Crossref PubMed Scopus (41) Google Scholar Although corroborating studies will be important, the use of ex vivo models for hemostasis training appears to be the only area to date for which the literature currently supports the adoption of simulator work into the standard curriculum. However, no investigators have yet defined the standard learning curve for hemostasis skill of trainees or assessed the impact of ex vivo work on hastening competency in this area to confirm whether the benefit reaches the 25% threshold set forth in this PIVI. From the published experience to date, many other areas of simulator-based education appear to hold promise but will require more thorough investigation and validation to warrant calls for widespread adoption. There are only limited data examining the ability of endoscopy simulators to be used as tools to assess endoscopic skill. The limited literature available in this respect has shown these models to lack performance metrics of adequate sensitivity or reliability for use as meaningful assessment tools. At best, some simulators have been able to distinguish novice from experts, but generally they have failed to differentiate gradations of skill level. To date, a simulator-based assessment tool with predictive validity has yet to be developed for any endoscopic procedure. The available literature examining simulation devices (computer and animal tissue models) in endoscopy skills assessment for some of the more common endoscopic procedures is summarized in the full Web version of the PIVI (available at http://www.asge.org/publications/publications.aspx?id=11958). The goal of simulation training should be to decrease the number of clinical procedures needed to reach clinical proficiency, with the understanding that a certain amount of clinical training and experience will always be necessary for any procedure. The benefit of using simulators for training can only be validated in an adequately powered, controlled trial that demonstrates that the incorporation of simulators in a training program leads to acquisition of technical and cognitive competency with fewer clinical endoscopic procedures than required by traditional supervised endoscopic instruction without access to the simulator. Although shortening the learning curve for trainees to reach objective levels of proficiency will be the primary endpoint for future validation study, another potential endpoint that would promote adoption of simulators into standard curricula would be demonstration of significant cost and manpower reductions for instructors during the training process. Future studies will need to consider the transfer efficiency ratio to determine whether the modest benefits observed justify the particular time and costs involved. Future studies to validate a role for simulators in training will also need to address issues of generalizability. Clearly the potential benefits of simulator-based work may depend as much on the particular baseline experience level of the trainee as the attributes of the educational exercise itself. A simulator that is able to assess the actual clinical skills (technical and cognitive) of a clinician performing endoscopy may be a useful tool for credentialing (or recredentialing). A prerequisite for any validation of such a tool would be a previous consensus on how to measure competency for a particular procedure in real patients. Simulators designed for assessing clinical skills can only be valid if performance on the simulator is correlated with existing accepted clinical performance metrics on actual patients. For example, a study that evaluates the ability of a colonoscopy simulator to assess the clinical skills of an endoscopist should determine whether performance on the simulator is correlated with an endoscopist's cecal intubation rate or adenoma detection rate as endpoints in an adequately powered and controlled study. Any study that evaluates competency needs to include endoscopists at different levels of training and not just limit itself to studying endoscopists at the two ends of the competence spectrum. At a minimum, a simulator designed for assessing endoscopic skill must be able to distinguish endoscopists with beginning level (equivalent to a first-year trainee), intermediate level (average second-year trainee), fully trained (third-year trainee or recent graduate), and expert level (endoscopic instructor with at least 5 years training experience) endoscopic skills. Furthermore, the evaluation should be correlated with actual clinical outcomes to validate the performance of the simulator. In the opinion of this committee, it is not acceptable to use for high-stakes examination a simulator that has not been sufficiently validated specifically for such a purpose. For many existing procedures, it is clear that the ability to determine whether a simulator meets the threshold set forth in this PIVI for adoption as a standard training tool will require a more complete understanding of the standard learning curve. Once this is known, the 25% improvement may be reached for certain procedures as a result of some combination of more robust models and better use of existing models in the training program. The road to reaching the target for simulator-based skills assessments will require both improvement in the way in which we assess actual clinical procedure performance and advances in the simulators themselves. Specifically, this will require the following: 1Development of accurate, reliable, and validated clinical patient-based assessment tools for all endoscopic procedures to establish learning curves and benchmark clinical performance abilities.2Development of simulated cases of increased difficulty to allow discrimination of differing grades of ability (not just novice/expert).3Development of clinically relevant, accurate, and reliable simulator assessment metrics that effectively distinguish the different grades of ability and correlate with clinical performance benchmarks in prospective validation studies. As professional societies work to better define the constituents and benchmarks for competency in various procedures, investigators and providers of simulators will need to focus their efforts on addressing current unmet needs, determining which trainees get the most benefit from which tools, and reducing the cost of simulation to improve access and use. Dr Haycock received a grant from Olympus Keymed. Dr Cohen is a consultant for Olympus USA.