Ensuring Competency in Simulated Open Aortic Aneurysm Repair Using a Newly Developed Procedure-specific Assessment Tool- A Reliable Tool with Credible Pass-fail Standard for Assessment During Training, Testing and Certification

Abstract

Introduction: There is an increasing need to establish structured simulation-based training programs in open repair of infrarenal aortic aneurysms (AAA) to address the current challenges such as work hour restrictions and decreasing frequency of open surgical repairs with the widespread shift to endovascular treatment (1,2). Simulation-based training programs for open AAA repair require a dedicated assessment tool with a credible pass-fail score to ensure minimum competences before apprenticeship on patients. Previous studies have used existing surgical assessment tools (3,4), however an up-to-date, and procedure-specific tool covering all steps of open AAA repair has yet to be developed and validated using a contemporary framework. The aims of the study were to develop a procedure-specific assessment tool for simulated open AAA repair, gather validity evidence for the tool, and establish a pass/fail standard for assessment. Methods: Validity evidence was explored using the contemporary framework by Messick (5). Three vascular surgeons with expertise in open AAA repair, and a professor in medical education with expertise in assessment and validation, developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) to ensure content evidence. Vascular surgical trainees with varying experience were invited to perform an open AAA repair on an abdominal model for open aortic surgery with pulsatile flow. All participants received e-learning materials prior to the operations. To ensure the quality and integrity of data, all procedures were performed in a standardized simulation-based environment with the presence of a supervisor who provided guidance when needed. The procedures were video recorded, anonymised, and scored independently by three expert vascular surgeons in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass-fail standard to determine consequences of testing. Results: A total of 16 novices and nine experienced open vascular surgeons participated in the study. The OPERATE consists of twelve items including four global scores (Figure 1). A high internal consistency for all items (Cronbach's alpha .92) and a high inter-rater reliability (Cronbach's alpha .94) were achieved. The novices performed significantly inferior from the experienced group with mean OPERATE scores of 15.1 and 25.6 (p=0.01), respectively (Figure 2). The pass/fail standard based on the Angoff method was 27.7. One novice passed the test while six experienced failed to achieve the pass-fail score. Conclusion: Strong validity evidence with a credible pass-fail standard was established for the newly developed procedure-specific assessment tool OPERATE, demonstrating its reliable use for structured formative assessment during training and summative assessment during testing or certification. The OPERATE has high discriminatory abilities and a high pass-fail standard emphasising the need for novice vascular surgeons to train in a simulation-based environment to achieve the minimum competences and skillsets before apprenticeship training on patients. The low scores in the experienced group underlines that familiarisation with the simulation equipment must be ensured before performance is assessed.Figure 2Box and whiskers plot showing the scores of the two groups.View Large Image Figure ViewerDownload Hi-res image Download (PPT) Disclosure: Nothing to disclose