Assessing Competence in Endovascular Aortic Repair

Abstract

Introduction - Endovascular aortic repair (EVAR) is an expert task. A mastery learning approach is warranted. To date no validated assessment tools of competence in EVAR exist. The primary aims were to develop evaluation documents for learning progress and skill acquisition for A) stent graft sizing and selection1, and B) operator’s procedural competence in EVAR2. Methods - A) The test for EVAR stent graft planning was developed based on a literature review. An international cross sectional study examined evidence of validity for the assessment. Consultants (N=22) from radiology and vascular surgery with varying experience in EVAR planning (novices, intermediates, and experts) were presented with computed tomography angiography from three patients. Test scores were based on z-scores using the anatomical measurements and graft choices of the experts as reference. B) A Delphi panel of international EVAR experts was established (N=32, vascular surgery (N=21) and radiology (N=11)). Iterative Delphi rounds were executed. The first round was based on triangulation of a literature review, stent graft instructions for use, and interviews with experts. The panellists rated the importance of competence assessment items on a five point Likert scale. Consensus was defined as 80% of the panel rating an item four or five in the primary round and 90% in subsequent. Consensus on the final assessment tool was defined as Cronbach’s alpha >0.8 after a minimum of three rounds. Results - A) The stent planning test demonstrated reliable results (ICC=0.83, p < 0.001) and high internal consistency (Cronbach’s α= 0.91, p < 0.001). Overall, experts performed significantly better than novices and intermediates (p < 0.002 and p < 0.005, respectively). The anatomical sub-test discriminated significantly between experts and novices (Mann Whitney U, p = 0.002), experts and intermediates (p = 0.010), and novices and intermediates (p = 0.036). In the stent selection sub-test experts performed significantly better than both novices and intermediates (p = 0.002 and p = 0.007), but no difference was found between novices and intermediates (p = 1). B) Delphi study: Three rounds of surveys were conducted with a completion rate of 100% in the first two rounds and 91% in round three. The expert panel reached consensus on seven pivotal assessment items, Cronbach’s alpha=0.82. The resulting rating scale, EVARATE, covers key elements in EVAR competency. Each assessment item has well defined grades with explicit anchors from unacceptable to superior performance on a five point Likert scale. Conclusion - This study presented a standardised and objective assessment tool of competence in EVAR supported by strong validity evidence. A novel seven-item rating scale for assessment of operator’s competence was developed, based on international expert consensus. These tests of EVAR competence hold promise for teaching novel operators at entry to practise.