European Society for Vascular Surgery (ESVS) Certification of Theoretical and Practical Competences in Basic Vascular Ultrasound: Validity Investigation of the Assessment Tools

Abstract

The aim of this study was to gather validity evidence for the Assessment of basic Vascular Ultrasound Expertise (AVAUSE) tool, and to establish a pass/fail score for each component, to support decisions for certification. A cross sectional validation study performed during the European Society for Vascular Surgery's annual meeting. Validity evidence was sought for the theoretical test and two practical tests based on Messick's framework. The participants were vascular surgeons, vascular surgical trainees, sonographers, and nurses with varying experience levels. Five vascular ultrasound experts developed the theoretical and two practical test components of the AVAUSE tool for each test component. Two stations were set up for carotid examinations and two for superficial venous incompetence (SVI) examinations. Eight raters were assigned in pairs to each station. Three methods were used to set pass/fail scores: contrasting groups' method; rater consensus; and extended Angoff. Nineteen participants were enrolled. Acceptable internal consistency reliability (Cronbach's alpha) for the AVAUSE theoretical (0.93), carotid (0.84), and SVI (0.65) practical test were shown. In the carotid examination, inter-rater reliability (IRR) for the two rater pairs was good: 0.68 and 0.78, respectively. The carotid scores correlated significantly with years of experience (Pearson's r=0.56, p=.013) but not with number of examinations in the last five years. For SVI, IRR was excellent at 0.81 and 0.87. SVI performance scores did not correlate with years of experience and number of examinations. The pass/fail score set by the contrasting groups' method was 29 points out of 50. The rater set pass/fail scores were 3.0 points for both carotid and SVI examinations and were used to determine successful participants. Ten of 19 participants passed the tests and were certified. Validity evidence was sought and established for the AVAUSE comprehensive tool, including pass/fail standards. AVAUSE can be used to assess competences in basic vascular ultrasound, allowing operators to progress towards independent practice.