The Ultrasound Competency Assessment Tool (UCAT): Development and Evaluation of a Novel Competency-based Assessment Tool for Point-of-care Ultrasound.

Abstract

Point-of-care ultrasound (POCUS) has become an integral diagnostic and interventional tool. Barriers to POCUS training persist, and it continues to remain heterogeneous across training programs. Structured POCUS assessment tools exist, but remain limited in their feasibility, acceptability, reliability, and validity; none of these tools are entrustment-based. The objective of this study was to derive a simple, entrustment-based POCUS competency assessment tool and pilot it in an assessment setting. This study was composed of two phases. First, a three-step modified Delphi design surveyed 60 members of the Canadian Association of Emergency Physicians Emergency Ultrasound Committee (EUC) to derive the anchors for the tool. Subsequently, the derived ultrasound competency assessment tool (UCAT) was used to assess trainee (N=37) performance on a simulated FAST examination. The intraclass correlation (ICC) for inter-rater reliability and Cronbach's alpha for internal consistency were calculated. A statistical analysis was performed to compare the UCAT to other competency surrogates. The three-round Delphi had 22, 26, and 26 responses from the EUC members. Consensus was reached, and anchors for the domains of preparation, image acquisition, image optimization, and clinical integration achieved approval rates between 92 and 96%. The UCAT pilot revealed excellent inter-rater reliability (with ICC values of 0.69-0.89; p<0.01) and high internal consistency (α= 0.91). While UCAT scores were not impacted by level of training, they were significantly impacted by the number of previous POCUS studies completed. We developed and successfully piloted the UCAT, an entrustment-based bedside POCUS competency assessment tool suitable for rapid deployment. The findings from this study indicate early validity evidence for the use of the UCAT as an assessment of trainee POCUS competence on FAST. The UCAT should be trialed in different populations performing several POCUS study types.