Setting Mastery Standards In Hepato-Pancreato-Biliary Surgical Ultrasound

Abstract

Presenter: Ellen Hagopian MD | Hackensack Meridian School of Medicine at Seton Hall University Background: A standardized curriculum in Hepato-Pancreato-Biliary (HPB) ultrasound (US), grounded in a Mastery skills framework, was developed in 2014 for HPB fellows in Americas Hepato-Pancreato-Biliary Association-Fellowship Council accredited programs. We previously developed two assessment tools for laparoscopic (LAPUS) and open US (IOUS) in HPB Surgery, each of which contains 14 items (skills) and a global performance item, using the O-Score entrustment scale, in which 1=“I had to do it”, 2= “I had to talk them through”, 3= “I had to prompt them from time to time”; 4=”I needed to be in the room just in case” and 5=“I did not need to be there.” A formative skills practicum utilizing these tools is offered annually to HPB fellows approaching graduation to provide individual performance feedback. The aim of this work was to establish Mastery standards for our two assessments, and to apply those standards to a sample of graduating fellows, providing consequential validity evidence within Messick’s framework. Methods: Standard setting: Faculty with expertise in HPB Surgery and HPB US from the membership of Americas Hepato-Pancreato-Biliary Association participated in standard setting exercises. Faculty were asked to rate each item and global US performance for both IOUS and LAPUS: What is the probability that a graduating HPB fellow, who is well-prepared for independent practice, is able to perform this skill? Please rate as a percentage. Item-level Mastery Angoff cut scores were determined as the mean rating across all judges. The overall Mastery Angoff cut score was determined as the mean cut score across all items. Mastery Angoff cut scores were converted to the 5-point entrustment scale (i.e., cut score*5). ellow performance: At the 2019 annual US skills practicum, fellow performances were evaluated using the assessment tools and the Mastery Angoff standards. This study met criteria for exemption by the University of Illinois-Chicago Institutional Review Board (2019-0265). Results: A total of 15 faculty were invited to participate, 12 of whom returned the questionnaire. To account for outliers, the highest and lowest faculty ratings were removed, thus standards were based on 10 faculty experts. For IOUS, the overall mean cut score across all items was 91.48% (SD 0.04) and for LAPUS, the overall mean cut score across all items was 89.96% (SD 0.04). The mean global IOUS performance cut score was 96.00% (SD 0.07), whereas the mean global LAPUS performance cut score was 87.80% (SD 0.14). (See Table) A total of 13 of 18 fellows who participated in the 2019 US skills practicum agreed to have their de-identified data evaluated. Overall, fellows did not meet standards in 11/14 IOUS items and in 13/14 LAPUS items. The mean global performance entrustment score was 3.62 (SD 0.65, range 2-4) in IOUS and 3.62 (SD 0.77, range 2-5) in LAPUS, which did not meet the entrustment standards of 4.80 and 4.39, respectively. Conclusion: AHPBA-FC HPB fellows prior to graduation are not meeting Mastery Standards for HPB US performance determined by a panel of expert faculty. A change in educational policy or curriculum may be needed to improve fellow performance.