55: COMPETENCY ASSESSMENT TOOL FOR MINIMALLY INVASIVE ESOPHAGECTOMY: DEVELOPMENT AND PRELIMINARY CONSTRUCT VALIDITY AND RELIABILITY RESULTS

Abstract

Abstract Background and aim Performance of the surgical team is essential for patient outcome. Minimally invasive esophagectomy (MIE) is a complex and technically demanding procedure and implementation is associated with a substantial learning curve. A validated, procedure specific and scientifically robust competency assessment tool (CAT) could provide structured feedback and can be a tool to improve patient outcome. The aim is to develop a MIE-CAT and assess construct validity, intra-observer reliability and inter-observer reliability. Methods Based on defined key steps from a Delphi study and refinement with a core group of MIE experts to establish content validity and obtain user agreement, the MIE-CAT was developed. Eighteen independent and blinded international MIE experts (> 120 MIEs performed) followed an online workshop to train in using the MIE-CAT and assessed Ivor Lewis MIE videos that were performed from 2010–2020 by four MIE surgeons from the Esophageal Center East Netherlands. To analyze construct validity, 32 videos were divided into four groups (novice, intermediate, advanced and expert) based on the surgical team’s experience. For intra-rater and inter-rater reliability, the same video was scored twice by the same and different experts. Results The MIE-CAT consists of eight procedural phases, starting with the mobilization of the greater curvature and ending with creation of the intrathoracic anastomosis. Each phase contains four domains (exposure, execution, adverse events and end-product quality) and are scored with a 1–4 Likert scale. A total of 72 video assessments were used for analysis. Substantial performance variation between the MIE-CAT scores was seen (median 98 [IQR, 84–113.4]; range = 47–126). Median MIE-CAT scores were 79.5 [IQR 74–96.3], 91 [87.5–107], 101 [84.8–108] and 101 [95.5–108.3] for respectively the novice, intermediate, advanced and expert group (Figure 1). Significant differences between the average MIE-CAT scores of the experience groups were found with a Kruskal Wallis (P = 0.002). Good intra-rater reliability was observed (ICC, 0.71, 95% CI [0.35, 0.89) and moderate inter-rater reliability was observed (ICC, 0.54, 95% CI [0.10, 0.87). Conclusion A specific competency assessment tool to score the MIE performance, the MIE-CAT, was developed. It has the potential to be used for surgical training, safe implementation, performance improvement and research purposes.