Development and Pilot Testing of an Assessment Tool for Performance of Invasive Mediastinal Staging

Abstract

To develop and evaluate a surgical trainee competency assessment instrument for invasive mediastinal staging, including cervical mediastinoscopy and endobronchial ultrasound (EBUS), a comprehensive instrument was developed, the Thoracic Competency Assessment Tool-Invasive Staging (TCAT-IS), using expert review and simulated and clinical pilot-testing. Validity and reliability evidence were collected, and item analysis was performed. Initially, a 27-item instrument was developed, which underwent expert review with members of the Canadian Association of Thoracic Surgeons (n= 86) in 2014 to 2015 (response rate, 57%). TCAT-IS was refined to 29 items in 4 competency areas: preoperative, general operative, mediastinoscopy, and EBUS. Further refinements were made based on simulated use. The final version was then used to assess competency of 5 thoracic trainees performing invasive mediastinal staging in live patients. Participants were assessed during 20 mediastinoscopy and 8 EBUS procedures, with 47 total assessments completed. Reliability (Cronbach's alpha= 0.94), interrater reliability (κ= 0.80), and correlation with an established global competency scale (κ= 0.75) were high. The most difficult items were "set up and adjust EBUS equipment" and "identify vascular anatomy (EBUS)." Feedback questionnaires from trainees (response rate, 80%) and surgeons (response rate, 100%) were consistently positive regarding user friendliness, utility as an assessment tool, and educational benefit. Participants believed the tool "facilitated communicating feedback to the trainee with specific areas to work on." TCAT-IS is an effective tool for assessing competence in invasive staging and may enhance instruction. This initial test establishes early validity and reliability evidence, supporting the use of TCAT-IS in providing structured, specific, formative assessments of competency.