Development and Pilot Testing of an Assessment Tool for Performance of Anatomic Lung Resection

Abstract

To meet the need for competency assessment in thoracic surgery education, we developed and tested an instrument to assess trainees' ability to perform anatomic lung resection for cancer. The Thoracic Competency Assessment Tool-Anatomic Resection for Lung Cancer (TCAT-ARC) was developed through a multistep process involving logical analysis, expert review, and simulation-based and clinical pilot testing. Validity evidence was gathered during a 6-month clinical study of trainees performing anatomic lung resections and assessments of practicing surgeons. Feedback was gathered via post-encounter questionnaires. A 35-item instrument was developed and was tested in the clinical validation study. Seven trainees in 4 North American institutions participated and completed 64 anatomic lung resections. Reliability was high (α= 0.93). Interobserver reliability (k= 0.73) and correlation with an existing global competency scale (k= 0.68) were moderately high. Item analysis revealed the most difficult and discriminatory items, which matched well with a conceptual understanding of lung resection. Both trainees and assessors viewed the instrument as highly educationally effective and user-friendly. Practicing surgeons outperformed trainees. The TCAT-ARC demonstrated early evidence of validity and reliability in assessing performance of anatomic lung resection. The instrument may be most useful early in training and as a means for providing fine-grained formative feedback about which steps have been mastered and which still require improvement. The TCAT-ARC may be used in training programs to aid in the development of trainees' competency and as a part of an aggregate assessment of trainees' overall mastery of the procedure and readiness for independent practice.