Validation of a clinical practice ability instrument for surgical training

Abstract

Objective To develop and validate an instrument to measure the progression of clinical practice ability among surgical trainees. Study Design Prospective validation study. Setting Three academic training institutions: Massachusetts Eye and Ear Infirmary, Brigham and Women's Hospital, Harvard School of Public Health. Subjects and Methods A clinical practice instrument (CPI) was created and prospectively validated in otolaryngology trainees at three institutional sites. The instrument is completed during an oral examination of a trainee who is presented with a patient scenario; the scenario yields one clinical diagnosis, one therapeutic intervention, and one interventional complication. On the basis of the trainee's responses, 21 item scores are calculated focusing on key areas within the diagnostic and management paradigm. Internal consistency, reproducibility, inter-rater reliability, discriminant and criterion validity, and responsiveness to change over time were evaluated. Results There were 87 administrations of the instrument over a period of 22 months. The subjects (n = 32) spanned the range of training from medical student to clinical fellow. There was strong internal consistency within each domain (Cronbach's alpha 0.85-0.94, one-sided 95% confidence interval > 0.81). Inter-rater reliability was high (kappa 0.66, absolute agreement 72%). Reproducibility of item scores was excellent (intraclass correlation coefficient [ICC] final summary score 0.78, global item score 0.83). Discriminant validity ( P < 0.0001), criterion validity ( P = 0.03), and responsiveness to change ( P = 0.0055) were also established. Conclusion The CPI serves as a reliable, reproducible, and valid means to measure the progression of clinical practice ability in trainees as they transition from “novice” to “attending equivalent” surgeons.