Predicting Resident Competence for Otolaryngology Key Indicator Procedures.

Abstract

Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs. From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity. A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation. Predictive modeling can inform assessment benchmarks for competency-based surgical education. NA Laryngoscope, 133:3341-3345, 2023.