Reliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scores.

Abstract

Evaluation of faculty anesthesiologists' clinical supervision is psychometrically reliable. Supervision scores often are used for ongoing professional practice and teaching evaluations. We evaluated whether anesthesiologists' clinical supervision rank could be determined reliably using 6- vs. 12-month data collection intervals and, for each, determined specificity (quartiles vs. halves). To serve as a comparator/control group, we analyzed anesthesiologists' evaluations of the work habits of nurse anesthetists. Residents evaluated 73 anesthesiologists and anesthesiologists evaluated 72 nurse anesthetists with whom they worked in operating rooms for ≥60 min, daily. Scores were adjusted for rater leniency. The reliability of ranking in halves and quartiles was determined and comparisons made. Clinical supervision and work habit scores, respectively. Using 1 year of data, 5% of anesthesiologists had scores that were reliably in the lowest quartile and 21% had scores that were reliably in the lower half. No (0%) anesthesiologists were reliably in the highest quartile and 19% were reliably in the upper half. Corresponding percentages for nurse anesthetists' work habits scores were 3%, 10%, 0%, and 1%. The primary factor limiting the reliability of ranking in quartiles was the number of operating room days worked. Ranks based on raw (unadjusted) scores falsely misclassified 60% of anesthesiologists and 88% of nurse anesthetists. The percentages of anesthesiologists who could be ranked reliably was less when using a shorter evaluation period (6 months). Using mixed effects regression to control for rater leniency, anesthesiologists' ranks based on supervision scores can be determined reliably for halves and quartiles, but fewer than half the anesthesiologists will be reliably ranked. The same ranking principles can be applied for evaluation of nurse anesthetists' work habits. Even when very high G-scores of 0.90 are obtained, ranking individuals into smaller groups (e.g., deciles) or using raw (observed) ranks is unreliable.