DISCUSSION IMPROVES INTERRATER RELIABILITY IN STRUCTURED PEER REVIEW

Abstract

S172 Introduction: Peer review of adverse outcomes has been proposed as a means of evaluating clinical competency and recommended for the purpose of physician relicensure. The expert witness in medical malpractice represents a unique form of peer review which utilizes individual opinion. Review of the medical peer review literature suggests that agreement among physicians performing individual reviews is only slightly better than that expected by chance [1]. Methods: Five board-certified anesthesiologists were trained in the use of two structured peer review (SPR) models [2,3]. The level of agreement with respect to identification of the adverse outcome (indicator), the severity of the injury (outcome score), and the cause of the adverse event (error classification) for 50 case abstracts before and after discussion was determined. Evaluations were confidential and no attempt was made during the discussion to achieve consensus. Interrater reliability was measured using the kappa statistic [4,5] for a) each unique indicator chosen by one or more reviewers, b) he associated outcome score and c) error classification for each SPR model. Results: Kappa values with the two SPR models were statistically indistinguishable throughout the study. For both, interrater reliability for indicators, outcome scores, and error classification was in the "poor" to "fair-to-good" range (although p<0.001 compared to random evaluations.) A dramatic increase in interrater reliability followed group discussion for all variables (Table 1) regardless of the SPR system.Table 1Discussion: Even highly structured SPR systems do not achieve agreement among reviewers acting independently. Any system of peer review, be it for recredentialing, relicensure, or medical malpractice, should not utilize the implicit opinions of individual reviewers. Rather, a SPR system should be utilized which includes discussion of the case among a group of peers.