Development and Interrater Agreement of a Novel Classification System Combining Medical and Surgical Adverse Event Reporting

Abstract

Categorization systems for adverse events are not standardized across care settings and specialties and do not always include near miss events (events where there was potential for patient harm, but where no actual harm occurred), making it difficult to effectively assess patient safety for quality improvement. To develop and assess interrater agreement on a classification system for adverse events reporting that incorporates events in both inpatient and outpatient settings across medical and surgical subspecialties including near miss events. A cross-sectional study in a tertiary care center including 174 patient cases occurring from 2018 to 2020 was carried out. Data were abstracted from a Department of Otorhinolaryngology-Head and Neck Surgery Quality Assurance database. The cases were comprised of near miss and adverse events occurring in adult and pediatric patients in inpatient, outpatient, and emergency department settings. The ratings took place in March and April of 2022. Four raters (2 attending physicians and 2 senior resident physicians) were recruited to classify these cases according to 3 classification systems: the National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP), Clavien-Dindo, and our novel Quality Improvement Classification System (QICS). The primary outcome was overall interrater agreements using Fleiss κ. Across all 4 raters grading 174 cases, the NCC-MERP, Clavien-Dindo, and QICS received a κ score. Fair-to-moderate interrater reliability was observed between the resident and attending physician groups across the 3 classification systems: NCC-MERP (κ = 0.33; 95% CI, 0.30-0.35), Clavien-Dindo (κ = 0.47; 95% CI, 0.43-0.50), and QICS (κ = 0.42; 95% CI, 0.39-0.44). Strong interrater concordance was observed for complications across all scenarios. This cross-sectional study found that the new QICS classification scheme was applicable to wide-ranging clinical scenarios with a focus on patient-centered outcomes including near miss events. In addition, QICS allowed for the comparison of patient outcome data in a multitude of settings.