Interrater reliability of a near-miss risk index for incident learning systems in radiation oncology

Abstract

PurposeTools for assessing the severity and risk of near-miss events in radiation oncology are few and needed. Recent work has described guidelines for the use of a 5-tier near-miss risk index (NMRI) for the classification of near-miss events. The purpose of this study was to assess the reliability of the NMRI among users in a radiation oncology department. Methods and materialsReliability of the NMRI was assessed using an online survey distributed to members of a radiation oncology department. The survey contained 70 events extracted from the department’s incident learning system (ILS). Survey participants rated each event using the NMRI guidelines, reported their attendance to weekly ILS meetings (used as a surrogate for familiarity with the ILS), and indicated their familiarity with the radiation oncology workflow. Interrater reliability was determined using Krippendorff’s alpha. Use of the NMRI to rate actual events during 5 weekly ILS meetings was also assessed and interrater reliability determined. ResultsTwenty-eight survey respondents represented a wide variety of care providers. Krippendorff’s alpha was calculated for the whole respondent cohort to be 0.376, indicating fair agreement among raters. Respondents who had the most participation at ILS meetings (n = 4) had moderate agreement with an alpha of 0.501. Interestingly, there were significant differences in reliability and median NMRI scores between professions. NMRI use during weekly NMRI meetings (80 events rated), participants showed moderate reliability (alpha = 0.607). ConclusionsUsing the NMRI guidelines, raters from a wide variety of professions were able to assess the severity of near-miss incidents with fair agreement. Those experienced with the ILS showed better agreement, and higher agreement was seen during multidisciplinary ILS meetings. These data support the use the indices such as the NMRI for near-miss risk assessment in patient safety and prioritization of process improvements in radiation oncology.